[07:31:01] greetings [08:40:49] morning [09:11:07] hello! [09:20:48] hello [10:40:57] dhinus: volans macos quick fix for making functional tests faster! https://gitlab.wikimedia.org/toolforge-repos/sample-static-buildpack-app/-/merge_requests/1 [10:55:23] dcaro: nice thanks! I'm still confused by why the standard uwsgi would sometimes install quickly, but sometimes not [11:00:29] I have no idea why it is so slow on rosetta but I'd guess that in the layers of lima-kilo we're not fully arm, so it's getting tanslated once or twice. https://werkzeug.palletsprojects.com/en/stable/deployment/uwsgi/#installing says to go that way with an implied they do do wheels directly... not sure why [11:01:24] Damianz: yes we're definitely not fully arm in lima-kilo, so translation might have an impact [11:02:25] but I did some testing with a simple debian VM, installed amd64 kind, started a k8s cluster and inside that one "pip install uwsgi" was instant, no compilation [11:02:29] It's quite amazing just how good rosetta is these days [11:02:56] so I'm not sure what happens in the lima-kilo vm, maybe the difference is the base docker image used during the tekton build [11:04:02] * taavi lunch [11:04:20] in any case, I think using pyuwsgi makes sense and hopefully will make it fast everywhere [11:15:05] sorry was in a meeting [11:28:27] I'm trying to test rebuilding sample-complex-app in lima-kilo, but now I have a different problem: the kube config is missing in tf-test [11:28:48] I tried restarting both ldap and maintain-kubeusers, but that didn't help.. [11:38:37] ok I fixed it by manually calling ./container/utils/add-lima-kilo-tool-account.sh inside the foxtrot-ldap pod [11:38:51] the same command that should have been run by ansible [11:41:20] and I can confirm that the build is now much faster using pyuwsgi, the functional test "build (slow)" completes in less than 1 min [11:49:15] there's a toolforge_ldap_.... script that should do that [11:49:45] if you rebooted you lima-kilo, it might be needed, as the storage is not persistent I think (did not get deep on it yet) [11:51:01] how does uwsgi perform as a static file server as compared to something like apache/nginx (from the php buildpack)? [11:51:02] the script is `toolforge_load_users_to_ldap.sh` [11:52:31] taavi: no special difference, mostly less code iirc, but happy to re-evaluate/change if you find it's better [11:53:31] asking because uwsgi's docs recommend leaving static file serving to a 'proper' web server https://uwsgi-docs.readthedocs.io/en/latest/StaticFiles.html [11:54:01] it could be done also with aptfile instead, and use nginx without having to pull a runtime, but did not spend the time to get it working [11:56:47] taavi: hmm, I understand something different from that help page, as in uwsgi is designed as a solution for PaaS to allow individual users to handle their static files [11:57:05] The uWSGI project has ISPs and PaaS (that is, the hosting market) as the main target, where generally you would want to avoid generating disk I/O on a central server and have each user-dedicated area handle (and account for) that itself. More importantly still, you want to allow customers to customize the way they serve static assets without bothering your system administrator(s). [11:57:42] (that does not change though that it can be implemented with nginx/lihtttpd/apache/php/node/...) [11:57:58] I did consider switching to gunicorn/uvicorn and whitenoise which is at least to me the right way of doing it without a fronting server but that would be way more complex [11:58:00] * dcaro lunch [11:58:27] I think that note is more about proxy buffering protecting the instance, which in this usage happens [11:58:31] we can try to come up with a "best practice" static server with a known good recipe [11:59:15] though I'd expect most users to have a mixture of static+dynamic, tying them to a language of sort [11:59:33] anyhow, happy to change it if anyone has ideas :) [12:00:03] I'm just happy to be able to test my pending MRs I'd [12:06:58] dcaro: I recreated lima-kilo and /data/project/tf-test/.kube/config is there, I'm not sure why it wasn't there before [12:07:19] thanks for the pointer to toolforge_load_users_to_ldap.sh [12:30:56] https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/11 [12:51:28] if anyone will be around for the co-working space slot I could show what I have so far with loki and ask a few questions on the proper way to puppetize the secrets parts [13:06:20] I can join in 10mn [13:06:39] ack [15:05:24] taavi, dcaro: Is it ok if I "play" with the haproxy in toolsbeta a bit? I have an idea for how to move the iw rewrite there, but it needs some exploratory testing. I'm looking for a shortcut to standing up a local haproxy with proper TLS to do my testing on. [15:05:41] bd808: yeah, totally fine [15:07:49] awesome. thanks. [15:08:20] bd808: +1, thanks for the notice :) [15:13:50] even better if you do it on the inactive instance [15:14:41] oh, active-passive should make it easier to stay out of the way. Will do. [15:25:11] for some reason by rebuilt 3 times now lima-kilo is broken with no changes running functional tests, so I'll have to come back to testing my last set of MRs... got to do the accounts now [15:47:21] do you have any more details on why it broke? I'll try rebuilding mine, but I'm running something so might take a bit [15:50:32] just rebuilt mine and running the tests [15:55:03] The build reports publishing image to harbor but the status never gets out of running [15:55:17] volans: re the certificate authentication stuff, some time ago I wrote this https://git.majava.org/config/puppet/tree/modules/profile/files/kubernetes/management/export_certs.py to automate having up-to-date certs signed by the k8s internal CA (for use cases like the current prometheus cert) [15:56:13] nice, why we don't use it? :D [15:57:06] Damianz: on my lima-kilo a few tests passed but then this one failed with "failed to create fsnotify watcher: too many open files": maintain-harbor/delete-stale-toolforge-artifacts [15:57:43] I just had a build pass so will see how far it gets this run [15:57:45] oh, that one is a known one, there's an ansible variable iirc [15:57:55] taavi: that looks like a nicer https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/ed0b480a8341a35661d99038c4a01f0f56fe123e/modules/kubeadm/files/admin_scripts/wmcs-k8s-get-cert.sh [15:58:35] we can replace this with that :) [16:00:34] dhinus: it took more than 10min but the last run just worked for me... so I guess I had some random state that worked it's self out [16:00:36] it will need to get wrapped in a timer or something, and a way to distribute the certs (maybe run on the hosts that need it? but then you need the kubectl certs to run the cert creation itself xd) [16:31:32] foxtrot ldap failed to start on my rebuild [16:31:42] https://www.irccloud.com/pastebin/1McE9W7b/ [16:31:45] looking [16:32:28] I think it was changed from deployment to stateful set? [16:33:40] ah yep, is there something depending on it being a deployment? [16:34:13] yep, the ansible stuff to get the pods [16:34:20] (so it runs scripts on them) [16:34:42] sorry :/ [16:34:47] np, patching [16:34:56] (who does stuff breaks stuff :) ) [16:40:38] we just needs tests for the tests that test things that are testing things that have tests [16:41:10] fix for the statefulset https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/292 [16:41:19] lolz [16:54:39] quick review, just revealing the logs-api in tools openapi.json spec: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1035 [16:55:12] dcaro: why is that setting not needed for any other apis? [16:55:33] it was there for components also, defaults to true for the rest [16:55:50] for the logs-api still defaults to false, so needs setting to true [16:56:21] I'll do the cleanup on the api-gateway code/setting later, but this enables us to easily flip the switch if needed [16:57:04] ah [16:57:04] +1 [17:00:04] thanks :) [17:11:27] ahem... I can't get OP in here to change topic... [17:11:55] one second [17:12:01] no hurry :) [17:13:50] https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/31 [17:14:19] <3 [17:17:51] !isssync [17:17:55] !issync [17:17:55] Syncing #wikimedia-cloud-admin (requested by Majavah) [17:17:57] Set /cs flags #wikimedia-cloud-admin volans +Afiortv [17:19:32] ^ bd808: shouldn't that op down after doing the changes? [17:27:26] * dhinus off, and out tomorrow. see you on Monday! [18:00:44] taavi: hmmm... yeah I thought ircservserv only took +o when making changes. Its bee a minute since I had that code loaded into my head though. [18:01:21] add iw to https://toolhub.wikimedia.org/lists/192 [18:01:24] *added [18:02:27] can lists in toolhub be edited by anyone? I started that mostly for myself [18:02:46] but might be a good way to keep track of tools we should care about [18:02:53] *as in, maintenance care about [18:03:30] Toolhub lists are "owned" by the creator. there is a feature request in the backlog to change that and y'all own the project now so... patches welcome? [18:03:42] :) ack [18:03:45] T317610 [18:03:46] T317610: Make lists editable by multiple people - https://phabricator.wikimedia.org/T317610 [18:04:15] Raymond_Ndi.be did some changes though? https://usercontent.irccloud-cdn.com/file/hnszZuZ7/image.png [18:04:22] admin privileges? [18:04:24] Toolhub is very "MVP". It works but there is a lot more that it could do [18:04:43] dcaro: yeah, super users are sneaky like that [18:05:31] https://toolhub.wikimedia.org/members?groups_id=1 -- these are "root" users who can edit anything [18:05:31] ack, might be an ugly workaround until we get that feature in [18:06:15] * bd808 prunes astinson from that list :/ [18:06:32] hopefully when we fill the backfills we'll be able to start working on more stuff [18:07:01] (or so the legend goes) [18:07:30] I won't hold my breath. Toolhub development halted when Slavina and Raymond were moved to work on Toolforge. [18:08:39] the "new" software team is mostly the same team we had built to work on Toolhub as far as I understand it; the long missing SWE team to help the SRE team [18:10:29] there's the tools-platform team that inherits toolforge and toolhub no? [18:11:41] that one currently has dhinu.s, Raymond_Ndib.e, koml.a and me, but we are missing backfills too [18:12:09] (afaik) [18:13:15] that's functionally the prior Toolhub team, but yes the plan may be to scale it up slightly. There were 2-4 of us on Toolhub over the 2.5 years that project was active. [18:16:28] definitely won't be able to focus as much on it as you did, but I hope we can do a bit more than currently we do [18:19:12] I have in the past been poked by someone I can't remember about toolinfo in toolsadmin for reasons of filling in toolhub, but it's not something I ever look at [18:25:41] okok, /me clocking out [18:25:52] cya on monday! (or before if I fool around) [18:25:54] * dcaro off [19:53:57] Damianz: there are very much folks who look at Toolhub to find out about what tools exist, where the code is, where the help docs are, etc. But I also get that it is a weird site outside of the wikis and the Toolforge direct experience which makes it harder to get everyone to use. [19:54:40] My unrealized dream is finishing the MediaWiki extension that puts all the data into the wikis via Lua modules and special pages [23:41:47] I didn't get back to the iw problem today, but I'm going to try again tomorrow.