[07:59:23] morning [08:12:55] Roo.k: it applies to anything that is in the production network (bare metal hosts, ganeti VMs, ...), those systems included. I think you might be able to workaround things like pip using the http proxy (https://wikitech.wikimedia.org/wiki/HTTP_proxy), but only for tests and such. As taav.i said for any system running in production, the rule is that it should not depend on external systems to get reinstalled from scratch. [08:15:28] hello! any idea which tool sends the email "PROBLEM alert - cloudbackup2001/Disk space is CRITICAL" it's not in my inbox, but it does trigger an auto-reply from Bryan Davis to all of the SREs (root@). I fixed most of the problematic script, but I can't find this one (context: https://phabricator.wikimedia.org/T347835 ) [08:16:37] XioNoX: sounds like something sent from Icinga? [08:16:50] yeah I'd say icinga as well [08:17:58] ah, right, but only sent to the cloud team? [08:18:27] sounce:icinga, team:wmcs [08:18:28] https://usercontent.irccloud-cdn.com/file/3KiOPLQF/image.png [08:18:50] let me check if I have emails [08:19:25] yeah found it! [08:19:42] 👍 [08:19:46] it's in alert1001:/etc/icinga/objects/notification_commands.cfg and it's not pretty :) [08:20:22] thanks! [08:20:55] I think we can skip emails for most of our icinga alerts, but should be discussed team-wide first [08:21:32] I'll add the proper header to avoid the auto-replies, then it's up to you [08:21:43] thanks :) [09:55:35] * taavi lunch [11:10:46] * arturo was in a rabbit hole reading about kubevela after david's email [11:10:55] * arturo errand, be back later [13:25:03] * arturo food time [13:58:03] I found a fun issue from the gitlab image/chart publishing pipeline: https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/22 [14:02:05] arturo: you might be interested in: https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/34 [14:10:23] taavi: the stacked MRs might not be such a good idea :/ [14:10:45] as that will create one image for every MR, so if you have 10MRs, the images from the last one will already push out the images the first one built [14:14:33] if there's a better workflow, I'm more than happy to switch to it :/ [14:15:20] Better is quite subjective xd [14:15:55] lol [14:17:22] we could either not build images by default, but that makes it a bit annoying, as we would have to manually trigger it somehow, try to build only for merge requests to main (that would only build for the first stacked MR), or try somehow to trigger only for the last stacked MR (that seems a bit more complicated). Or put a super high limit on the amount of images [14:18:03] hmm, I wonder how much space do they take, if the stacked images are very similar, the extra space should be kinda small [14:21:59] it also means one image per [14:22:07] commit (as in prod image) [14:22:49] another alternative would be to use multi-commit MRs [14:23:16] so instead of 10 MRs with 10 dev images each + 10 prod images, you end up with 1 dev image and 1 prod image [14:34:36] taavi: looking [15:52:10] * dcaro off [16:22:04] * arturo finished reviewing 100 patches and goes offline [16:38:43] toolsbeta harbor ran out of disk, I freed up some space (T348337) so it should last until monday [16:38:43] T348337: toolsbeta harbor instance ran out of disk - https://phabricator.wikimedia.org/T348337 [16:41:22] * taavi off [16:53:09] XD visionary timing!