[07:00:39] Morning! [08:19:59] TZAG! IIRC there was a tool that surfaced how many edits were made by Toolforge tools - does anyone have a link to that? [08:21:41] there was a statistics page with that info somewhere... looking (I think that it might have been moved) [08:28:40] bitergia was the name, looking [08:30:48] not sure now xd [08:39:21] we got it: https://wmcs-edits.wmcloud.org/#wmcs-edits [08:39:53] got to get my laptop plugged in at some coffee or something somewhere, be back in a bit [09:08:46] * dcaro back [09:18:24] I lost connection to toolforge bastion, having trouble getting to cloudcumin too, anyone has issues? [09:19:13] wfm [09:19:25] okok, good xd [09:19:58] both working fine here [09:20:19] okok, might just be my connection, got a scare xd [09:34:09] hmmm... I remember in the past we had this kind of issues from time to time, and rebooting fixed it, it seemed to be calico messing up the natting rules or something and needing to clean all of them from start [09:34:17] (still looking though) [09:35:01] did it trigger any monitoring? we might need to add more if not [09:35:29] I created T429578 to track the PAWS alert [09:35:29] T429578: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578 [09:37:21] volans: it did not, we don't have active monitoring from within jobs yet (there's a task somewhere xd) [09:38:23] k [09:40:55] +1 for setting it up though ;) [09:41:23] we were thinking on having some 'node health checker' daemon pod on every node doing some basic checks [09:44:11] is thois something you can detect only from within a pod or maybe also via k8s apis or node monitoring? [09:44:40] I also created T429579 to track the "exim paniclog" email that was sent out two days ago [09:44:41] T429579: Sporadic "exim paniclog" on tools-bastion host - https://phabricator.wikimedia.org/T429579 [09:46:10] hmmm.... the bastion and the workers all go out through the same ips? [09:47:09] oh, it works now [09:48:06] :/ [09:49:56] hmm... worker nfs-57 started working, nfs-80 still fails [09:50:33] huh... If I run a curl to google.com, then it start working [09:50:45] maybe some route cache somewhere? [09:51:10] yep, working now steadily on worker nfs-80 :/ [09:59:40] brb [11:56:54] hmpf... I don't understand what's happening, it seems it's intermittent, now it fails sometimes when trying to run it on worker-103 [11:57:00] https://www.irccloud.com/pastebin/cUUESEpW/ [11:58:15] it did get there though: [11:58:16] 11:57:54.841592 [0-0] == Info: connect to 207.241.237.3 port 443 from 192.168.13.8 port 47084 failed: Connection refused [11:58:38] might be just them throttling, let me check if I get any failures from the bastion [11:59:40] hmm... none fails so far [12:00:11] anyone knows if we use the same ip to get out from bastion as from the workers? (I think so, but maybe something changed) [12:13:31] it should be yeah, what does 'curl ifconfig.me' say ? [12:16:34] one sec [12:17:48] hah, the bastion goes through ip6 [12:18:35] that might explain yeah [12:20:15] pods use ip4 [12:20:30] and they use different public ips [12:20:40] 185.15.56.99 for pods, 185.15.56.1 for bastion [12:21:42] wait no, 99 bastion, .1 the pods [12:31:15] yep, it has all the signs of some kind of throttling, I can reproduce so far on 6 different worker nodes [12:31:32] (sometimes they work, sometimes they get connection refused) [12:33:26] Is there a way for us to see how many outgoing requests are we sending there? [12:33:35] maybe cloudgw might have a nat table [12:35:37] we seem to have ~23 connections currently there [12:35:42] https://www.irccloud.com/pastebin/dcCRA0HD/ [12:36:05] iabot? [12:37:00] probably some tools fetching stuff, there's a few that interact with internet archive [12:37:11] let me see if I can track them down [12:38:29] it's natted through the worker ips it seems, so I have which workers, one step closer id [12:38:30] xd [12:41:41] tool-arquibot seems to be doing some [12:45:31] it has gone down a bit [12:48:25] also iabot yep [12:48:30] directly from the vm [12:49:04] so seems to me valid traffic (maybe can be improved a bit) [12:49:44] should I try to contact internet archive to see if they have limits and maybe increase them from our network? [12:50:47] I'll add to the weekly meet :) [13:15:33] do we still think the power supply alert on cloudbackup2003 is related to switch maintenance? Even though it's still firing today? [13:31:42] mmh probably not, I see the alert has severity task, yet I can't find a task for it, expected ? [13:44:59] I don't think I know what severity: task means -- should there be an auto task created? [13:45:22] I was going to make one [13:46:57] T429608 [13:46:58] T429608: Power Supply - Status - issue on cloudbackup2003:9290 - https://phabricator.wikimedia.org/T429608 [13:47:23] that's correct yes there should be a task open automatically, per https://wikitech.wikimedia.org/wiki/Alertmanager#How_do_I_get_an_alert_to_open_a_task? [13:47:46] hm. if there is one I can't find it either. [13:48:50] opening a task to get tasks opened [13:49:06] maybe first we should open a task about opening that task? [13:50:08] on it [13:52:24] T429609 [13:52:25] T429609: PowerSupplyFailure for cloudbackup2003 did not open a task even though it had severity: task - https://phabricator.wikimedia.org/T429609 [13:57:16] I think at some point we disabled the task creation for all severities, and that affected severity: task as well :) [13:57:24] probably we should restore it just for that severity [13:58:17] that would explain indeed [14:01:29] it was removed in: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198517 [16:42:20] I claimed T425303 but I realized I don't know anything about the deployment flow for toolhub. andrewbogott maybe you built it in the past, or apps with a similar flow? [16:42:20] T425303: Replace Bullseye base image with Trixie before August 2026 EOL deadline - https://phabricator.wikimedia.org/T425303 [16:42:52] I found the blubber definition for the pipeline, but I don't know how to test changes [16:44:25] I would like to test what happens if I change the base image here https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/toolhub/+/refs/heads/main/.pipeline/blubber.yaml#4 [16:46:26] I have also never deployed it. bd808 might know who has [16:57:36] I didn't want to bug b.ryan while he's out, and I was hoping you had some knowledge :) [16:57:53] dhinus, andrewbogott: https://wikitech.wikimedia.org/wiki/Toolhub.wikimedia.org [16:58:31] bd808: thanks, I'll go through that page! [16:59:20] It should all be testable locally. At least in theory. I haven’t looked at the test deployment for a very long time. [17:02:03] I'm trying to understand how to trigger a rebuild of the docker image, and if I should try that locally, or if there's any pipeline that builds the image for a patch (before merging) [17:04:21] * dcaro off [17:04:23] cya on monday! [17:04:27] dhinus: atsukoito just did config changes and follow up maintenance to switch search backends. They may be helpful if you get stuck. The container is built by CI. It will publish after a merge to gerrit. The makefile will build it locally for you. [17:04:47] ack thanks! [17:05:34] * bd808 disappears back into the mist [17:52:08] volans: I'm still confused by your comment on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302236/16/hieradata/common/profile/openstack/codfw1dev.yaml -- there is a key called 'cumin_masters' in hieradata/cloud/eqiad1.yaml but there is also a 'cumin_masters' key in the prod path for eqiad. So how can I look up the key in that file and not in e.g. common.yaml? Doesn't the key need a fully::qualified::path for me [17:52:09] to specify which hierarchy to look up? [17:53:36] yes, sorry, I copy pasted the existing alias usage and forgot to replace it with fqdn [17:55:33] I haven't used alias much so still don't really understand. I can look for examples... but you're at least convinced that there's no way to do it /without/ using alias? [17:57:05] without alias you have to hardcode the values twice, that is a pain when they change, with alias you are just saying use those values from this other place [17:57:28] and alias can point to a different lookup path [17:57:54] yes like profile::openstack::base::openstack_control_nodes: "%{alias('profile::openstack::eqiad1::openstack_control_nodes')}" [17:58:18] * andrewbogott tries it [17:58:29] as for using alternatively the "global-per-openstack-installation" values directly, I'm not sure if our puppettization has some form that makes the resolution of eqiad1 and codfw1dev automatic [17:58:36] PCC should tell you if it works ;) [18:36:29] pcc seems to be taking an early weekend [18:37:23] * volans logging off, tty on monday