[02:20:26] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:45:07] 10Mail, 06Infrastructure-Foundations, 10Znuny: DKIM / DMARC for domains on VRTS - https://phabricator.wikimedia.org/T428540 (10Krd) 03NEW [06:20:26] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:21:34] FIRING: DiskSpace: Disk space ganeti1039:9100:/ 3.908% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=ganeti1039 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:36:35] RESOLVED: DiskSpace: Disk space ganeti1039:9100:/ 2.352% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=ganeti1039 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:57:40] 10netbox, 06Infrastructure-Foundations: Netbox: add IPAddress validator to ensure unique dns_name - https://phabricator.wikimedia.org/T428546 (10ayounsi) 03NEW p:05Triage→03Low [08:15:26] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:25:06] cdanis: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1299415 proper location this time! I'd be interested in pointers on how to deploy to turnilo-next first. [08:45:26] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:20:26] FIRING: [7x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:20:26] FIRING: [7x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:29:23] !log shut sub-interfaces for row A/B legacy vlans on cr1-codfw T427357 [12:29:23] topranks: Not expecting to hear !log here [12:29:23] T427357: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357 [12:50:47] XioNoX: https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Code_deployment/configuration_changes basically but it will be helmfile.d/dse-k8s-services/turnilo-next [12:51:40] cdanis: thanks! brouberol wrote https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Turnilo#(re)Deploying_turnilo too in the meantime [12:54:25] nice [13:21:58] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11999553 (10ayounsi) 05Open→03Resolved a:03ayounsi Maintenance done, all servers except Ganeti and the ones mentioned by @jcres... [13:26:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#11999603 (10ayounsi) [13:27:42] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11999608 (10MatthewVernon) ms swift in codfw looks OK after this work, thanks. [13:51:39] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11999792 (10ops-monitoring-bot) VM netflow2004.codfw.wmnet switching disk type to drbd [14:04:20] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11999859 (10ops-monitoring-bot) VM rpki2003.codfw.wmnet switching disk type to drbd [14:08:00] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11999886 (10MoritzMuehlenhoff) All Ganeti nodes are back in service [14:10:02] cdanis: it's live on https://turnilo-next.wikimedia.org/#webrequest_sampled_live/ [14:10:15] XioNoX: thx! mind if we leave it like that for a day? [14:10:24] yeah of course [14:10:51] https://usercontent.irccloud-cdn.com/file/TxFhKvxV/Screenshot%20From%202026-06-09%2016-07-57.png https://usercontent.irccloud-cdn.com/file/xB1GkKHO/Screenshot%20From%202026-06-09%2016-07-16.png [14:13:29] alphabetical is much better than before, thx [14:13:31] https://w.wiki/Qkd6 [14:38:58] XioNoX, topranks: I've upgraded routinator in codfw to 0.15.2, let me know if you notice any issues. otherwise I'd followup with eqiad tomorrow [14:39:36] journald output on rpki2003 looks all fine to me [14:40:26] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:41:50] moritzm: checked a few core routers they all seem happy, have synced the full set of routes [14:41:53] thanks! [14:43:30] ack, thanks [15:58:14] jhathaway: o/ I have some time to discuss https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1293593 if you are available [15:59:17] sure, I was working on reviewing the current patch as we speak [16:03:23] ah okok, I am around for 10/15 mins more, this is why I asked :) [16:03:28] otherwise we always miss each other [16:05:05] true! [16:05:35] I am working the puppet request window, but happy to hop on a call after that, if you are still around [16:05:57] patch looks good in general, just trying to understand the etag bits a little more [16:06:06] okok ping me when done! [20:52:33] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed