[00:06:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:06:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:11:25] FIRING: [2x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:41:25] FIRING: [2x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:36:25] RESOLVED: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:41:42] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12089148 (10ayounsi) Please keep in mind that if a site has more than one instance with the gnmi class, it will split the targets between them, so it's not possible to have a full "test" host. T... [08:52:34] 10netops, 06Infrastructure-Foundations, 06SRE: Blackbox probe for TLS cert expriy failing on multiple eqiad SR-Linux nodes - https://phabricator.wikimedia.org/T429242#12089176 (10ayounsi) I had a quick look at the doc and indeed I couldn't find a way to increase the check interval. [12:09:17] 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12090057 (10ops-monitoring-bot) Draining ganeti2048.codfw.wmnet of running VMs [12:33:58] 10SRE-tools, 06Infrastructure-Foundations: Diffscan: investigate IPv6 support and explore other scanning tooling - https://phabricator.wikimedia.org/T265329#12090166 (10ayounsi) I created a quick Netbox [[ https://netbox.wikimedia.org/extras/export-templates/35/ | export template ]] that lists all the known IP... [14:42:39] moritzm: o/ Janis confirmed that going to bookworm for aux-k8s-etcd1* and 2* should be fine, shall I proceed? [14:43:14] sounds good! [15:28:53] 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12090952 (10MoritzMuehlenhoff) [15:37:36] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Do we need prometheus-ethtool-exporter? - https://phabricator.wikimedia.org/T371375#12090984 (10LSobanski) Is there a plan to also remove prometheus-ethtool-exporter from Trixie hosts provisioned before the change was merged? Collab gets lo... [15:44:45] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Do we need prometheus-ethtool-exporter? - https://phabricator.wikimedia.org/T371375#12091022 (10MoritzMuehlenhoff) >>! In T371375#12090984, @LSobanski wrote: > Is there a plan to also remove prometheus-ethtool-exporter from Trixie hosts pro... [15:47:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599#12091030 (10Papaul) ` Hello Papaul, Apologies for the late response. We are currently coordinating with our internal team for further investigation. We will keep yo... [15:56:04] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Do we need prometheus-ethtool-exporter? - https://phabricator.wikimedia.org/T371375#12091066 (10LSobanski) You are right, my assumption was invalid and this was in fact a Bookworm host. [16:22:57] elukey: [16:23:01] eggggg IRC [16:23:48] ? [16:52:20] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12091218 (10Dzahn) [16:53:55] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12091221 (10Dzahn) re: contint2002 - This is not the main CI server, that is currently contint1002, but it is still used to a certain extent. That... [17:01:16] o/ we'd like to migrate the remaining checks for the docker registry to prometheus https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306351 [19:44:36] 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12091876 (10Scott_French) Thanks for your patience while I was out last week, @ayounsi. In short, conf2004 (or any conf* host) will require manu... [23:42:24] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12092634 (10Eevans) [23:42:50] 10netops, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12092635 (10Eevans)