[02:06:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:06:40] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:01:29] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052#12326132 (10SLyngshede-WMF)
[08:43:51] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052#12326457 (10SLyngshede-WMF) Traffic is moving to ULSFO: https://grafana.wikimedia.org/d/000000093/cdn-frontend-network?from=now-30m&to=now&timezone...
[09:21:05] <XioNoX>	 moritzm: of course there is a complication :) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1342191
[09:22:38] <moritzm>	 looking
[09:22:52] <wikibugs>	 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984#12326599 (10ayounsi)
[09:26:36] <moritzm>	 XioNoX: looks good
[09:26:44] <XioNoX>	 thx
[09:31:34] <XioNoX>	 moritzm: cool, fully up and running now
[09:31:58] <moritzm>	 nice!
[09:43:19] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work, 07Kubernetes: Calico IPv4 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12326742 (10Gehel)
[10:06:40] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:01:55] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052#12327258 (10Fabfur)
[11:04:30] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052#12327268 (10cmooney)
[11:21:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:45:23] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12327750 (10cmooney) @Jclark-ctr thanks for the help with the switches in rack A1 and the cabling.  Both //lsw1-a1-eqiad// and //ssw1-a1-eqiad// are reachabl...
[12:53:32] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052#12327789 (10dkertesz) 05Open→03Resolved
[13:46:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:13:36] <wikibugs>	 10CFSSL-PKI, 06Infrastructure-Foundations, 13Patch-For-Review: Add a load balanced service in front of PKI - https://phabricator.wikimedia.org/T436809#12328343 (10elukey) Another interesting problem! We have two ports serving cfssl traffic on every pki node, 443 and 8443:  * Port 443 uses mTLS for client aut...
[14:19:26] <wikibugs>	 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984#12328397 (10ayounsi)
[14:48:00] <wikibugs>	 10CFSSL-PKI, 06Infrastructure-Foundations, 13Patch-For-Review: Add a load balanced service in front of PKI - https://phabricator.wikimedia.org/T436809#12328692 (10jhathaway) If we are comfortable with only HMAC signing for k8s, that I would be comfortable with only HMAC signing for our servers.
[16:31:23] <wikibugs>	 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Add tracing support to Spicerack - https://phabricator.wikimedia.org/T418874#12329598 (10LSobanski)
[19:49:34] <XioNoX>	 topranks: https://gnmic.openconfig.net/changelog/#v0490-16-september-2026 another release, and I haven't had time to upgrade all nodes to 0.48 yet :)
[19:49:46] <XioNoX>	 "TLS certificate hot reload for target connections" will be useful for us
[20:14:12] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12330829 (10Jclark-ctr) @cmooney Unplugged and reseated B7, and the link came back up. I also connected the links between the D8 and D1 spines with the A1 sp...