[02:29:26] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 14d 10h 37m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [02:49:15] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:49:15] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:29:26] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 14d 6h 37m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [07:50:04] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12423824 (10ayounsi) [07:50:05] 10netops, 06Infrastructure-Foundations: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563#12423825 (10ayounsi) [09:28:32] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12424097 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d0aeef7c-1051-4ca5-b9ec-22a2953df15a) set by cmooney@cumin1004 for 1:00:00 on 18 host(s) and their servic... [09:29:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12424104 (10ayounsi) We should probably try to cross check devices: * Connect a mgmt port showing the issue on a neighbor switch (te... [09:47:02] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12424186 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=15f83802-d41c-4de8-9b3f-978d7aa13ee8) set by cmooney@cumin1004 for 1:00:00 on 18 host(s) and their servic... [10:03:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12424279 (10cmooney) >>! In T440631#12424104, @ayounsi wrote: > We should probably try to cross check devices: > * Reboot the mgmt s... [10:08:01] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12424315 (10cmooney) [10:10:11] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12424333 (10cmooney) I've made a start on this tackling //ssw1-d1-eqiad// first, and will do the same on //ssw1-d8-eqiad// in a few days if all seems ok. Reaso... [10:29:47] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 14d 2h 36m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [12:54:17] FIRING: SystemdUnitFailed: netbox_report_librenms_run.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:56:13] 10netbox, 06Infrastructure-Foundations: Netbox deply fails to run sudo service uwsgi-netbox restart - https://phabricator.wikimedia.org/T440981 (10ayounsi) 03NEW [12:56:29] 10netbox, 06Infrastructure-Foundations: Netbox deply fails to run sudo service uwsgi-netbox restart - https://phabricator.wikimedia.org/T440981#12425131 (10ayounsi) [13:19:17] FIRING: [2x] SystemdUnitFailed: netbox_report_accounting_run.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:24:17] FIRING: [2x] SystemdUnitFailed: netbox_report_accounting_run.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:24:36] 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Netbox deply fails to run sudo service uwsgi-netbox restart - https://phabricator.wikimedia.org/T440981#12425248 (10ayounsi) I asked Claude to have a look at this task, here is its reply. I manually checked that it was indeed correct, there are sta... [13:41:12] 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Netbox deploy fails to run sudo service uwsgi-netbox restart - https://phabricator.wikimedia.org/T440981#12425348 (10LSobanski) [14:04:17] FIRING: [2x] SystemdUnitFailed: replicate-krb-database.service on krb1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:09:17] FIRING: [4x] SystemdUnitFailed: replicate-krb-database.service on krb1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:26:46] 07Puppet, 06SRE: Ensure filenames invalid in windows are not commited to operations/puppet - https://phabricator.wikimedia.org/T353487#12425596 (10LSobanski) a:05jhathaway→03None [14:29:47] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 13d 22h 36m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [14:30:43] 10Mail, 06Infrastructure-Foundations, 07Sustainability (Incident Followup): Follow up for mx1001 incident: 2023-05-17 MXQueueHigh on mx1001 - https://phabricator.wikimedia.org/T337257#12425623 (10LSobanski) 05Open→03Invalid This incident involved exim, which we no longer run on mx servers. [14:34:21] FIRING: [4x] SystemdUnitFailed: replicate-krb-database.service on krb1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:38:24] 10Mail, 06Infrastructure-Foundations, 06SRE, 10Wikimedia-Mailing-lists: Replace Exim on lists.wikimedia.org with Postfix - https://phabricator.wikimedia.org/T378021#12425663 (10jhathaway) a:05jhathaway→03None [14:38:36] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Enable inbound Zendesk SMTP Connector support - https://phabricator.wikimedia.org/T393131#12425664 (10jhathaway) 05Open→03Resolved [14:39:03] 10Mail, 06Infrastructure-Foundations: Exim: add lists and auto-generated headers - https://phabricator.wikimedia.org/T347831#12425669 (10jhathaway) a:05jhathaway→03None [14:40:03] 10Mail, 06Infrastructure-Foundations, 06SRE: Log tls cipher information - https://phabricator.wikimedia.org/T381927#12425671 (10jhathaway) 05Open→03Resolved [14:40:40] 07Puppet: validate systemd units - https://phabricator.wikimedia.org/T392629#12425674 (10jhathaway) a:05jhathaway→03None [14:41:18] 07Puppet: puppetserver: CRL updates not picked up by compile only hosts - https://phabricator.wikimedia.org/T392709#12425675 (10jhathaway) a:05jhathaway→03None [14:41:31] 07Puppet: Non-ca puppetservers do not check the CA certificate revocation list or CRL - https://phabricator.wikimedia.org/T392637#12425676 (10jhathaway) a:05jhathaway→03None [14:42:18] 07Puppet: facter cli throws an exception on hosts with lvm - https://phabricator.wikimedia.org/T391526#12425679 (10jhathaway) a:05jhathaway→03None [14:49:17] FIRING: [4x] SystemdUnitFailed: replicate-krb-database.service on krb1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:49:29] 10Mail, 06Infrastructure-Foundations: Add additional RSA DKIM keys with 2048 bit sizes - https://phabricator.wikimedia.org/T365389#12425701 (10jhathaway) a:05jhathaway→03None [15:04:17] FIRING: [3x] SystemdUnitFailed: replicate-krb-database.service on krb1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:09:05] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Mark all traffic from upload.wikimdeia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990 (10cmooney) 03NEW p:05Triage→03Medium [15:09:21] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Mark all traffic from upload.wikimdeia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12425814 (10cmooney) [15:09:56] 10Mail, 06Infrastructure-Foundations: Some emails coming from Gerrit are being tagged as suspicious by Gmail - https://phabricator.wikimedia.org/T226884#12425821 (10jhathaway) a:05jhathaway→03None [15:11:11] 10Mail, 06Infrastructure-Foundations: Failed email communication for an User account - https://phabricator.wikimedia.org/T434744#12425825 (10jhathaway) @Neechalkaran if you can send another test email, I can check the server logs [15:13:32] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Terminal configuration for cookbooks - https://phabricator.wikimedia.org/T429129#12425835 (10jhathaway) a:05jhathaway→03None [15:13:41] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12425836 (10Nirmos) [15:14:11] 10Mail, 06Collaboration-Services, 06Infrastructure-Foundations, 06SRE, and 3 others: Replace Exim on VRTS servers with Postfix - https://phabricator.wikimedia.org/T378028#12425837 (10jhathaway) a:05jhathaway→03None [15:15:58] 10Mail, 06Collaboration-Services, 06Infrastructure-Foundations, 10Phabricator, 06SRE: Replace Exim on phabricator servers with Postfix - https://phabricator.wikimedia.org/T378029#12425861 (10jhathaway) a:05jhathaway→03None [15:16:26] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12425864 (10cmooney) [15:17:48] 10Mail, 06Infrastructure-Foundations, 06ServiceOps, 06SRE: Sendmail network error (deployment) - https://phabricator.wikimedia.org/T407723#12425871 (10jhathaway) a:05jhathaway→03None [15:18:35] 07Puppet, 06Infrastructure-Foundations: Improve the user experience adding new nodes to puppet - https://phabricator.wikimedia.org/T389932#12425874 (10jhathaway) a:05jhathaway→03None [15:40:29] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: modify move-server script to allow for 1G->10G upgrade during process - https://phabricator.wikimedia.org/T440999 (10cmooney) 03NEW p:05Triage→03Low [15:54:21] FIRING: [2x] SystemdUnitFailed: krb5-admin-server.service on krb1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:54:17] FIRING: [2x] SystemdUnitFailed: krb5-admin-server.service on krb1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:59:17] FIRING: [2x] SystemdUnitFailed: krb5-admin-server.service on krb1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:12:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014 (10cmooney) 03NEW p:05Triage→03High [17:12:38] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12426764 (10cmooney) [17:12:41] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12426765 (10cmooney) [17:22:49] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: modify move-server script to allow for 1G->10G upgrade during process - https://phabricator.wikimedia.org/T440999#12426846 (10RobH) Cathal and I chatted about this in IRC so I'm summarizing my points here for task visibility. I don't really wanna slow d... [17:29:11] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: modify move-server script to allow for 1G->10G upgrade during process - https://phabricator.wikimedia.org/T440999#12426889 (10RobH) Edit addition. I understand this a bit better, it would be mainly for the 85 wikikube hosts in rows A/B 1) new 10G conne... [17:38:38] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: modify move-server script to allow for 1G->10G upgrade during process - https://phabricator.wikimedia.org/T440999#12426949 (10cmooney) >>! In T440999#12426889, @RobH wrote: > I understand this a bit better, it would be mainly for the 85 wikikube hosts in... [17:54:17] FIRING: [2x] SystemdUnitFailed: krb5-admin-server.service on krb1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:59:17] FIRING: [2x] SystemdUnitFailed: krb5-admin-server.service on krb1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:15:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12427147 (10cmooney) [18:27:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12427186 (10cmooney) [18:29:48] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 13d 18h 36m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [18:59:17] FIRING: [2x] SystemdUnitFailed: krb5-admin-server.service on krb1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:04:17] FIRING: [2x] SystemdUnitFailed: krb5-admin-server.service on krb1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:50:58] sorry for the krb alert spam, working on it now [20:32:38] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1354876 this should take care of the admin services running on non-masters if anyone has time to review [21:59:48] RESOLVED: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:29:48] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 13d 14h 36m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry