[00:01:25] !log dzahn@cumin1004 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie [00:01:42] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12372162 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1004 for host zuul1005.eqiad.wmnet with OS tr... [00:10:23] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 613.29 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:10:37] PROBLEM - MariaDB Replica Lag: s3 on db1240 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 630.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:17:12] !log dzahn@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage [00:21:26] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [puppet] - 10https://gerrit.wikimedia.org/r/1345087 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [00:22:37] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage [00:32:16] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12372206 (10Dzahn) I reimaged zuul1006 and zuul1007 with the cookbook and no issues. zuul1005 kept having the issue we have previously seen... [00:46:24] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.32 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:50:32] (03PS1) 10Andrea Denisse: kubernetes: Add dummy KLAXON_CORTO_TOKEN for oncall [labs/private] - 10https://gerrit.wikimedia.org/r/1345674 (https://phabricator.wikimedia.org/T439495) [01:08:56] (03PS1) 10TrainBranchBot: Branch commit for wmf/1.47.0-wmf.22 [core] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345678 (https://phabricator.wikimedia.org/T438218) [01:08:58] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/1.47.0-wmf.22 [core] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345678 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [01:09:03] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345679 [01:09:03] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345679 (owner: 10TrainBranchBot) [01:16:43] (03Merged) 10jenkins-bot: Branch commit for wmf/1.47.0-wmf.22 [core] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345678 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [01:17:20] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345679 (owner: 10TrainBranchBot) [01:20:52] PROBLEM - Host wikikube-worker1016 is DOWN: PING CRITICAL - Packet loss = 60%, RTA = 3082.05 ms [01:21:30] RECOVERY - Host wikikube-worker1016 is UP: PING OK - Packet loss = 0%, RTA = 18.10 ms [01:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:49:18] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () [02:00:05] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0200) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0200) [02:00:35] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:06:38] RECOVERY - MariaDB Replica Lag: s3 on db1240 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:08:03] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:22:38] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [02:24:26] RECOVERY - dump of s4 in codfw on backupmon1001 is OK: Last dump for s4 at codfw (db2239) taken on 2026-09-29 01:02:41 (180 GiB, +0.1 %) https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [02:27:40] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 226.32 ms [02:29:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [02:37:30] FIRING: Traffic bill over quota: Alert for device cr2-magru.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [02:39:01] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:57:30] RESOLVED: Traffic bill over quota: Alert for device cr2-magru.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [03:00:05] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0300) [03:01:57] (03PS1) 10TrainBranchBot: testwikis to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345691 (https://phabricator.wikimedia.org/T438218) [03:01:59] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by mwpresync@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345691 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [03:02:58] (03Merged) 10jenkins-bot: testwikis to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345691 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [03:03:22] !log mwpresync@deploy1003 Started scap sync-world: testwikis to 1.47.0-wmf.22 refs T438218 [03:03:26] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [03:37:09] (03PS7) 10Ryan Kemper: kafka: converge topic config from hieradata [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) [03:39:25] !log mwpresync@deploy1003 Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs T438218 (duration: 36m 03s) [03:39:28] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [04:00:05] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0400) [04:01:05] !log mwpresync@deploy1003 Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s) [04:31:05] (03PS1) 10KartikMistry: Update cxserver to 2026-09-25-114231-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345697 [04:31:55] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm [04:32:17] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1033 [04:32:18] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033 [04:43:35] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage [04:44:16] (03CR) 10LWatson: [C:03+1] carousel: fix carousel image preload observer root [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345633 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [04:44:38] (03CR) 10LWatson: [C:03+1] Make carousel images are visible in the margins [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345627 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [04:47:59] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage [04:51:35] !log manually restarting dump_cloud_ip_range on puppetserver1001 after failure [04:51:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:03:02] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm [05:07:25] (03PS1) 10Fabfur: external_clouds_vendors: minor logging fix [puppet] - 10https://gerrit.wikimedia.org/r/1345850 [05:17:55] PROBLEM - Host wikikube-worker1289 is DOWN: PING CRITICAL - Packet loss = 100% [05:19:28] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm [05:19:40] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1034 [05:19:40] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034 [05:20:25] RECOVERY - Host wikikube-worker1289 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [05:21:12] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm [05:21:35] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1035 [05:21:35] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035 [05:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:31:14] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage [05:32:59] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage [05:34:47] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage [05:38:15] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage [05:50:29] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm [05:54:01] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm [05:55:39] 06SRE, 06Data-Engineering, 10Kafka-Infrastructure, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Monitor kafka-configurator runs and pending changes - https://phabricator.wikimedia.org/T439506 (10RKemper) 03NEW [05:57:36] (03PS2) 10Giuseppe Lavagetto: kubernetes::deployment_server: setup daily timer to reload certificates [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0600) [06:00:05] marostegui, cezmunsta, and federico3: Time to snap out of that daydream and deploy Primary database switchover. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0600). [06:01:28] 06SRE, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 4 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12372533 (10RKemper) [06:02:50] (03PS1) 10Ryan Kemper: aptrepo: import kafka-configurator from apt-staging [puppet] - 10https://gerrit.wikimedia.org/r/1345852 (https://phabricator.wikimedia.org/T435282) [06:06:10] FIRING: BFDdown: BFD session down between cr1-codfw and fe80::26fc:4e07:1041:5917 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [06:11:10] RESOLVED: BFDdown: BFD session down between cr1-codfw and fe80::26fc:4e07:1041:5917 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [06:19:06] (03PS1) 10Kosta Harlan: CheckUser: Enable ClientHintsInEditRequest on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345854 (https://phabricator.wikimedia.org/T401808) [06:19:37] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344405 (https://phabricator.wikimedia.org/T436480) (owner: 10Southparkfan) [06:19:39] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345854 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [06:29:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [06:39:01] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:45:05] (03PS1) 10Jelto: etherpad: update image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345857 (https://phabricator.wikimedia.org/T438866) [06:48:30] Deploying cxserver. [06:49:18] (03CR) 10KartikMistry: [C:03+2] Update cxserver to 2026-09-25-114231-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345697 (owner: 10KartikMistry) [06:49:49] (03CR) 10Jelto: [C:03+2] etherpad: update image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345857 (https://phabricator.wikimedia.org/T438866) (owner: 10Jelto) [06:51:57] (03PS2) 10Matthias Mullie: Cache extracted carousel data [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345632 (https://phabricator.wikimedia.org/T439182) [06:52:05] (03Merged) 10jenkins-bot: Update cxserver to 2026-09-25-114231-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345697 (owner: 10KartikMistry) [06:52:18] (03Merged) 10jenkins-bot: etherpad: update image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345857 (https://phabricator.wikimedia.org/T438866) (owner: 10Jelto) [06:55:11] (03CR) 10Brouberol: [C:03+1] aptrepo: import kafka-configurator from apt-staging [puppet] - 10https://gerrit.wikimedia.org/r/1345852 (https://phabricator.wikimedia.org/T435282) (owner: 10Ryan Kemper) [06:56:52] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [06:57:31] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [07:00:05] Amir1, urbanecm, and awight: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) UTC morning backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0700). [07:00:05] Matthias and kostajh: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:06] o/ [07:00:21] I'll get started [07:00:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345626 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:00:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345632 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:00:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345627 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [07:00:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345633 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [07:01:39] !log kartik@deploy1003 helmfile [staging] START helmfile.d/services/cxserver: apply [07:02:00] !log kartik@deploy1003 helmfile [staging] DONE helmfile.d/services/cxserver: apply [07:03:52] (03Merged) 10jenkins-bot: Skip expensive work if known to be useless [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345626 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:03:54] (03Merged) 10jenkins-bot: Cache extracted carousel data [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345632 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:04:03] (03Merged) 10jenkins-bot: Make carousel images are visible in the margins [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345627 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [07:04:04] (03Merged) 10jenkins-bot: carousel: fix carousel image preload observer root [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345633 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [07:04:49] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1345626|Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632|Cache extracted carousel data (T439182)]], [[gerrit:1345627|Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633|carousel: fix carousel image preload observer root (T439422)]] [07:04:54] T439182: [Regression] MediaViewer hook adds 30% to Backend Pageview Timing on mobile (300 -> 400ms) - https://phabricator.wikimedia.org/T439182 [07:04:54] T439422: [Image Browsing] Carousel images should extend beyond content margins - https://phabricator.wikimedia.org/T439422 [07:06:33] PROBLEM - OSPF status on cr4-ulsfo is CRITICAL: OSPFv2: 2/4 UP : OSPFv3: 2/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [07:07:03] matthiasmullie: can you ping me when you're finished please? [07:07:03] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 8/9 UP : OSPFv3: 8/9 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [07:07:09] kostajh: will do! [07:07:29] PROBLEM - OSPF status on cr2-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [07:07:39] FIRING: CoreBGPDown: Core BGP session down between cr2-codfw and cr4-ulsfo (198.35.26.157) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Confed_ulsfo&var-bgp_neighbor=cr4-ulsfo - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [07:08:10] FIRING: [4x] BFDdown: BFD session down between cr2-codfw and 198.35.26.157 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [07:09:06] !log mlitn@deploy1003 mlitn: Backport for [[gerrit:1345626|Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632|Cache extracted carousel data (T439182)]], [[gerrit:1345627|Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633|carousel: fix carousel image preload observer root (T439422)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now [07:09:06] be verified there. [07:11:08] !log mlitn@deploy1003 mlitn: Continuing with deployment [07:12:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr4-ulsfo (198.35.26.157) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [07:14:50] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org [07:18:06] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12372676 (10fgiunchedi) I set jbod for cloudcephosd1055 via the mgmt web interface, and tried cephosd-efi recipe, which prompted me for a partitioning choi... [07:18:28] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345626|Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632|Cache extracted carousel data (T439182)]], [[gerrit:1345627|Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633|carousel: fix carousel image preload observer root (T439422)]] (duration: 13m 39s) [07:18:33] T439182: [Regression] MediaViewer hook adds 30% to Backend Pageview Timing on mobile (300 -> 400ms) - https://phabricator.wikimedia.org/T439182 [07:18:33] T439422: [Image Browsing] Carousel images should extend beyond content margins - https://phabricator.wikimedia.org/T439422 [07:18:38] kostajh: the floor is yours [07:18:48] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org [07:19:01] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:21:30] matthiasmullie: thx! [07:21:50] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345854 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [07:22:36] (03Merged) 10jenkins-bot: CheckUser: Enable ClientHintsInEditRequest on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345854 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [07:22:45] !log kartik@deploy1003 helmfile [codfw] START helmfile.d/services/cxserver: apply [07:22:58] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1345854|CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] [07:23:01] T401808: Allow Client Hints data to be used to prevent the save of edits - https://phabricator.wikimedia.org/T401808 [07:23:19] !log kartik@deploy1003 helmfile [codfw] DONE helmfile.d/services/cxserver: apply [07:24:17] !log kartik@deploy1003 helmfile [eqiad] START helmfile.d/services/cxserver: apply [07:24:47] !log kartik@deploy1003 helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [07:26:00] !log Updated cxserver to 2026-09-25-114231-production [07:26:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:27:10] !log kharlan@deploy1003 kharlan: Backport for [[gerrit:1345854|CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:27:57] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet [07:30:23] !log kharlan@deploy1003 kharlan: Continuing with deployment [07:31:35] (03PS1) 10Matthias Mullie: carousel: keep the hide button attached to the carousel [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345862 (https://phabricator.wikimedia.org/T439507) [07:31:47] (03PS1) 10Matthias Mullie: carousel: keep the hide button attached to the carousel [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345863 (https://phabricator.wikimedia.org/T439507) [07:33:44] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet [07:34:10] kostajh: can you ping me when you're done? I'll do another :p [07:35:39] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345854|CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] (duration: 12m 41s) [07:35:43] T401808: Allow Client Hints data to be used to prevent the save of edits - https://phabricator.wikimedia.org/T401808 [07:37:47] matthiasmullie: sure, I'm done [07:38:00] aight thanks [07:38:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345862 (https://phabricator.wikimedia.org/T439507) (owner: 10Matthias Mullie) [07:38:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345863 (https://phabricator.wikimedia.org/T439507) (owner: 10Matthias Mullie) [07:39:07] (03Merged) 10jenkins-bot: carousel: keep the hide button attached to the carousel [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345862 (https://phabricator.wikimedia.org/T439507) (owner: 10Matthias Mullie) [07:39:19] (03Merged) 10jenkins-bot: carousel: keep the hide button attached to the carousel [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345863 (https://phabricator.wikimedia.org/T439507) (owner: 10Matthias Mullie) [07:39:42] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1345862|carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863|carousel: keep the hide button attached to the carousel (T439507)]] [07:39:46] T439507: Image Browsing: keep the carousel hide button attached to the carousel - https://phabricator.wikimedia.org/T439507 [07:41:07] (03Abandoned) 10Fabfur: external_clouds_vendors: minor logging fix [puppet] - 10https://gerrit.wikimedia.org/r/1345850 (owner: 10Fabfur) [07:43:51] !log mlitn@deploy1003 mlitn: Backport for [[gerrit:1345862|carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863|carousel: keep the hide button attached to the carousel (T439507)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:44:22] !log mlitn@deploy1003 mlitn: Continuing with deployment [07:49:40] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345862|carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863|carousel: keep the hide button attached to the carousel (T439507)]] (duration: 09m 57s) [07:49:43] T439507: Image Browsing: keep the carousel hide button attached to the carousel - https://phabricator.wikimedia.org/T439507 [07:49:50] !log UTC morning backport window done [07:49:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:52:30] (03CR) 10Ilias Sarantopoulos: ml-services: expose gpt-oss-safeguard-20b publicly (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [07:58:01] (03CR) 10Ozge: [C:03+1] ml-services: expose gpt-oss-safeguard-20b publicly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [07:58:55] (03CR) 10Ozge: [C:03+1] "cool, let's test if this will work." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [07:59:15] (03CR) 10Ozge: [C:03+1] ml-services: expose gpt-oss-safeguard-20b publicly (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [08:03:43] (03PS1) 10Muehlenhoff: thumbor-plugins: Rebuild against latest package versions in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1345920 [08:08:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker2269:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker2269 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [08:10:54] (03PS1) 10Slyngshede: P:IDP Tomcat 11 for CAS 8.X [puppet] - 10https://gerrit.wikimedia.org/r/1345921 (https://phabricator.wikimedia.org/T439509) [08:10:56] !log installing PHP 8.4 security updates [08:10:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:14:03] 06SRE, 06Traffic: Prometheus Ferm MSS export does not work for IPv6 - https://phabricator.wikimedia.org/T433672#12372837 (10cmooney) 05Open→03Resolved a:03cmooney Thanks @CDobbins for getting to the bottom of this! [08:15:11] (03CR) 10Ilias Sarantopoulos: [C:03+2] ml-services: expose gpt-oss-safeguard-20b publicly (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [08:15:19] (03PS1) 10Slyngshede: P:tofurkey enable Tofurkey for DRMRS [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) [08:17:36] (03Merged) 10jenkins-bot: ml-services: expose gpt-oss-safeguard-20b publicly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [08:18:57] !log isaranto@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [08:21:06] (03CR) 10Slyngshede: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [08:21:26] (03CR) 10Slyngshede: "Enable Tofurkey on DRMRS before enabling everywhere." [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [08:21:46] (03CR) 10Muehlenhoff: [C:03+2] Remove cumin1003 from the alertmanager and tcpircbot configs [puppet] - 10https://gerrit.wikimedia.org/r/1345087 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [08:26:28] (03CR) 10Muehlenhoff: [C:03+1] "My spot checks were also fine and PCC looks also good (the only puzzling thing is that mx1001/2001 are still present despite being long go" [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [08:26:37] (03CR) 10Cathal Mooney: [C:03+1] "LGTM!" [homer/public] - 10https://gerrit.wikimedia.org/r/1344682 (https://phabricator.wikimedia.org/T439136) (owner: 10Ayounsi) [08:28:14] (03CR) 101Veertje: "I've minimised the changes to the LoginForm specific for the WM install" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (owner: 101Veertje) [08:37:24] (03PS1) 10Slyngshede: P:idp add missing dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1345925 [08:38:28] (03CR) 10Slyngshede: [V:03+2 C:03+2] P:idp add missing dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1345925 (owner: 10Slyngshede) [08:39:01] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:39:36] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-f7-eqiad [08:39:43] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f7-eqiad [08:39:46] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9485/console" [puppet] - 10https://gerrit.wikimedia.org/r/1345921 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [08:40:15] jouncebot: nowandnext [08:40:15] No deployments scheduled for the next 1 hour(s) and 19 minute(s) [08:40:15] In 1 hour(s) and 19 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1000) [08:40:31] (03CR) 10Urbanecm: [C:03+2] ReassignMentees: Actually deduplicate reassignMenteesJob [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344712 (https://phabricator.wikimedia.org/T418194) (owner: 10Urbanecm) [08:40:38] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-e6-eqiad [08:40:44] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e6-eqiad [08:40:53] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-e7-eqiad [08:40:59] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e7-eqiad [08:41:08] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-f6-eqiad [08:41:16] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad [08:41:23] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-f5-eqiad [08:41:30] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f5-eqiad [08:41:54] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-e5-eqiad [08:42:01] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-eqiad [08:42:52] (03CR) 10Ayounsi: [C:03+2] Remove "hostkey_algorithm" in all sites except eqiad/codfw [homer/public] - 10https://gerrit.wikimedia.org/r/1344682 (https://phabricator.wikimedia.org/T439136) (owner: 10Ayounsi) [08:44:15] (03Merged) 10jenkins-bot: Remove "hostkey_algorithm" in all sites except eqiad/codfw [homer/public] - 10https://gerrit.wikimedia.org/r/1344682 (https://phabricator.wikimedia.org/T439136) (owner: 10Ayounsi) [08:45:08] (03CR) 10A-pizzata: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343954 (https://phabricator.wikimedia.org/T437961) (owner: 10A-pizzata) [08:46:34] (03PS1) 10Dpogorzelski: liftwing-studio: rebuild images on the 20260927 bases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345926 [08:47:46] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344712 (https://phabricator.wikimedia.org/T418194) (owner: 10Urbanecm) [08:47:59] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: rebuild images on the 20260927 bases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345926 (owner: 10Dpogorzelski) [08:48:04] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] liftwing-studio: rebuild images on the 20260927 bases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345926 (owner: 10Dpogorzelski) [08:48:16] (03Merged) 10jenkins-bot: ReassignMentees: Actually deduplicate reassignMenteesJob [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344712 (https://phabricator.wikimedia.org/T418194) (owner: 10Urbanecm) [08:48:35] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1344712|ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] [08:48:38] T418194: Mentors still having mentees after removing themselves - https://phabricator.wikimedia.org/T418194 [08:49:16] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [08:50:13] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [08:52:47] !log urbanecm@deploy1003 urbanecm: Backport for [[gerrit:1344712|ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:53:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker2269:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker2269 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [08:54:19] !log urbanecm@deploy1003 urbanecm: Continuing with deployment [08:57:55] (03CR) 10Arnaudb: "Comparing the two charts it seems that the major two differences are the Recreate strategy and the ConfigMap for Aphlict, which would requ" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341856 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [08:59:43] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344712|ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] (duration: 11m 07s) [08:59:46] T418194: Mentors still having mentees after removing themselves - https://phabricator.wikimedia.org/T418194 [09:04:15] (03PS1) 10Muehlenhoff: Replace grant for cumin1003 with cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1345931 (https://phabricator.wikimedia.org/T427897) [09:04:26] (03PS1) 10Cathal Mooney: Add reverse zone delegations for new dse-k8s-eqiad range 10.68.0.0/17 [dns] - 10https://gerrit.wikimedia.org/r/1345933 (https://phabricator.wikimedia.org/T430658) [09:05:37] (03CR) 10CI reject: [V:04-1] Add reverse zone delegations for new dse-k8s-eqiad range 10.68.0.0/17 [dns] - 10https://gerrit.wikimedia.org/r/1345933 (https://phabricator.wikimedia.org/T430658) (owner: 10Cathal Mooney) [09:06:03] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1345931 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:06:41] (03PS1) 10Ayounsi: Exclude ae interfaces from CoreOutboundSaturation alert [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) [09:06:49] (03CR) 10CI reject: [V:04-1] Exclude ae interfaces from CoreOutboundSaturation alert [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:07:57] (03Abandoned) 10Cathal Mooney: Add reverse zone delegations for new dse-k8s-eqiad range 10.68.0.0/17 [dns] - 10https://gerrit.wikimedia.org/r/1345933 (https://phabricator.wikimedia.org/T430658) (owner: 10Cathal Mooney) [09:08:49] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 359403696 and 47 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [09:09:34] (03CR) 10Ayounsi: "recheck" [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:09:47] 06SRE, 06Infrastructure-Foundations: Sunset rootless Cumin - https://phabricator.wikimedia.org/T439522 (10MoritzMuehlenhoff) 03NEW [09:10:32] (03CR) 10Muehlenhoff: [C:03+2] Replace grant for cumin1003 with cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1345931 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:10:33] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade Cumin hosts to Trixie - https://phabricator.wikimedia.org/T427897#12373070 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff All core Cumin servers are on Trixie. The cloucumin hosts are tracked via T436667 and cuminunpri... [09:11:50] (03CR) 10Ayounsi: "CI error seems unrelated to the change itself." [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:15:02] (03CR) 10Cathal Mooney: [C:03+1] "indeed yeah." [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:18:18] (03CR) 10Tiziano Fogli: "recheck" [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:19:15] (03PS1) 10Dreamy Jazz: maintenance: Set FOREACHWIKI_IGNORE_ERRORS for some PSI jobs [puppet] - 10https://gerrit.wikimedia.org/r/1345937 (https://phabricator.wikimedia.org/T432110)