[00:01:13] FIRING: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [00:02:02] !incidents [00:02:03] 8317 (UNACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [00:02:08] ack 8317 [00:04:58] RESOLVED: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:06:13] RESOLVED: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [00:16:15] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1017.eqiad.wmnet with OS bookworm [00:17:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.1% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:21:40] FIRING: [3x] ProbeDown: Service aqs1017-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:26:40] FIRING: [4x] ProbeDown: Service aqs1017-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:36:40] FIRING: [4x] ProbeDown: Service aqs1017-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:41:40] RESOLVED: [4x] ProbeDown: Service aqs1017-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:49:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:06:52] FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [01:07:47] (03PS1) 10David Martin: wikifunctions: Add 2 callback tests to check-wf-services.py [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334186 (https://phabricator.wikimedia.org/T421848) [01:11:10] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1334188 [01:11:10] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1334188 (owner: 10TrainBranchBot) [01:19:45] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1334188 (owner: 10TrainBranchBot) [01:22:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.11% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:22:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:32:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:01:23] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:09:12] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 48s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:15:19] (03CR) 10RLazarus: [C:03+1] modules: Prepare mesh.configuration / networkpolicy minor version bump (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1328738 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [02:15:55] (03CR) 10RLazarus: [C:03+1] "Nice, thanks!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331858 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:27:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:35:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.42% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:40:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.55% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:45:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:00:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:02:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:12:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:27:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:29:59] (03PS1) 10Catrope: Email confirmation A/A test: make registration cutoff consistent [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1334268 (https://phabricator.wikimedia.org/T435135) [03:30:42] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1334268 (https://phabricator.wikimedia.org/T435135) (owner: 10Catrope) [03:41:08] FIRING: GnmiInterfaceCountersDrop: ... [03:41:08] cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr2-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [03:50:38] (03CR) 10Samwilson: [C:03+1] "Looks good to me now. Thanks!" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333246 (owner: 10Ladsgroup-claude) [04:07:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:21:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:26:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:29:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:39:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:45:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:47:35] (03CR) 10Samwilson: [C:03+1] "Done" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333246 (owner: 10Ladsgroup-claude) [04:49:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:00:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:00:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:02:02] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334323 [05:02:28] PROBLEM - Host mr1-magru.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [05:06:52] FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [05:07:30] RECOVERY - Host mr1-magru.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 114.28 ms [05:15:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:18:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:27:03] (03CR) 10Samwilson: "Looks good, works in my tests. :)" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333247 (owner: 10Ladsgroup-claude) [05:33:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:47:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:51:26] 10ops-eqiad, 06DC-Ops: Unresponsive management for cloudcephosd1036.mgmt:22 - https://phabricator.wikimedia.org/T436894 (10phaultfinder) 03NEW [05:56:46] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host vrts2002.codfw.wmnet [05:57:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:58:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.59% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T0600) [06:00:05] marostegui, cezmunsta, and federico3: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Primary database switchover. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T0600). [06:00:31] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host vrts1004.eqiad.wmnet [06:03:18] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts2002.codfw.wmnet [06:04:19] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1004.eqiad.wmnet [06:04:28] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet [06:11:04] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet [06:23:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:26:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 14.77% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:27:03] !log Upgrading CI Jenkins on contint1003 # T436812 [06:27:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:33:26] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1333280 (https://phabricator.wikimedia.org/T436703) (owner: 10Kamila Součková) [06:41:20] (03CR) 10JMeybohm: [C:03+2] kubernetes: Add a comment about kubelet cgroup driver selection [puppet] - 10https://gerrit.wikimedia.org/r/1332719 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [06:43:59] 06SRE, 10LDAP-Access-Requests: Grant Access to wmf, for WMF staff/contractors nda group for gsduser - https://phabricator.wikimedia.org/T435852#12284148 (10Aklapper) 05Open→03Declined > Is this going to give me access to the web request tables and other data cubes in turnilo and superset? Depends on t... [06:44:38] (03CR) 10Mahmoud-abdelsattar: "recheck" [extensions/Wikibase] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333866 (https://phabricator.wikimedia.org/T434204) (owner: 10Thiemo Kreuz (WMDE)) [06:45:02] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12284150 (10MoritzMuehlenhoff) [06:46:06] !log installing libxml2 security updates [06:46:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:52:16] !log jmm@cumin2003 START - Cookbook sre.wdqs.restart-nginx-envoy rolling restart_daemons on A:wcqs-public [06:54:05] !log jmm@cumin2003 END (PASS) - Cookbook sre.wdqs.restart-nginx-envoy (exit_code=0) rolling restart_daemons on A:wcqs-public [06:55:23] (03CR) 10Arnaudb: [C:03+2] Revert^2 "gitlab: discard firewall throttling on the primary" [puppet] - 10https://gerrit.wikimedia.org/r/1332731 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [06:58:24] (03CR) 10Arnaudb: [C:03+2] Revert^2 "gitlab: point gitlab.wikimedia.org at the CDN" [dns] - 10https://gerrit.wikimedia.org/r/1332733 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [06:58:40] !log arnaudb@dns1006 START - running authdns-update [07:00:04] Amir1, urbanecm, and awight: Time to snap out of that daydream and deploy UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T0700). [07:00:04] Thiemo_WMDE and Hide_on_rosie: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:01:16] !log arnaudb@dns1006 END - running authdns-update [07:02:15] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 03 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/Wikibase] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333866 (https://phabricator.wikimedia.org/T434204) (owner: 10Thiemo Kreuz (WMDE)) [07:05:02] PROBLEM - Gitlab SSH healthcheck git daemon on gitlab.wikimedia.org is CRITICAL: connect to address gitlab.wikimedia.org and port 22: Connection refused https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [07:05:14] this is normal ↑ that check should be updated [07:05:35] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Requesting Kerberos access for kamila - https://phabricator.wikimedia.org/T436703#12284205 (10MLechvien-WMF) > @MLechvien-WMF as my manager, can you please approve this? I approve that request [07:09:14] ACKNOWLEDGEMENT - Gitlab SSH healthcheck git daemon on gitlab.wikimedia.org is CRITICAL: connect to address gitlab.wikimedia.org and port 22: Connection refused Arnaud Bran known, due to move behind CDN, check update in progress - The acknowledgement expires at: 2026-09-07 07:08:46. https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [07:09:52] o/ Hide_on_rosie, need a deployer? [07:10:13] Yes. [07:10:33] gotcha. you're the only one on this window now; ready to get started? [07:10:35] WikimediaDebug installed. [07:10:41] Im ready [07:10:43] awesome :) [07:12:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by chlod@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1333872 (https://phabricator.wikimedia.org/T436651) (owner: 10Tryvix1509) [07:13:00] (03Merged) 10jenkins-bot: core-Permissions.php: Allow English Wikiquote administrators to grant and remove the confirmed user permission [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1333872 (https://phabricator.wikimedia.org/T436651) (owner: 10Tryvix1509) [07:13:19] (03PS1) 10Samwilson: Return early in CoreHandler if no URL match is found [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1334595 (https://phabricator.wikimedia.org/T436897) [07:13:50] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1228.eqiad.wmnet with reason: Onsite maintenance [07:14:09] 06SRE, 10LDAP-Access-Requests: Grant Access to wmf, for WMF staff/contractors nda group for gsduser - https://phabricator.wikimedia.org/T435852#12284221 (10Aklapper) 05Declined→03Open [07:15:17] !log Power off db1228 for maintenance [07:15:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:15:36] !log chlod@deploy1003 Started scap sync-world: Backport for [[gerrit:1333872|core-Permissions.php: Allow English Wikiquote administrators to grant and remove the confirmed user permission (T436651)]] [07:15:39] T436651: Allow English Wikiquote administrators to grant and remove the confirmed user permission - https://phabricator.wikimedia.org/T436651 [07:15:59] (03PS1) 10Arnaudb: icinga: check gitlab SSH against gitlab-ssh.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1334598 (https://phabricator.wikimedia.org/T425441) [07:17:45] (03PS1) 10Slyngshede: data.yaml update information for htriedman [puppet] - 10https://gerrit.wikimedia.org/r/1334601 [07:18:39] (03CR) 10CI reject: [V:04-1] data.yaml update information for htriedman [puppet] - 10https://gerrit.wikimedia.org/r/1334601 (owner: 10Slyngshede) [07:19:15] (03PS2) 10Slyngshede: data.yaml update information for htriedman [puppet] - 10https://gerrit.wikimedia.org/r/1334601 [07:20:10] !log chlod@deploy1003 chlod, tryvix1509: Backport for [[gerrit:1333872|core-Permissions.php: Allow English Wikiquote administrators to grant and remove the confirmed user permission (T436651)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:20:15] Hide_on_rosie: please check :) [07:22:11] !log push pfw policies - T436729 [07:22:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:22:44] Check OK, ready for next strp [07:22:47] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1334601 (owner: 10Slyngshede) [07:22:48] *step [07:22:49] awesome [07:22:50] !log chlod@deploy1003 chlod, tryvix1509: Continuing with deployment [07:24:48] (03CR) 10Ayounsi: [C:03+2] Remove POPs GRE tunnels OSPF [homer/public] - 10https://gerrit.wikimedia.org/r/1332723 (owner: 10Ayounsi) [07:24:53] (03CR) 10Arnaudb: [C:03+2] icinga: check gitlab SSH against gitlab-ssh.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1334598 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [07:26:23] (03Merged) 10jenkins-bot: Remove POPs GRE tunnels OSPF [homer/public] - 10https://gerrit.wikimedia.org/r/1332723 (owner: 10Ayounsi) [07:27:30] !log chlod@deploy1003 Finished scap sync-world: Backport for [[gerrit:1333872|core-Permissions.php: Allow English Wikiquote administrators to grant and remove the confirmed user permission (T436651)]] (duration: 11m 54s) [07:27:33] T436651: Allow English Wikiquote administrators to grant and remove the confirmed user permission - https://phabricator.wikimedia.org/T436651 [07:27:52] done :) [07:28:51] Thanks! [07:29:17] no prob [07:29:46] !log UTC morning backport window done [07:29:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:31:02] RECOVERY - Gitlab SSH healthcheck git daemon on gitlab.wikimedia.org is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [07:35:42] (03PS1) 10Marostegui: installserver: Do not format db1270 [puppet] - 10https://gerrit.wikimedia.org/r/1334626 [07:38:25] (03CR) 10Marostegui: [C:03+2] installserver: Do not format db1270 [puppet] - 10https://gerrit.wikimedia.org/r/1334626 (owner: 10Marostegui) [07:39:33] !log elukey@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cassandra-dev2001.codfw.wmnet [07:41:08] FIRING: GnmiInterfaceCountersDrop: ... [07:41:08] cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr2-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [07:48:18] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed again - https://phabricator.wikimedia.org/T436743#12284289 (10Marostegui) @Jclark-ctr db1228 is now off so you can proceed and update anything you see that needs update. Thanks! [07:49:16] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host cassandra-dev2001.codfw.wmnet [07:51:34] 06SRE, 06Commons, 10MediaWiki-File-management, 06Traffic, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12284293 (10Tgr) Possibly Varnish normalizes thumb URLs but doesn't normalize original URLs (which until... [07:53:27] (03CR) 10Arnaudb: [V:03+2 C:03+2] "Gitlab looks stable behind CDN" [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1325854 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [07:58:49] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 23 hosts with reason: Changing sanitarium master in s2 [07:59:14] (03PS1) 10Muehlenhoff: Bump changelog for new release [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1334666 [07:59:16] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cassandra-dev2001.codfw.wmnet [07:59:18] !log elukey@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cassandra-dev2001.codfw.wmnet [07:59:25] !log elukey@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cassandra-dev2001.codfw.wmnet [08:00:05] dancy and hashar: Deploy window MediaWiki train - Utc-7+Utc-0 Version (secondary timeslot) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T0800) [08:01:22] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Bump changelog for new release [debs/wmf-laptop] - 10https://gerrit.wikimedia.org/r/1334666 (owner: 10Muehlenhoff) [08:03:14] !log Move s2 sanitarium from db1156 to db1271 T434287 [08:03:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:03:17] T434287: Move s2 sanitarium (db1155:3312) from db1156 to db1271 - https://phabricator.wikimedia.org/T434287 [08:04:34] 06SRE, 06Traffic: Benthos - remove the kafka output module - https://phabricator.wikimedia.org/T398916#12284308 (10Fabfur) 05Open→03Declined [08:06:36] 06SRE, 06Infrastructure-Foundations, 10netops, 10Observability-Metrics: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12284322 (10cmooney) 05Open→03Resolved >>! In T435855#12274375, @ayounsi wrote: > Max RTT doesn't show when usi... [08:08:37] (03PS1) 10Marostegui: mariadb: Move s2 sanitarium master [puppet] - 10https://gerrit.wikimedia.org/r/1334708 (https://phabricator.wikimedia.org/T407942) [08:09:37] (03CR) 10Cathal Mooney: [C:03+1] "LGTM nice!" [alerts] - 10https://gerrit.wikimedia.org/r/1321906 (owner: 10Ayounsi) [08:10:54] (03CR) 10Marostegui: [C:03+2] mariadb: Move s2 sanitarium master [puppet] - 10https://gerrit.wikimedia.org/r/1334708 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [08:11:07] !log uploaded wmf-laptop 1.0.7 to apt.wikimedia.org [08:11:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:11:46] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for cloudcephosd1036.mgmt:22 - https://phabricator.wikimedia.org/T436894#12284337 (10Jclark-ctr) a:03Jclark-ctr [08:12:25] elukey@cumin1003 upgrade-firmware (PID 2400072) is awaiting input [08:16:03] (03CR) 10Ayounsi: [C:03+2] Add TransportLinksInUsageNoRedundancy alert [alerts] - 10https://gerrit.wikimedia.org/r/1321906 (owner: 10Ayounsi) [08:18:07] (03Merged) 10jenkins-bot: Add TransportLinksInUsageNoRedundancy alert [alerts] - 10https://gerrit.wikimedia.org/r/1321906 (owner: 10Ayounsi) [08:21:35] 10SRE-Access-Requests, 06Infrastructure-Foundations, 10LDAP-Access-Requests, 13Patch-For-Review: Superset data access request for abibendall - https://phabricator.wikimedia.org/T430938#12284368 (10Volans) p:05High→03Medium [08:25:37] (03PS1) 10DCausse: cirrus-streaming-updater: bump to latest image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334719 (https://phabricator.wikimedia.org/T436736) [08:26:25] !log elukey@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cassandra-dev2001.codfw.wmnet [08:27:48] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cassandra-dev2001.codfw.wmnet with OS bookworm [08:30:53] FIRING: [3x] GnmiInterfaceCountersDrop: cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [08:32:31] !log jnuche@deploy1003 Started deploy [releng/jenkins-deploy@b146ae7] (releasing): T436812 to backup server [08:33:06] !log jnuche@deploy1003 Finished deploy [releng/jenkins-deploy@b146ae7] (releasing): T436812 to backup server (duration: 01m 28s) [08:33:53] (03PS1) 10Volans: admin: add kamila to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1334723 (https://phabricator.wikimedia.org/T436703) [08:34:07] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [08:34:46] (03CR) 10CI reject: [V:04-1] admin: add kamila to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1334723 (https://phabricator.wikimedia.org/T436703) (owner: 10Volans) [08:35:53] FIRING: [3x] GnmiInterfaceCountersDrop: cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [08:36:40] !log jnuche@deploy1003 Started deploy [releng/jenkins-deploy@b146ae7] (releasing): T436812 to prod server [08:37:19] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [08:37:44] !log jnuche@deploy1003 Finished deploy [releng/jenkins-deploy@b146ae7] (releasing): T436812 to prod server (duration: 01m 21s) [08:37:50] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [08:38:00] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-text_magru [08:38:47] (03PS2) 10Volans: admin: add kamila to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1334723 (https://phabricator.wikimedia.org/T436703) [08:39:41] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 24 hosts with reason: Changing sanitarium master in s5 [08:40:34] (03CR) 10Volans: admin: add kamila to analytics-privatedata-users (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1334723 (https://phabricator.wikimedia.org/T436703) (owner: 10Volans) [08:41:36] (03CR) 10Majavah: [C:03+2] P:elasticsearch: Remove unused profiles [puppet] - 10https://gerrit.wikimedia.org/r/1332772 (owner: 10Majavah) [08:42:04] !log Move s5 sanitarium from db1161 to db1275 T434776 [08:42:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:42:07] T434776: Move s5 sanitarium master from db1161 to db1275 - https://phabricator.wikimedia.org/T434776 [08:43:34] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [08:43:42] (03PS3) 10Majavah: elasticsearch: Remove unused classes [puppet] - 10https://gerrit.wikimedia.org/r/1332773 [08:43:58] (03PS1) 10Marostegui: mariadb: Update s5 sanitarium master [puppet] - 10https://gerrit.wikimedia.org/r/1334727 (https://phabricator.wikimedia.org/T434776) [08:45:35] (03CR) 10Marostegui: [C:03+2] mariadb: Update s5 sanitarium master [puppet] - 10https://gerrit.wikimedia.org/r/1334727 (https://phabricator.wikimedia.org/T434776) (owner: 10Marostegui) [08:45:45] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage [08:45:50] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-text_magru [08:45:55] (03CR) 10Majavah: [C:03+2] "PS3 is a manual rebase" [puppet] - 10https://gerrit.wikimedia.org/r/1332773 (owner: 10Majavah) [08:49:01] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage [08:49:19] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-text_codfw [08:49:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:50:11] (03CR) 10Ladsgroup: [C:03+2] Package the project and stop using pkg_resources [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333246 (owner: 10Ladsgroup-claude) [08:54:37] (03PS4) 10Jelto: service::catalog: Set ipip_encapsulation for citoid, cxserver in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1333691 (https://phabricator.wikimedia.org/T420436) [08:55:42] !log blake@cumin1003 START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-codfw [08:56:37] (03Merged) 10jenkins-bot: Package the project and stop using pkg_resources [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333246 (owner: 10Ladsgroup-claude) [08:57:04] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-text_codfw [08:57:44] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-text_drmrs [08:58:15] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Changing sanitarium master in s6 [09:00:47] (03PS1) 10Marostegui: mariadb: Change s6 sanitarium master [puppet] - 10https://gerrit.wikimedia.org/r/1334736 (https://phabricator.wikimedia.org/T434775) [09:01:05] 06SRE, 10LDAP-Access-Requests: Grant Access to wmf, for WMF staff/contractors nda group for gsduser - https://phabricator.wikimedia.org/T435852#12284455 (10Volans) p:05Triage→03Medium [09:02:53] (03CR) 10Marostegui: [C:03+2] mariadb: Change s6 sanitarium master [puppet] - 10https://gerrit.wikimedia.org/r/1334736 (https://phabricator.wikimedia.org/T434775) (owner: 10Marostegui) [09:03:21] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host aqs1018.eqiad.wmnet with OS bookworm [09:03:26] !log Move s6 sanitarium from db1165 to db1279 T434775 [09:03:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:03:29] T434775: Move s6 sanitarium master from db1165 to db1279 - https://phabricator.wikimedia.org/T434775 [09:03:31] (03PS3) 10Ladsgroup: Route all logging through thumbor's logger [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333247 (owner: 10Ladsgroup-claude) [09:05:10] (03CR) 10Ladsgroup: Route all logging through thumbor's logger (032 comments) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333247 (owner: 10Ladsgroup-claude) [09:05:22] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-text_drmrs [09:05:43] (03CR) 10Ayounsi: [C:03+1] sre.hosts.provision: make the FQDN check for SM dynamic [cookbooks] - 10https://gerrit.wikimedia.org/r/1333692 (https://phabricator.wikimedia.org/T419892) (owner: 10Elukey) [09:06:52] FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [09:06:54] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-text_esams [09:07:25] FIRING: [3x] ProbeDown: Service aqs1018-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:08:57] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2001.codfw.wmnet with OS bookworm [09:09:22] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cassandra-dev2001.codfw.wmnet with OS bookworm [09:10:59] (03CR) 10Filippo Giunchedi: [C:04-1] "Looks good! See inline" [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [09:12:25] FIRING: [4x] ProbeDown: Service aqs1018-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:12:52] jouncebot: nowandnext [09:12:53] For the next 0 hour(s) and 47 minute(s): MediaWiki train - Utc-7+Utc-0 Version (secondary timeslot) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T0800) [09:12:53] In 0 hour(s) and 47 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1000) [09:13:30] (03PS1) 10Zabe: HookHandler: Fix bail out condition in onLocalUserCreated subscriber [extensions/OATHAuth] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1334743 (https://phabricator.wikimedia.org/T436791) [09:14:16] (03CR) 10Zabe: [C:03+2] HookHandler: Fix bail out condition in onLocalUserCreated subscriber [extensions/OATHAuth] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1334743 (https://phabricator.wikimedia.org/T436791) (owner: 10Zabe) [09:14:42] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-text_esams [09:15:37] !log put traffic on Lumen codfw<->eqiad link as it is stable T435810 [09:15:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:15:40] T435810: Lumen eqiad<->codfw transport outage (Aug 2026). - https://phabricator.wikimedia.org/T435810 [09:17:36] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1018.eqiad.wmnet with reason: host reimage [09:17:59] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-text_eqsin [09:18:28] (03CR) 10Slyngshede: [C:03+2] data.yaml update information for htriedman [puppet] - 10https://gerrit.wikimedia.org/r/1334601 (owner: 10Slyngshede) [09:20:11] (03PS1) 10Muehlenhoff: Allow cumin1004 in one more tcpircbot setting [puppet] - 10https://gerrit.wikimedia.org/r/1334747 (https://phabricator.wikimedia.org/T427897) [09:20:30] (03Merged) 10jenkins-bot: HookHandler: Fix bail out condition in onLocalUserCreated subscriber [extensions/OATHAuth] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1334743 (https://phabricator.wikimedia.org/T436791) (owner: 10Zabe) [09:20:53] FIRING: [3x] GnmiInterfaceCountersDrop: cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [09:21:03] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1334743|HookHandler: Fix bail out condition in onLocalUserCreated subscriber (T436791)]] [09:21:07] T436791: InvalidArgumentException: Can't persist a key for user with no central ID available - https://phabricator.wikimedia.org/T436791 [09:22:01] (03CR) 10Btullis: [C:03+2] dumps: web: Produce access logs to the webrequest.dumps.v1 stream [puppet] - 10https://gerrit.wikimedia.org/r/1328206 (https://phabricator.wikimedia.org/T425087) (owner: 10Btullis) [09:22:25] FIRING: [4x] ProbeDown: Service aqs1018-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:23:03] (03PS1) 10Marostegui: db1174: Remove from dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1334748 (https://phabricator.wikimedia.org/T436904) [09:23:23] (03PS1) 10Blake: localmemcached: Add restart policy if we should be a sidecar. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334735 (https://phabricator.wikimedia.org/T417800) [09:23:32] (03PS1) 10Blake: statsd: Upgrade to 1.0.5 (copy) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334738 (https://phabricator.wikimedia.org/T417800) [09:23:32] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1018.eqiad.wmnet with reason: host reimage [09:23:46] (03CR) 10Marostegui: [C:03+2] db1174: Remove from dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1334748 (https://phabricator.wikimedia.org/T436904) (owner: 10Marostegui) [09:23:54] (03PS1) 10Blake: statsd: Enable a native k8s sidecar option. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334739 (https://phabricator.wikimedia.org/T417800) [09:24:12] (03PS1) 10Blake: mesh: upgrade to 1.3.3 (copy) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334741 (https://phabricator.wikimedia.org/T417800) [09:24:20] (03PS1) 10Blake: mesh: Add switch to enable k8s native sidecar behaviour. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334742 (https://phabricator.wikimedia.org/T417800) [09:24:48] !log marostegui@cumin1003 dbctl commit (dc=all): 'Remove db1174 from dbctl T436904', diff saved to https://phabricator.wikimedia.org/P96323 and previous config saved to /var/cache/conftool/dbconfig/20260903-092448-marostegui.json [09:24:52] T436904: decommission db1174.eqiad.wmnet - https://phabricator.wikimedia.org/T436904 [09:25:07] (03CR) 10Clément Goubert: "If I understand correctly, the goal is to set this up, then shift the ATS routing from `wdqs2-main-external.discovery.wmnet` and `wdqs2-sc" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333815 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [09:25:22] !log zabe@deploy1003 zabe: Backport for [[gerrit:1334743|HookHandler: Fix bail out condition in onLocalUserCreated subscriber (T436791)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:25:31] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-text_eqsin [09:25:40] (03CR) 10Ayounsi: [C:03+1] Allow cumin1004 in one more tcpircbot setting [puppet] - 10https://gerrit.wikimedia.org/r/1334747 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:25:45] (03PS1) 10Marostegui: db1174: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1334749 (https://phabricator.wikimedia.org/T436904) [09:25:45] (03CR) 10Elukey: [C:03+2] sre.hosts.provision: make the FQDN check for SM dynamic [cookbooks] - 10https://gerrit.wikimedia.org/r/1333692 (https://phabricator.wikimedia.org/T419892) (owner: 10Elukey) [09:25:53] RESOLVED: [8x] GnmiInterfaceCountersDrop: cloudsw1-e4-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [09:25:54] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-text_eqiad [09:25:57] !log zabe@deploy1003 zabe: Continuing with deployment [09:26:30] (03CR) 10Marostegui: [C:03+2] db1174: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1334749 (https://phabricator.wikimedia.org/T436904) (owner: 10Marostegui) [09:27:25] FIRING: [4x] ProbeDown: Service aqs1018-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:27:33] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage [09:27:42] FIRING: [2x] JobUnavailable: Reduced availability for job fastnetmon in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:28:30] (03CR) 10Muehlenhoff: [C:03+2] Allow cumin1004 in one more tcpircbot setting [puppet] - 10https://gerrit.wikimedia.org/r/1334747 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:28:45] 06SRE, 10SRE-Access-Requests: Requesting access to Stat host stat1010 for Jose Aleman - https://phabricator.wikimedia.org/T436298#12284585 (10Volans) [09:29:25] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for citoid, cxserver in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1333691 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [09:30:16] 06SRE, 10SRE-Access-Requests: Requesting access to Stat host stat1010 for Jose Aleman - https://phabricator.wikimedia.org/T436298#12284588 (10Volans) p:05Triage→03Medium [09:30:33] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1334743|HookHandler: Fix bail out condition in onLocalUserCreated subscriber (T436791)]] (duration: 09m 30s) [09:30:37] T436791: InvalidArgumentException: Can't persist a key for user with no central ID available - https://phabricator.wikimedia.org/T436791 [09:31:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:32:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.82% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:32:25] FIRING: [4x] ProbeDown: Service aqs1018-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:32:42] FIRING: [3x] JobUnavailable: Reduced availability for job fastnetmon in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:33:27] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-text_eqiad [09:33:49] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-text_ulsfo [09:34:53] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad [09:35:00] (03PS1) 10Volans: admin: convert account for Jose Aleman [puppet] - 10https://gerrit.wikimedia.org/r/1334754 (https://phabricator.wikimedia.org/T436298) [09:35:01] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Changing sanitarium master in s7 [09:35:09] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for citoid, cxserver in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1333691 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [09:35:13] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage [09:35:27] !log Move s7 sanitarium from db1158 to db1273 T434775 [09:35:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:35:30] T434775: Move s6 sanitarium master from db1165 to db1279 - https://phabricator.wikimedia.org/T434775 [09:36:43] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Two VMs for the PKI infra - https://phabricator.wikimedia.org/T436903#12284669 (10MoritzMuehlenhoff) Sounds good to me. How about the disks, looking at pki1002, 40-50G should be perfectly fine and leave ample headroom? [09:37:25] FIRING: [4x] ProbeDown: Service aqs1018-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:37:47] !log ayounsi@cumin1003 START - Cookbook sre.ganeti.makevm for new host netflow2005.codfw.wmnet [09:37:48] !log ayounsi@cumin1003 START - Cookbook sre.dns.netbox [09:38:03] !log Move s7 sanitarium from db1158 to db1273 T434751 [09:38:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:38:06] T434751: Move s7 sanitarium master from db1158 to db1273 - https://phabricator.wikimedia.org/T434751 [09:38:12] (03PS1) 10Marostegui: mariadb: Change s7 sanitarium master [puppet] - 10https://gerrit.wikimedia.org/r/1334759 (https://phabricator.wikimedia.org/T434751) [09:38:34] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [09:39:01] (03CR) 10Marostegui: [C:03+2] mariadb: Change s7 sanitarium master [puppet] - 10https://gerrit.wikimedia.org/r/1334759 (https://phabricator.wikimedia.org/T434751) (owner: 10Marostegui) [09:39:11] !log fixed currently oncall pane in klaxon [09:39:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:39:15] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow2003.codfw.wmnet with reason: host reimage [09:39:32] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [09:39:32] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad [09:40:57] (03PS1) 10Ayounsi: Add netflow2004 and 2005 to Kafka ACL [puppet] - 10https://gerrit.wikimedia.org/r/1334762 (https://phabricator.wikimedia.org/T436517) [09:41:24] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-text_ulsfo [09:41:31] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1018.eqiad.wmnet with OS bookworm [09:41:43] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Stat host stat1010 for Jose Aleman - https://phabricator.wikimedia.org/T436298#12284722 (10Volans) @JAATPH Could you please also [[ https://phabricator.wikimedia.org/settings/panel/external/ | link your LDAP account ]] to your Phabricator... [09:41:50] !log ayounsi@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM netflow2005.codfw.wmnet - ayounsi@cumin1003" [09:41:53] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM netflow2005.codfw.wmnet - ayounsi@cumin1003" [09:41:53] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:41:53] !log ayounsi@cumin1003 START - Cookbook sre.dns.wipe-cache netflow2005.codfw.wmnet on all recursors [09:41:57] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netflow2005.codfw.wmnet on all recursors [09:42:26] !log ayounsi@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM netflow2005.codfw.wmnet - ayounsi@cumin1003" [09:42:31] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM netflow2005.codfw.wmnet - ayounsi@cumin1003" [09:43:25] !log ayounsi@cumin1003 START - Cookbook sre.hosts.reimage for host netflow2005.codfw.wmnet with OS trixie [09:44:29] (03CR) 10Volans: admin: convert account for Jose Aleman (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1334754 (https://phabricator.wikimedia.org/T436298) (owner: 10Volans) [09:45:32] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1334754 (https://phabricator.wikimedia.org/T436298) (owner: 10Volans) [09:47:24] (03PS1) 10Ayounsi: makevm: add --force to reimage cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1334770 [09:47:25] FIRING: [4x] ProbeDown: Service aqs1018-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:52:25] RESOLVED: [4x] ProbeDown: Service aqs1018-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:52:42] FIRING: [3x] JobUnavailable: Reduced availability for job fastnetmon in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:54:02] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1334762 (https://phabricator.wikimedia.org/T436517) (owner: 10Ayounsi) [09:55:13] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2001.codfw.wmnet with OS bookworm [09:55:38] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host netflow2003.codfw.wmnet with OS trixie [09:56:24] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cassandra-dev2001.codfw.wmnet with OS bookworm [09:56:59] moritzm: ^ all good (netflow2003 irc message) [09:57:09] nice! [09:57:40] (03PS1) 10Blake: memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1334775 (https://phabricator.wikimedia.org/T353511) [09:57:42] RESOLVED: [3x] JobUnavailable: Reduced availability for job fastnetmon in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:57:48] (03CR) 10Blake: [C:03+2] memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1334775 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [09:59:09] (03CR) 10JMeybohm: vap: Add support for overriding validationActions (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333256 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1000) [10:00:39] (03PS1) 10Jgiannelos: prv: Enable parsoid rendering for 4 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334777 [10:03:30] (03PS3) 10JMeybohm: validating-admission-policies: Update for k8s 1.34 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333038 (https://phabricator.wikimedia.org/T436380) [10:03:30] (03PS2) 10JMeybohm: validating-admission-policies: Cosmetic changes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333211 (https://phabricator.wikimedia.org/T436380) [10:03:30] (03PS3) 10JMeybohm: vap: Add support for overriding validationActions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333256 (https://phabricator.wikimedia.org/T436380) [10:03:45] !log ayounsi@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on netflow2005.codfw.wmnet with reason: host reimage [10:03:58] (03CR) 10Ladsgroup: [C:03+2] Return early in CoreHandler if no URL match is found [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1334595 (https://phabricator.wikimedia.org/T436897) (owner: 10Samwilson) [10:05:00] (03Merged) 10jenkins-bot: Return early in CoreHandler if no URL match is found [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1334595 (https://phabricator.wikimedia.org/T436897) (owner: 10Samwilson) [10:06:37] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1334723 (https://phabricator.wikimedia.org/T436703) (owner: 10Volans) [10:07:32] (03CR) 10Samwilson: [C:03+1] Route all logging through thumbor's logger [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333247 (owner: 10Ladsgroup-claude) [10:08:37] (03CR) 10Ladsgroup: [C:03+2] Route all logging through thumbor's logger [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333247 (owner: 10Ladsgroup-claude) [10:08:49] (03CR) 10Volans: [C:03+2] "Approved on task" [puppet] - 10https://gerrit.wikimedia.org/r/1333280 (https://phabricator.wikimedia.org/T436703) (owner: 10Kamila Součková) [10:09:02] RECOVERY - LDAP -writable server- on ldap-rw1001 is OK: LDAP OK - 0.007 seconds response time https://wikitech.wikimedia.org/wiki/LDAP%23Troubleshooting [10:09:29] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow2005.codfw.wmnet with reason: host reimage [10:09:30] (03Abandoned) 10Volans: admin: add kamila to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1334723 (https://phabricator.wikimedia.org/T436703) (owner: 10Volans) [10:09:45] (03PS2) 10Volans: admin: convert account for Jose Aleman [puppet] - 10https://gerrit.wikimedia.org/r/1334754 (https://phabricator.wikimedia.org/T436298) [10:10:05] (03PS5) 10Ladsgroup: Tidy up resource and rounding leftovers [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333248 (owner: 10Ladsgroup-claude) [10:10:18] (03CR) 10CI reject: [V:04-1] Tidy up resource and rounding leftovers [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333248 (owner: 10Ladsgroup-claude) [10:10:20] (03PS1) 10Cathal Mooney: gnmic: Disable some less-important subscriptions for JunOS devices [puppet] - 10https://gerrit.wikimedia.org/r/1334781 (https://phabricator.wikimedia.org/T435909) [10:10:49] (03CR) 10Volans: [C:03+2] admin: convert account for Jose Aleman [puppet] - 10https://gerrit.wikimedia.org/r/1334754 (https://phabricator.wikimedia.org/T436298) (owner: 10Volans) [10:11:19] !log Move s8 sanitarium from db1167 to db1281 T434778 [10:11:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:11:23] T434778: Move s8 sanitarium master from db1167 to db1281 - https://phabricator.wikimedia.org/T434778 [10:11:46] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for cloudcephosd1036.mgmt:22 - https://phabricator.wikimedia.org/T436894#12284839 (10Jclark-ctr) 05Open→03Resolved Replaced cable link returned [10:11:49] (03PS6) 10Ladsgroup: Tidy up resource and rounding leftovers [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333248 (owner: 10Ladsgroup-claude) [10:12:18] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Changing sanitarium master in s8 [10:12:33] (03PS7) 10Ladsgroup: Tidy up resource and rounding leftovers [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333248 (owner: 10Ladsgroup-claude) [10:12:58] (03CR) 10Jelto: [V:03+1 C:03+1] "lgtm now, helm template contains the expected warn and audit now:" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333256 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [10:15:02] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Stat host stat1010 for Jose Aleman - https://phabricator.wikimedia.org/T436298#12284849 (10Volans) @JAATPH patch merged, kerberos principal created, you should have received an email about it and how to reset your password. Within ~30 min... [10:15:05] (03PS2) 10Cathal Mooney: gnmic: Disable some less-important subscriptions for JunOS devices [puppet] - 10https://gerrit.wikimedia.org/r/1334781 (https://phabricator.wikimedia.org/T435909) [10:15:15] (03PS1) 10Marostegui: mariadb: Change s8 sanitarium master [puppet] - 10https://gerrit.wikimedia.org/r/1334783 (https://phabricator.wikimedia.org/T434778) [10:15:48] !log btullis@deploy1003 helmfile [staging] START helmfile.d/services/eventgate-analytics: sync [10:15:56] !log btullis@deploy1003 helmfile [staging] DONE helmfile.d/services/eventgate-analytics: sync [10:15:58] (03CR) 10Marostegui: [C:03+2] mariadb: Change s8 sanitarium master [puppet] - 10https://gerrit.wikimedia.org/r/1334783 (https://phabricator.wikimedia.org/T434778) (owner: 10Marostegui) [10:16:01] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host aqs1019.eqiad.wmnet with OS bookworm [10:16:02] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Requesting Kerberos access for kamila - https://phabricator.wikimedia.org/T436703#12284861 (10Volans) 05Open→03Resolved a:03Volans @Raine patch merged, kerberos principal created, you should have received an email about it and... [10:16:04] (03Merged) 10jenkins-bot: Route all logging through thumbor's logger [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333247 (owner: 10Ladsgroup-claude) [10:16:29] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage [10:17:16] !log btullis@deploy1003 helmfile [codfw] START helmfile.d/services/eventgate-analytics: sync [10:17:42] !log btullis@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: sync [10:19:00] (03CR) 10Jelto: [C:03+1] "lgtm" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333038 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [10:20:07] FIRING: [2x] ProbeDown: Service aqs1019-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:20:18] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage [10:23:02] 06SRE, 10LDAP-Access-Requests: Request for LDAP NDA Access for CentralNotice Metrics for Yahya - https://phabricator.wikimedia.org/T436294#12284883 (10Volans) @Aklapper as per https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#Access_Levels basic access to some dataset in turnilo/superset just requi... [10:24:47] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Power alert for cr1-eqiad old line cards - https://phabricator.wikimedia.org/T436814#12284886 (10Jclark-ctr) Removed FPC 4 and 5 and installed blanking panels [10:25:07] FIRING: [4x] ProbeDown: Service aqs1019-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:25:23] (03PS4) 10Giuseppe Lavagetto: aptrepo: add thirdparty/gvisor to newer distros as well [puppet] - 10https://gerrit.wikimedia.org/r/1328540 (https://phabricator.wikimedia.org/T435758) [10:25:23] (03PS3) 10Giuseppe Lavagetto: profile::containerd: format according to our style guide [puppet] - 10https://gerrit.wikimedia.org/r/1330128 [10:25:23] (03PS3) 10Giuseppe Lavagetto: profile::containerd: add gVisor support [puppet] - 10https://gerrit.wikimedia.org/r/1330129 (https://phabricator.wikimedia.org/T435796) [10:25:24] (03PS3) 10Giuseppe Lavagetto: containerd: use hiera parameters to detect if dragonfly is enabled [puppet] - 10https://gerrit.wikimedia.org/r/1330130 [10:25:25] (03PS3) 10Giuseppe Lavagetto: kubernetes::staging: make containerd support gVisor runtime [puppet] - 10https://gerrit.wikimedia.org/r/1330131 (https://phabricator.wikimedia.org/T435796) [10:25:27] (03PS3) 10Giuseppe Lavagetto: profile::kubernetes::node: Add gvisor labels [puppet] - 10https://gerrit.wikimedia.org/r/1330187 (https://phabricator.wikimedia.org/T436212) [10:26:15] (03PS2) 10Btullis: logstash: Stop consuming the webrequest.dumps.dev0 stream from Kafka [puppet] - 10https://gerrit.wikimedia.org/r/1328208 (https://phabricator.wikimedia.org/T425087) [10:26:37] RESOLVED: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [10:28:11] (03CR) 10Gkyziridis: ml-services: Initial deployment of Qwen3.8-27B-FP8 model on experimental ns. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333204 (https://phabricator.wikimedia.org/T436644) (owner: 10Gkyziridis) [10:29:21] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host netflow2005.codfw.wmnet with OS trixie [10:29:21] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host netflow2005.codfw.wmnet [10:30:05] (03CR) 10Gkyziridis: [C:03+2] ml-services: Initial deployment of Qwen3.8-27B-FP8 model on experimental ns. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333204 (https://phabricator.wikimedia.org/T436644) (owner: 10Gkyziridis) [10:32:38] (03Merged) 10jenkins-bot: ml-services: Initial deployment of Qwen3.8-27B-FP8 model on experimental ns. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333204 (https://phabricator.wikimedia.org/T436644) (owner: 10Gkyziridis) [10:33:17] (03PS1) 10Elukey: sre.network.tls: add the intermediate CA cert [cookbooks] - 10https://gerrit.wikimedia.org/r/1334793 (https://phabricator.wikimedia.org/T375513) [10:33:32] mvernon@cumin2003 reimage (PID 3029818) is awaiting input [10:36:53] 06SRE, 07SRE-Unowned, 10Stashbot: Find or invent a method for archiving SAL page messages - https://phabricator.wikimedia.org/T378369#12284967 (10Volans) [10:38:22] FIRING: GnmiInterfaceCountersDrop: ... [10:38:22] lsw1-e1-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-e1-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [10:39:29] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1019.eqiad.wmnet with reason: host reimage [10:40:22] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2001.codfw.wmnet with OS bookworm [10:41:12] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [10:43:22] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1019.eqiad.wmnet with reason: host reimage [10:43:22] RESOLVED: GnmiInterfaceCountersDrop: ... [10:43:23] lsw1-e1-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-e1-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [10:44:00] RECOVERY - LDAP -writable server- on ldap-rw2001 is OK: LDAP OK - 0.104 seconds response time https://wikitech.wikimedia.org/wiki/LDAP%23Troubleshooting [10:46:59] ^ ldap-rw1001/2001 is me testing the trixie/MDB migration, all expected [10:51:11] (03PS2) 10Blake: memcached: continue rollout of PKI use [puppet] - 10https://gerrit.wikimedia.org/r/1334780 (https://phabricator.wikimedia.org/T353511) [10:52:07] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-upload_codfw [10:53:06] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-logging-external in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334801 (https://phabricator.wikimedia.org/T420436) [10:53:09] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-logging-external in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334802 (https://phabricator.wikimedia.org/T420436) [10:53:11] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-analytics* in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334803 (https://phabricator.wikimedia.org/T420436) [10:53:13] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-analytics* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334804 (https://phabricator.wikimedia.org/T420436) [10:53:16] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-main in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334805 (https://phabricator.wikimedia.org/T420436) [10:53:18] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-main in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334806 (https://phabricator.wikimedia.org/T420436) [10:55:23] (03CR) 10Jelto: "We can start with eventgate-logging-external, because it's the smallest eventgate service event- and rps-wise" [puppet] - 10https://gerrit.wikimedia.org/r/1334801 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [10:55:26] PROBLEM - MegaRAID on an-worker1144 is CRITICAL: CRITICAL: 12 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [10:59:12] (03PS1) 10Gerrit maintenance bot: mariadb: Promote db2249 to x1 master [puppet] - 10https://gerrit.wikimedia.org/r/1334811 (https://phabricator.wikimedia.org/T436922) [10:59:30] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-upload_codfw [11:01:04] (03PS1) 10Fabfur: TEST COMMIT PLEASE IGNORE [puppet] - 10https://gerrit.wikimedia.org/r/1334812 [11:01:26] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-upload_drmrs [11:01:59] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1019.eqiad.wmnet with OS bookworm [11:02:12] jouncebot: nowandnext [11:02:12] No deployments scheduled for the next 0 hour(s) and 57 minute(s) [11:02:13] In 0 hour(s) and 57 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1200) [11:02:14] (03CR) 10Slyngshede: "Done" [puppet] - 10https://gerrit.wikimedia.org/r/1325272 (owner: 10Ammarpad) [11:02:21] (03CR) 10Slyngshede: [C:03+1] admin: Add new SSH public key for Ammarpad [puppet] - 10https://gerrit.wikimedia.org/r/1325272 (owner: 10Ammarpad) [11:02:24] (03CR) 10Clément Goubert: [C:03+2] mediawiki: enable forward of fatal metrics to statsd exporter [puppet] - 10https://gerrit.wikimedia.org/r/1049625 (https://phabricator.wikimedia.org/T356814) (owner: 10Cwhite) [11:02:36] (03CR) 10Slyngshede: [C:03+2] admin: Add new SSH public key for Ammarpad [puppet] - 10https://gerrit.wikimedia.org/r/1325272 (owner: 10Ammarpad) [11:02:40] (03PS1) 10Gerrit maintenance bot: mariadb: Promote db1278 to x1 master [puppet] - 10https://gerrit.wikimedia.org/r/1334813 (https://phabricator.wikimedia.org/T436923) [11:02:46] (03PS1) 10Gerrit maintenance bot: wmnet: Update x1-master alias [dns] - 10https://gerrit.wikimedia.org/r/1334814 (https://phabricator.wikimedia.org/T436923) [11:03:16] (03PS2) 10Fabfur: TEST COMMIT PLEASE IGNORE [puppet] - 10https://gerrit.wikimedia.org/r/1334812 [11:07:24] (03PS2) 10Jgiannelos: prv: Enable parsoid rendering for 4 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334777 (https://phabricator.wikimedia.org/T436919) [11:07:34] (03CR) 10Fabfur: [C:04-1] TEST COMMIT PLEASE IGNORE (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1334812 (owner: 10Fabfur) [11:08:00] (03CR) 10Fabfur: TEST COMMIT PLEASE IGNORE (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1334812 (owner: 10Fabfur) [11:08:42] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 07 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329314 (https://phabricator.wikimedia.org/T433483) (owner: 10Abijeet Patro) [11:09:12] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-upload_drmrs [11:10:02] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-upload_esams [11:10:07] RESOLVED: [4x] ProbeDown: Service aqs1019-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:10:22] (03CR) 10Fabfur: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1334812 (owner: 10Fabfur) [11:12:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:14:17] !log cgoubert@deploy1003 Started scap sync-world: mediawiki: enable forward of fatal metrics to statsd exporter [11:14:48] (03Abandoned) 10Fabfur: TEST COMMIT PLEASE IGNORE [puppet] - 10https://gerrit.wikimedia.org/r/1334812 (owner: 10Fabfur) [11:15:26] !log cgoubert@deploy1003 cgoubert: mediawiki: enable forward of fatal metrics to statsd exporter synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:17:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:17:40] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-upload_esams [11:18:09] Is Gerrit having issues or is my home internet somehow borked again (totally plausible)? I see to no longer able to connect to Gerrit with git, but an ongoing ping to gerrit.wikimedia.org looks fine (but maybe only hitting the CDN?) [11:18:23] *seem [11:18:46] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-upload_eqsin [11:22:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:22:55] !log cgoubert@deploy1003 cgoubert: Continuing with deployment [11:23:09] (03PS1) 10Marostegui: db2207: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1334822 [11:23:19] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed again - https://phabricator.wikimedia.org/T436743#12285210 (10Jclark-ctr) a:05Marostegui→03Jclark-ctr @Marostegui Updated the following firmware: BIOS: 1.21.1 → 1.22.1 Backplane: 7.10 → 7.16 PERC H755: 52.21.0-4606 → 52.38.0-6897 SSD firmware:... [11:23:53] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed again - https://phabricator.wikimedia.org/T436743#12285215 (10Marostegui) Thanks! [11:23:54] (03CR) 10Marostegui: [C:03+2] db2207: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1334822 (owner: 10Marostegui) [11:24:25] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db2207: db2207 repool [11:24:58] !log cgoubert@deploy1003 Finished scap sync-world: mediawiki: enable forward of fatal metrics to statsd exporter (duration: 12m 01s) [11:26:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:26:39] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-upload_eqsin [11:30:03] FIRING: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:32:45] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host aqs1020.eqiad.wmnet with OS bookworm [11:33:22] FIRING: [2x] GnmiInterfaceCountersDrop: lsw1-a4-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [11:34:25] (03PS1) 10Jforrester: Periodic jobs: Add mainstash_metrics [puppet] - 10https://gerrit.wikimedia.org/r/1334836 (https://phabricator.wikimedia.org/T430940) [11:35:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:35:42] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-upload_eqiad [11:36:33] FIRING: [3x] ProbeDown: Service aqs1020-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:39:55] (03PS1) 10Muehlenhoff: Create a separate role openldap::rw_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1334839 (https://phabricator.wikimedia.org/T331699) [11:40:19] (03PS8) 10Ladsgroup-claude: Tidy up resource and rounding leftovers [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333248 [11:40:19] (03PS4) 10Ladsgroup-claude: Replace python-memcached with pymemcache [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333249 [11:40:19] (03PS5) 10Ladsgroup-claude: Call ImageMagick 7's magick instead of the deprecated convert [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333250 [11:40:20] (03PS4) 10Ladsgroup-claude: Clean up requirements and add a dependency bump path [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333251 [11:40:21] (03PS6) 10Ladsgroup-claude: Add a unit test layer that runs without the container [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333252 [11:40:22] (03PS4) 10Ladsgroup-claude: Replace flake8 with ruff and apply its fixes [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 [11:40:26] (03PS3) 10Ladsgroup-claude: Apply ruff format [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333254 [11:40:33] (03CR) 10CI reject: [V:04-1] Create a separate role openldap::rw_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1334839 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [11:41:33] FIRING: [6x] ProbeDown: Service aqs1020-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:42:42] (03PS1) 10Marostegui: instances.yaml: Add db1282 [puppet] - 10https://gerrit.wikimedia.org/r/1334843 (https://phabricator.wikimedia.org/T407942) [11:42:59] (03PS3) 10Clément Goubert: cxserver: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333172 (https://phabricator.wikimedia.org/T429175) [11:43:26] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1282 [puppet] - 10https://gerrit.wikimedia.org/r/1334843 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [11:43:27] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-upload_eqiad [11:44:08] (03PS2) 10Muehlenhoff: Create a separate role openldap::rw_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1334839 (https://phabricator.wikimedia.org/T331699) [11:44:45] (03CR) 10CI reject: [V:04-1] Create a separate role openldap::rw_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1334839 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [11:45:20] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting Kerberos access for kamila - https://phabricator.wikimedia.org/T436703#12285310 (10Raine) Working, awesome sauce, thanks! [11:45:26] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1282 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96328 and previous config saved to /var/cache/conftool/dbconfig/20260903-114526-marostegui.json [11:45:30] T407942: Productionize db12[65-90] - https://phabricator.wikimedia.org/T407942 [11:45:36] (03CR) 10KartikMistry: [C:03+2] cxserver: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333172 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [11:46:02] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1282: Pooling db1282 into s6 [11:47:22] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1020.eqiad.wmnet with reason: host reimage [11:48:36] (03Merged) 10jenkins-bot: cxserver: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333172 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [11:49:46] (03PS3) 10Muehlenhoff: Create a separate role openldap::rw_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1334839 (https://phabricator.wikimedia.org/T331699) [11:50:06] I'll deploy cxserver. [11:51:04] (03CR) 10JMeybohm: [C:03+2] vap: Add support for overriding validationActions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333256 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [11:51:09] (03CR) 10JMeybohm: [C:03+2] validating-admission-policies: Cosmetic changes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333211 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [11:51:14] (03CR) 10JMeybohm: [C:03+2] validating-admission-policies: Update for k8s 1.34 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333038 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [11:51:33] FIRING: [4x] ProbeDown: Service aqs1020-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:51:50] !log kartik@deploy1003 helmfile [staging] START helmfile.d/services/cxserver: apply [11:52:01] !log kartik@deploy1003 helmfile [staging] DONE helmfile.d/services/cxserver: apply [11:54:55] (03CR) 10Btullis: [C:03+2] logstash: Stop consuming the webrequest.dumps.dev0 stream from Kafka [puppet] - 10https://gerrit.wikimedia.org/r/1328208 (https://phabricator.wikimedia.org/T425087) (owner: 10Btullis) [11:54:56] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1020.eqiad.wmnet with reason: host reimage [11:55:15] !log kartik@deploy1003 helmfile [codfw] START helmfile.d/services/cxserver: apply [11:55:42] !log installing rsync security updates [11:55:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:55:44] !log kartik@deploy1003 helmfile [codfw] DONE helmfile.d/services/cxserver: apply [11:55:57] (03PS1) 10Santiago Faci: Deploy GrowthBook 5.0.0 to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334845 (https://phabricator.wikimedia.org/T434257) [11:56:33] FIRING: [4x] ProbeDown: Service aqs1020-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:56:46] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-upload_magru [11:56:49] !log kartik@deploy1003 helmfile [eqiad] START helmfile.d/services/cxserver: apply [11:57:16] !log kartik@deploy1003 helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [11:58:40] !log cxserver: Use urldownloader LVS endpoint (T429175) [11:58:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:58:43] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [12:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1200) [12:01:27] (03Merged) 10jenkins-bot: validating-admission-policies: Update for k8s 1.34 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333038 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [12:01:33] FIRING: [4x] ProbeDown: Service aqs1020-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:01:59] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12285377 (10elukey) @Jclark-ctr the new change is merged, I was able to provision 1055 and 1056, but 1053/54 seem not reachable yet so I am not sure how to... [12:02:13] (03Merged) 10jenkins-bot: validating-admission-policies: Cosmetic changes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333211 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [12:02:16] (03Merged) 10jenkins-bot: vap: Add support for overriding validationActions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333256 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [12:02:16] (03PS1) 10Btullis: 7za: Use cgroups to determine the maximum RAM and number of threads [dumps] - 10https://gerrit.wikimedia.org/r/1334846 (https://phabricator.wikimedia.org/T436913) [12:03:04] (03CR) 10Elukey: [C:03+1] Create a separate role openldap::rw_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1334839 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [12:03:49] (03CR) 10CI reject: [V:04-1] 7za: Use cgroups to determine the maximum RAM and number of threads [dumps] - 10https://gerrit.wikimedia.org/r/1334846 (https://phabricator.wikimedia.org/T436913) (owner: 10Btullis) [12:04:10] (03PS1) 10Jforrester: wikifunctions: Revert orchestrator to 2026-08-19-165650 due to Z12 stripping fallout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334849 [12:04:34] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-upload_magru [12:05:31] (03CR) 10Phuedx: [C:03+1] Deploy GrowthBook 5.0.0 to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334845 (https://phabricator.wikimedia.org/T434257) (owner: 10Santiago Faci) [12:06:45] (03PS2) 10Btullis: 7za: Use cgroups to determine the maximum RAM and number of threads [dumps] - 10https://gerrit.wikimedia.org/r/1334846 (https://phabricator.wikimedia.org/T436913) [12:07:28] !log slyngshede@cumin1003 START - Cookbook sre.cdn.roll-restart-purged rolling restart_daemons on A:cp-upload_ulsfo [12:08:13] (03CR) 10CI reject: [V:04-1] 7za: Use cgroups to determine the maximum RAM and number of threads [dumps] - 10https://gerrit.wikimedia.org/r/1334846 (https://phabricator.wikimedia.org/T436913) (owner: 10Btullis) [12:08:28] (03PS3) 10Hashar: Change update to exactly match the given image name [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1166856 [12:08:35] (03CR) 10CI reject: [V:04-1] Change update to exactly match the given image name [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1166856 (owner: 10Hashar) [12:09:52] (03CR) 10Muehlenhoff: [C:03+2] Create a separate role openldap::rw_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1334839 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [12:10:18] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: db2207 repool [12:11:23] btullis: I think you have a puppet-merge run waiting for your confirmation? [12:12:05] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1020.eqiad.wmnet with OS bookworm [12:13:05] (03PS4) 10Hashar: Change update to exactly match the given image name [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1166856 [12:13:44] (03CR) 10Trueg: [C:04-1] "since the url is the same for eqiad and codfw, we can move the values to the chart or at the least the base values.yaml" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333174 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [12:15:05] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-purged (exit_code=0) rolling restart_daemons on A:cp-upload_ulsfo [12:19:42] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [12:21:33] FIRING: [4x] ProbeDown: Service aqs1020-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:21:42] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [12:21:55] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [12:21:57] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [12:22:40] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1334856 (owner: 10L10n-bot) [12:23:48] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Power alert for cr1-eqiad old line cards - https://phabricator.wikimedia.org/T436814#12285481 (10Jclark-ctr) 05Open→03Resolved [12:25:15] !log filippo@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1065.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [12:26:33] RESOLVED: [4x] ProbeDown: Service aqs1020-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:27:45] (03PS1) 10Arnaudb: gitlab: let the envoy tlsproxy relay websocket upgrades [puppet] - 10https://gerrit.wikimedia.org/r/1334854 (https://phabricator.wikimedia.org/T425441) [12:28:04] (03CR) 10Arnaudb: [C:03+2] gitlab: let the envoy tlsproxy relay websocket upgrades [puppet] - 10https://gerrit.wikimedia.org/r/1334854 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [12:29:26] (03PS1) 10Arnaudb: trafficserver: map websockets for the gitlab instances [puppet] - 10https://gerrit.wikimedia.org/r/1334855 (https://phabricator.wikimedia.org/T425441) [12:30:21] !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [12:30:22] FIRING: CertAlmostExpired: gNMI TLS certificate for ssw1-d1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:30:33] !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [12:30:54] !log jayme@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [12:31:00] !log jayme@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [12:31:10] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285520 (10fgiunchedi) >>! In T435921#12282448, @VRiley-WMF wrote: > @fgiunchedi while trying to image the first server, I think it may be getting stuck on the RA... [12:31:16] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1282: Pooling db1282 into s6 [12:33:35] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1065.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [12:34:13] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1065.eqiad.wmnet with OS trixie [12:34:19] 06SRE, 10Pywikibot, 06Traffic, 10Wikidata, and 4 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12285533 (10DL2204) Over the last days, I constantly get maxlag errors, mostly when attempting to get items using wikibaseintegrator, but also (to a lesser e... [12:38:23] !log slyngshede@cumin1003 START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-magru and A:durum [12:39:13] (03CR) 10Santiago Faci: [C:03+2] Deploy GrowthBook 5.0.0 to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334845 (https://phabricator.wikimedia.org/T434257) (owner: 10Santiago Faci) [12:40:02] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-magru and A:durum [12:41:13] !log slyngshede@cumin1003 START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-ulsfo and A:durum [12:41:17] (03Merged) 10jenkins-bot: Deploy GrowthBook 5.0.0 to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334845 (https://phabricator.wikimedia.org/T434257) (owner: 10Santiago Faci) [12:42:29] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-ulsfo and A:durum [12:45:30] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply [12:45:51] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply [12:48:37] PROBLEM - SSH on an-druid1006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [12:49:29] PROBLEM - Druid historical on an-druid1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [12:49:39] !log jayme@deploy1003 helmfile [eqiad] START helmfile.d/admin 'sync'. [12:49:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:49:43] !log jayme@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'sync'. [12:49:52] !log jayme@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [12:50:01] !log jayme@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [12:50:02] !log fceratto@cumin1003 START - Cookbook sre.mysql.pool pool db2901: Test [12:50:04] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2901: Test [12:50:10] !log jayme@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'sync'. [12:50:15] !log jayme@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. [12:50:24] !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'sync'. [12:50:35] !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [12:50:43] !log jayme@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [12:50:49] !log jayme@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [12:50:54] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [12:50:58] !log jayme@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [12:51:11] !log jayme@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [12:51:20] !log jayme@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [12:51:32] !log jayme@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [12:51:33] !log fceratto@cumin1003 START - Cookbook sre.mysql.pool pool db2901: Test [12:51:41] !log jayme@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [12:51:46] !log jayme@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [12:51:54] !log jayme@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. [12:52:05] !log jayme@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. [12:52:12] jclark@cumin1003 reimage (PID 2442844) is awaiting input [12:52:12] !log jayme@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. [12:52:18] !log jayme@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. [12:52:25] !log jayme@deploy1003 helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [12:52:36] !log jayme@deploy1003 helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [12:52:48] !log fceratto@cumin1003 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Test [12:52:55] !log fceratto@cumin1003 START - Cookbook sre.mysql.depool depool db2901: Test [12:52:58] !log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Test [12:53:10] !log fceratto@cumin1003 START - Cookbook sre.mysql.pool pool db2901: Test [12:53:59] (03PS1) 10Blake: memcached: continue rollout of PKI use [puppet] - 10https://gerrit.wikimedia.org/r/1334872 (https://phabricator.wikimedia.org/T353511) [12:54:32] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1334872 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [12:54:43] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [12:55:59] !log fceratto@cumin1003 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Test [12:56:15] !log fceratto@cumin1003 START - Cookbook sre.mysql.pool pool db2901: Test [12:56:47] !log fceratto@cumin1003 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Test [12:56:54] !log fceratto@cumin1003 START - Cookbook sre.mysql.pool pool db2901: Test [12:57:59] Test! [12:58:37] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1066] - vriley@cumin1003" [12:58:51] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1066] - vriley@cumin1003" [12:58:51] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:59:02] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1066 [12:59:18] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1066 [12:59:26] (03CR) 10Blake: [C:03+2] memcached: continue rollout of PKI use [puppet] - 10https://gerrit.wikimedia.org/r/1334872 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [12:59:40] (03CR) 10Jforrester: [C:03+2] wikifunctions: Revert orchestrator to 2026-08-19-165650 due to Z12 stripping fallout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334849 (owner: 10Jforrester) [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: It is that lovely time of the day again! You are hereby commanded to deploy UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1300). [13:00:05] Mahmoud-WMDE: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:48] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1066.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:02:08] (03Merged) 10jenkins-bot: wikifunctions: Revert orchestrator to 2026-08-19-165650 due to Z12 stripping fallout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334849 (owner: 10Jforrester) [13:02:29] RECOVERY - Druid historical on an-druid1007 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:04:16] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [13:04:19] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [13:04:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:26] Hey @joucebot I'm here [13:04:26] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [13:04:35] (03CR) 10Jelto: [C:03+2] "note: The etherpad-next record should point at a dedicated external/non-sre ingress when a second ingress is available" [dns] - 10https://gerrit.wikimedia.org/r/1331613 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [13:04:42] !log jforrester@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [13:05:13] !log jforrester@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [13:05:20] !log jforrester@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [13:06:01] !log jforrester@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [13:06:06] !log jelto@dns1004 START - running authdns-update [13:08:38] !log jelto@dns1004 END - running authdns-update [13:08:53] (03PS1) 10Muehlenhoff: Use nftables for openldap::rw_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1334883 (https://phabricator.wikimedia.org/T331699) [13:09:31] !log installing libxslt bugfix updates from Trixie point release [13:09:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:09:41] vriley@cumin1003 netbox (PID 2459886) is awaiting input [13:09:56] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:10:57] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1066.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:11:21] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1067] - vriley@cumin1003" [13:11:25] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1067] - vriley@cumin1003" [13:11:25] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:11:56] !log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2901: Test [13:13:00] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1067 [13:13:19] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1067 [13:13:39] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1066.eqiad.wmnet with OS trixie [13:13:42] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.5 point update - https://phabricator.wikimedia.org/T427072#12285794 (10MoritzMuehlenhoff) [13:13:51] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285795 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1066.eqiad.wmnet with OS trixie [13:14:08] !log andrew@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [13:14:13] !log installing bash updates from trixie point release [13:14:14] !log andrew@cumin2003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [13:14:25] !log andrew@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [13:14:34] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:14:34] !log andrew@cumin2003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [13:14:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:14:38] Mahmoud-WMDE: do you still need a deployer? [13:14:56] TheresNoTime yes indeed! [13:15:14] I will do so now then :) [13:15:22] Thanks! [13:15:33] !log andrew@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [13:15:37] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [extensions/Wikibase] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333866 (https://phabricator.wikimedia.org/T434204) (owner: 10Thiemo Kreuz (WMDE)) [13:15:37] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:15:49] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:15:56] !log andrew@cumin2003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [13:16:30] I can’t deploy today, sorry [13:16:35] thanks TheresNoTime! [13:17:42] (03PS1) 10Blake: memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334888 [13:17:45] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1334888 (owner: 10Blake) [13:18:07] (03CR) 10CI reject: [V:04-1] memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334888 (owner: 10Blake) [13:18:26] (03PS2) 10Blake: memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334888 (https://phabricator.wikimedia.org/T353511) [13:18:29] no worries :) (just waiting on it to merge) [13:18:35] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1334888 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:20:42] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:22:04] !log cdobbins@cumin1003 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams [13:22:38] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm [13:22:46] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12285823 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1003 for host cloudcephosd1055.eqiad.wmnet with OS bookworm [13:23:26] (03CR) 10Elukey: [C:03+1] Use nftables for openldap::rw_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1334883 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [13:24:16] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:24:26] !log ayounsi@cumin1004 START - Cookbook sre.hosts.reimage for host netflow2004.codfw.wmnet with OS trixie [13:24:28] (03PS3) 10Blake: memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334888 (https://phabricator.wikimedia.org/T353511) [13:24:38] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1334888 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:26:35] (03CR) 10Blake: [C:03+2] memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334888 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:26:47] (03CR) 10Ayounsi: [C:03+1] gnmic: Disable some less-important subscriptions for JunOS devices [puppet] - 10https://gerrit.wikimedia.org/r/1334781 (https://phabricator.wikimedia.org/T435909) (owner: 10Cathal Mooney) [13:27:20] elukey@cumin1003 provision (PID 2467734) is awaiting input [13:27:29] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:28:01] (03CR) 10Ssingh: [C:03+1] "I haven't verified the issue on the k8s side but this at least in theory looks good." [puppet] - 10https://gerrit.wikimedia.org/r/1334855 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [13:28:23] FIRING: [2x] GnmiInterfaceCountersDrop: lsw1-a4-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [13:28:27] (03Merged) 10jenkins-bot: Remove revisionId from term fallback cache lines for properties [extensions/Wikibase] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333866 (https://phabricator.wikimedia.org/T434204) (owner: 10Thiemo Kreuz (WMDE)) [13:28:38] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1067.eqiad.wmnet with OS trixie [13:28:50] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12285875 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1067.eqiad.wmnet with OS trixie [13:28:53] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1333866|Remove revisionId from term fallback cache lines for properties (T434204)]] [13:28:56] T434204: [SPIKE] Determine how to invalidate property term cache entries on edit - https://phabricator.wikimedia.org/T434204 [13:29:42] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285878 (10VRiley-WMF) [13:31:25] !log slyngshede@cumin1003 START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-eqsin and A:durum [13:31:27] RECOVERY - SSH on an-druid1006 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [13:32:09] (03PS2) 10Elukey: sre.network.tls: add the intermediate CA cert [cookbooks] - 10https://gerrit.wikimedia.org/r/1334793 (https://phabricator.wikimedia.org/T375513) [13:32:27] (03PS1) 10Filippo Giunchedi: site: fix insetup_noferm role name for cloudvirt [puppet] - 10https://gerrit.wikimedia.org/r/1334892 (https://phabricator.wikimedia.org/T435921) [13:32:55] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-eqsin and A:durum [13:33:37] !log samtar@deploy1003 samtar, thiemowmde: Backport for [[gerrit:1333866|Remove revisionId from term fallback cache lines for properties (T434204)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:33:54] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [13:34:06] Mahmoud-WMDE: changes ready for testing on mwdebug ^ [13:34:26] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:34:28] TheresNoTime: Sure, I will start testing now [13:35:04] (03CR) 10Filippo Giunchedi: [C:03+2] "Trivial typo from yours truly, self-merging" [puppet] - 10https://gerrit.wikimedia.org/r/1334892 (https://phabricator.wikimedia.org/T435921) (owner: 10Filippo Giunchedi) [13:35:23] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:35:53] !log slyngshede@cumin1003 START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-drmrs and A:durum [13:35:59] (03PS1) 10Blake: memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334893 (https://phabricator.wikimedia.org/T353511) [13:36:42] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285914 (10fgiunchedi) [13:37:19] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-drmrs and A:durum [13:37:40] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285925 (10fgiunchedi) >>! In T435921#12282448, @VRiley-WMF wrote: > @fgiunchedi while trying to image the first server, I think it may be getting stuck on the RA... [13:37:42] !log slyngshede@cumin1003 START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-esams and A:durum [13:37:45] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1072] - vriley@cumin1003" [13:37:49] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1072] - vriley@cumin1003" [13:37:49] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:37:54] TheresNoTime: That worked, the implementation is working as expected! and the db read queries has been reduced for the revisionId. [13:37:59] ack :) [13:38:03] !log samtar@deploy1003 samtar, thiemowmde: Continuing with deployment [13:38:15] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:38:22] RESOLVED: [2x] GnmiInterfaceCountersDrop: lsw1-a4-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [13:38:53] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1072 [13:38:53] 06SRE, 06Traffic, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12285930 (10Bewfip) →14Duplicate dup:03T436001 [13:38:56] * TheresNoTime will never not read "thiemowmde" as "the meow de" [13:39:02] 06SRE, 06Traffic, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12285935 (10Bewfip) [13:39:31] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-esams and A:durum [13:40:08] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage [13:40:24] (03CR) 10Blake: [C:03+2] memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334893 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:40:28] :-D [13:40:41] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1072 [13:40:44] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285939 (10VRiley-WMF) Understood. It's strange because when I log into the console, it seems like it finishes (went to the login screen) [13:41:15] TheresNoTime: Is there anything else needed from my side? :) [13:41:15] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1072.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:41:50] (03PS1) 10Elukey: sre.hosts.provision: test [cookbooks] - 10https://gerrit.wikimedia.org/r/1334894 [13:41:53] Mahmoud-WMDE: not really, unless you want to test again once its finished syncing to prod? [13:42:06] TheresNoTime: you did not have to put that in my head [13:42:22] taavi: :D [13:42:43] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1333866|Remove revisionId from term fallback cache lines for properties (T434204)]] (duration: 13m 50s) [13:42:47] T434204: [SPIKE] Determine how to invalidate property term cache entries on edit - https://phabricator.wikimedia.org/T434204 [13:42:54] Mahmoud-WMDE: live on prod now :) [13:42:55] taavi: :-D [13:43:03] !log ayounsi@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on netflow2004.codfw.wmnet with reason: host reimage [13:43:21] TheresNoTime: Great, I will retest now! [13:43:44] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:43:54] (03PS2) 10Btullis: Remove the webrequest.dumps.dev0 stream from EventStreamConfig [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328202 (https://phabricator.wikimedia.org/T425087) [13:44:00] !log vriley@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1067.eqiad.wmnet with reason: host reimage [13:44:13] TheresNoTime: which k8s-* should I use for prod testing in WikimediaDebug? [13:44:25] (03CR) 10Ayounsi: [C:03+1] sre.network.tls: add the intermediate CA cert [cookbooks] - 10https://gerrit.wikimedia.org/r/1334793 (https://phabricator.wikimedia.org/T375513) (owner: 10Elukey) [13:44:32] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage [13:44:53] Mahmoud-WMDE: you should just be able to test with WikimediaDebug switched off now [13:45:14] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 03 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328202 (https://phabricator.wikimedia.org/T425087) (owner: 10Btullis) [13:45:33] vriley@cumin1003 provision (PID 2469386) is awaiting input [13:46:45] TheresNoTime: I might need it on to check the DB read queries for certain functions in https://performance.wikimedia.org/xhgui/run/view?id=6a9977775894b470430d6c10 .. any idea how? [13:47:02] !log cdobbins@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams [13:47:47] !log slyngshede@cumin1003 START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-codfw and A:durum [13:47:58] (03PS1) 10Blake: memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334897 (https://phabricator.wikimedia.org/T353511) [13:48:01] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1334897 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:48:14] TheresNoTime: I just scheduled another mediawiki-config patch, but I realise that it is the currently active window. is this bad form? Should I have made it the later backport window instead? [13:48:26] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1067.eqiad.wmnet with reason: host reimage [13:48:59] Mahmoud-WMDE: I'm not sure with that, sorry - does just checking the `XHGui` box in wikimediadebug do anything? [13:49:11] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-codfw and A:durum [13:49:34] TheresNoTime: Unfortunately not, it has to be ON .. no worries, I'll figure it out [13:49:34] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.5 point update - https://phabricator.wikimedia.org/T427072#12285986 (10MoritzMuehlenhoff) [13:49:45] TheresNoTime: Thanks for your efforts! [13:49:48] btullis: I can deploy that now if you're around for it? I'd say that's fine (adding something to a current window) if you also ping the deployer [13:49:55] Mahmoud-WMDE: good luck! :) [13:50:44] (03CR) 10Blake: [C:03+2] memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334897 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:51:19] !log slyngshede@cumin1003 START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-eqiad and A:durum [13:51:23] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow2004.codfw.wmnet with reason: host reimage [13:51:52] !log installing sqlite3 security updates [13:51:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:52:05] (03PS2) 10Elukey: sre.hosts.provision: avoid user locked out for Supermicro BMCs [cookbooks] - 10https://gerrit.wikimedia.org/r/1334894 (https://phabricator.wikimedia.org/T419892) [13:52:26] (03PS3) 10Btullis: 7za: Use cgroups to determine the maximum RAM and number of threads [dumps] - 10https://gerrit.wikimedia.org/r/1334846 (https://phabricator.wikimedia.org/T436913) [13:52:34] !log cdobbins@cumin1003 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw [13:52:57] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-eqiad and A:durum [13:54:35] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:55:04] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:55:19] (03PS4) 10Btullis: 7za: Use cgroups to determine the maximum RAM and number of threads [dumps] - 10https://gerrit.wikimedia.org/r/1334846 (https://phabricator.wikimedia.org/T436913) [13:55:42] TheresNoTime: Yes, I'm around. Many thanks. [13:56:03] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328202 (https://phabricator.wikimedia.org/T425087) (owner: 10Btullis) [13:56:58] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003" [13:57:13] I will be testing this URL https://meta.wikimedia.org/w/api.php?action=streamconfigs&streams=webrequest.dumps.dev0 with the WikimediaDebug extension enabled and verifying that the eventstream is removed successfully. [13:57:33] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1072.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:57:57] (03Merged) 10jenkins-bot: Remove the webrequest.dumps.dev0 stream from EventStreamConfig [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328202 (https://phabricator.wikimedia.org/T425087) (owner: 10Btullis) [13:58:22] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1328202|Remove the webrequest.dumps.dev0 stream from EventStreamConfig (T425087)]] [13:58:25] T425087: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087 [14:00:02] filippo@cumin1003 reimage (PID 2441832) is awaiting input [14:00:34] (03CR) 10Muehlenhoff: [C:03+2] Use nftables for openldap::rw_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1334883 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [14:01:30] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003" [14:01:30] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1065.eqiad.wmnet with OS trixie [14:01:32] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1072.eqiad.wmnet with OS trixie [14:01:52] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12286061 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1072.eqiad.wmnet with OS trixie [14:02:08] TheresNoTime: My test works. Please proceed with the deploy when convenient. [14:02:40] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286074 (10fgiunchedi) >>! In T435921#12285939, @VRiley-WMF wrote: > Understood. It's strange because when I log into the console, it seems like it finishes (went... [14:02:45] !log samtar@deploy1003 btullis, samtar: Backport for [[gerrit:1328202|Remove the webrequest.dumps.dev0 stream from EventStreamConfig (T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:02:52] (ack) [14:03:14] !log samtar@deploy1003 btullis, samtar: Continuing with deployment [14:03:35] !log slyngshede@cumin1003 START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox) [14:04:00] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286079 (10VRiley-WMF) Gotcha, no problem. It looks like 1067 went through (it's still finishing up) but I will rerun it. Thank you! [14:04:52] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [14:05:37] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [14:05:51] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [14:05:53] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1067.eqiad.wmnet with OS trixie [14:06:03] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [14:06:06] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12286086 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1067.eqiad.wmnet with OS trixie completed: -... [14:06:26] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286089 (10VRiley-WMF) [14:07:41] (03PS1) 10Muehlenhoff: Reimage ldap-rw[12]001 [puppet] - 10https://gerrit.wikimedia.org/r/1334906 (https://phabricator.wikimedia.org/T331699) [14:07:58] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1328202|Remove the webrequest.dumps.dev0 stream from EventStreamConfig (T425087)]] (duration: 09m 36s) [14:08:01] T425087: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087 [14:08:08] btullis: live on prod :) [14:08:43] !log elukey@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm [14:08:56] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm [14:08:59] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12286102 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1003 for host cloudcephosd1055.eqiad.wmne... [14:09:10] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12286103 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1003 for host cloudcephosd1055.eqiad.... [14:09:21] TheresNoTime: Many thanks once again. [14:10:42] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [14:10:48] np! [14:11:10] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1073] - vriley@cumin1003" [14:11:14] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1073] - vriley@cumin1003" [14:11:14] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:12:19] !log ayounsi@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host netflow2004.codfw.wmnet with OS trixie [14:13:28] (03PS1) 10Arnaudb: Revert^3 "gitlab: discard firewall throttling on the primary" [puppet] - 10https://gerrit.wikimedia.org/r/1334912 (https://phabricator.wikimedia.org/T425441) [14:13:56] (03PS1) 10Arnaudb: Revert^3 "gitlab: point gitlab.wikimedia.org at the CDN" [dns] - 10https://gerrit.wikimedia.org/r/1334914 (https://phabricator.wikimedia.org/T425441) [14:15:15] (03CR) 10Arnaudb: [C:03+2] Revert^3 "gitlab: discard firewall throttling on the primary" [puppet] - 10https://gerrit.wikimedia.org/r/1334912 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [14:15:56] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1073 [14:16:03] (03PS1) 10Blake: memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334910 (https://phabricator.wikimedia.org/T353511) [14:16:07] (03CR) 10Blake: [C:03+2] memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334910 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [14:16:13] vriley@cumin1003 netbox (PID 2476049) is awaiting input [14:16:14] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1073 [14:16:27] !log vriley@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1072.eqiad.wmnet with reason: host reimage [14:16:30] !log elukey@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm [14:16:45] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12286125 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1003 for host cloudcephosd1055.eqiad.wmne... [14:16:53] (03CR) 10Arnaudb: [C:03+2] Revert^3 "gitlab: point gitlab.wikimedia.org at the CDN" [dns] - 10https://gerrit.wikimedia.org/r/1334914 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [14:16:58] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1073.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:17:03] !log arnaudb@dns1006 START - running authdns-update [14:17:23] !log cdobbins@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw [14:17:40] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1074] - vriley@cumin1003" [14:17:44] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1074] - vriley@cumin1003" [14:17:44] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:18:04] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1074 [14:18:20] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1074 [14:18:54] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1074.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:19:38] !log arnaudb@dns1006 END - running authdns-update [14:21:03] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1072.eqiad.wmnet with reason: host reimage [14:21:12] (03CR) 10Filippo Giunchedi: [C:03+1] openstack: puppet-enc: Allow MEDIUMTEXT worth of Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1332802 (https://phabricator.wikimedia.org/T436561) (owner: 10Majavah) [14:21:44] (03PS1) 10Muehlenhoff: conftool::conftool2git: Switch to firewall::service [puppet] - 10https://gerrit.wikimedia.org/r/1334921 [14:22:37] !log elukey@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cassandra-dev2002.codfw.wmnet [14:22:42] (03CR) 10Majavah: [C:03+2] openstack: puppet-enc: Allow MEDIUMTEXT worth of Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1332802 (https://phabricator.wikimedia.org/T436561) (owner: 10Majavah) [14:24:32] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12286153 (10Geertivp) There is some technical inconsistecy within the system. I am registered as admin on the platform but this user account does not physically exist, so I can't login. Verifie... [14:26:08] (03PS2) 10Muehlenhoff: conftool::conftool2git: Switch to firewall::service [puppet] - 10https://gerrit.wikimedia.org/r/1334921 [14:27:13] !log cdobbins@cumin1003 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo [14:29:22] !log sudo cumin "A:cp-text" "disable-puppet 'merging CR 1334855'" T425441 [14:29:23] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1073.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:29:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:29:25] T425441: gitlab behind CDN - https://phabricator.wikimedia.org/T425441 [14:29:43] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1066.eqiad.wmnet with OS trixie [14:30:00] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286173 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1066.eqiad.wmnet with OS trixie executed with er... [14:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1430) [14:30:08] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1334921 (owner: 10Muehlenhoff) [14:30:10] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1066.eqiad.wmnet with OS trixie [14:30:28] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1066.eqiad.wmnet with OS trixie [14:31:06] (03CR) 10Ssingh: [C:03+2] trafficserver: map websockets for the gitlab instances [puppet] - 10https://gerrit.wikimedia.org/r/1334855 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [14:32:07] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1074.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:33:59] elukey@cumin1003 upgrade-firmware (PID 2479275) is awaiting input [14:36:22] FIRING: GnmiInterfaceCountersDrop: ... [14:36:23] lsw1-c6-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-c6-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:37:23] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host cassandra-dev2002.codfw.wmnet [14:37:37] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [14:37:38] (03PS3) 10Muehlenhoff: conftool::conftool2git: Switch to firewall::service [puppet] - 10https://gerrit.wikimedia.org/r/1334921 [14:38:22] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Two VMs for the PKI infra - https://phabricator.wikimedia.org/T436903#12286237 (10elukey) >>! In T436903#12284669, @MoritzMuehlenhoff wrote: > Sounds good to me. How about the disks, looking at pki1002, 40-50G should be perfectly fine and leave ample headro... [14:38:35] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [14:38:37] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1072.eqiad.wmnet with OS trixie [14:38:51] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12286240 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1072.eqiad.wmnet with OS trixie completed: -... [14:39:34] !log sudo cumin "A:cp-text" "run-puppet-agent --enable 'merging CR 1334855'": T425441 [14:39:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:39:37] T425441: gitlab behind CDN - https://phabricator.wikimedia.org/T425441 [14:42:56] (03PS2) 10Hashar: ci: logoutput on failure of ci-build-images script [puppet] - 10https://gerrit.wikimedia.org/r/1333795 (https://phabricator.wikimedia.org/T436775) [14:43:23] (03PS2) 10Hashar: ci: set ci-build-images image and update creates [puppet] - 10https://gerrit.wikimedia.org/r/1333796 (https://phabricator.wikimedia.org/T436775) [14:45:06] !log vriley@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1066.eqiad.wmnet with reason: host reimage [14:46:23] FIRING: [3x] GnmiInterfaceCountersDrop: cloudsw1-e4-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:47:23] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cassandra-dev2002.codfw.wmnet [14:47:25] !log elukey@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cassandra-dev2002.codfw.wmnet [14:48:49] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286301 (10VRiley-WMF) [14:48:53] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1066.eqiad.wmnet with reason: host reimage [14:49:54] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1073.eqiad.wmnet with OS trixie [14:50:10] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12286310 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1073.eqiad.wmnet with OS trixie [14:51:22] FIRING: [5x] GnmiInterfaceCountersDrop: cloudsw1-e4-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:52:29] !log cdobbins@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo [14:53:25] !log elukey@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cassandra-dev2002.codfw.wmnet [14:54:40] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations: attempts to reimage cp5022 are failing - https://phabricator.wikimedia.org/T436386#12286322 (10ssingh) cp5022 is back in production so I am resolving this. If anyone disagrees, please let me know and re-open. [14:54:42] (03PS11) 10Herron: sre.opensearch.roll-restart-reboot: include checklist items [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 [14:54:42] (03CR) 10Herron: "test-cookbook output: https://phabricator.wikimedia.org/paste/edit/96342/" [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (owner: 10Herron) [14:55:09] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations: attempts to reimage cp5022 are failing - https://phabricator.wikimedia.org/T436386#12286323 (10ssingh) Sorry, the previous comments did not show up (I did not refresh). [14:56:07] (03CR) 10Herron: "https://phabricator.wikimedia.org/P96342 rather" [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (owner: 10Herron) [14:58:31] (03CR) 10Ssingh: [C:03+2] Add urldownloader[12]00[34] to conf-tool [puppet] - 10https://gerrit.wikimedia.org/r/1333201 (https://phabricator.wikimedia.org/T429175) (owner: 10Muehlenhoff) [14:58:40] !log blake@cumin1003 END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-codfw [15:00:36] (03PS1) 10Blake: memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334941 (https://phabricator.wikimedia.org/T353511) [15:00:44] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1334941 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [15:00:48] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: cluster=urldownloader [15:02:16] (03CR) 10Gmodena: [C:03+2] "We can refactor the values post merge and deployment." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333174 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [15:02:25] (03CR) 10Blake: [C:03+2] memcached: continue PKI rollout [puppet] - 10https://gerrit.wikimedia.org/r/1334941 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [15:02:28] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox) [15:03:59] (03PS1) 10Sbisson: ArticleGuidance: Add the redirect configuration keys [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334945 (https://phabricator.wikimedia.org/T434487) [15:04:01] (03PS1) 10Sbisson: ArticleGuidance: Remove the experiment configuration keys [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334946 (https://phabricator.wikimedia.org/T434487) [15:04:15] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [15:04:23] (03PS1) 10Scott French: api-gateway: Allow auto_host_rewrite to be overridden [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334947 (https://phabricator.wikimedia.org/T427666) [15:04:41] (03Merged) 10jenkins-bot: wdqs_common: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333174 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [15:04:51] !log vriley@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1073.eqiad.wmnet with reason: host reimage [15:04:55] !log elukey@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cassandra-dev2002.codfw.wmnet [15:05:31] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [15:05:32] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1066.eqiad.wmnet with OS trixie [15:05:44] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286394 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1066.eqiad.wmnet with OS trixie completed: - clo... [15:05:57] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cassandra-dev2002.codfw.wmnet with OS bookworm [15:06:22] FIRING: [10x] GnmiInterfaceCountersDrop: cloudsw1-e4-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:06:41] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286401 (10VRiley-WMF) [15:09:30] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1073.eqiad.wmnet with reason: host reimage [15:11:23] FIRING: [12x] GnmiInterfaceCountersDrop: cloudsw1-e4-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:13:08] !log gmodena@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [15:15:55] !log gmodena@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [15:16:22] FIRING: [12x] GnmiInterfaceCountersDrop: cloudsw1-e4-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:17:35] (03PS2) 10Scott French: api-gateway: Allow auto_host_rewrite to be overridden [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334947 (https://phabricator.wikimedia.org/T427666) [15:18:50] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [15:21:40] (03PS3) 10Jgiannelos: prv: Enable parsoid rendering for more wikisource wikiss [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334777 (https://phabricator.wikimedia.org/T436919) [15:24:11] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2002.codfw.wmnet with reason: host reimage [15:24:11] !log gmodena@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [15:25:10] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [15:25:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [15:26:30] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:26:40] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [15:26:49] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [15:27:59] (03CR) 10Cathal Mooney: [C:03+2] gnmic: Disable some less-important subscriptions for JunOS devices [puppet] - 10https://gerrit.wikimedia.org/r/1334781 (https://phabricator.wikimedia.org/T435909) (owner: 10Cathal Mooney) [15:28:14] vriley@cumin1003 reimage (PID 2484622) is awaiting input [15:28:34] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2002.codfw.wmnet with reason: host reimage [15:29:37] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [15:31:23] FIRING: [11x] GnmiInterfaceCountersDrop: cr2-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:33:11] (03PS4) 10Jgiannelos: prv: Enable parsoid rendering for more wikisource wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334777 (https://phabricator.wikimedia.org/T436919) [15:35:14] 10SRE-SLO, 06Abstract Wikipedia team (27Q1 (Jul–Sep)), 07OKR-Work, 13Patch-For-Review: new SLI (1 of 2): server-side metrics on Abstract Wikipedia preview - https://phabricator.wikimedia.org/T434231#12286552 (10ecarg) @RLazarus Ah TY, I cloned directly off repo rather than forking first; [[ https://gitlab.... [15:36:22] FIRING: [9x] GnmiInterfaceCountersDrop: cr2-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:36:58] (03CR) 10Scott French: "One incidental discovery while preparing my staging tests for the cluster-specifier changes. As explained in the commit message, this take" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334947 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:38:27] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334777 (https://phabricator.wikimedia.org/T436919) (owner: 10Jgiannelos) [15:41:22] FIRING: [9x] GnmiInterfaceCountersDrop: cr2-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:43:44] (03CR) 10Ayounsi: [C:03+2] Add netflow2004 and 2005 to Kafka ACL [puppet] - 10https://gerrit.wikimedia.org/r/1334762 (https://phabricator.wikimedia.org/T436517) (owner: 10Ayounsi) [15:46:22] FIRING: [9x] GnmiInterfaceCountersDrop: cr2-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:48:29] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2002.codfw.wmnet with OS bookworm [15:49:17] (03PS2) 10Urbanecm: Growth: Remove now removed config variable [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325868 [15:49:23] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1334921 (owner: 10Muehlenhoff) [15:49:27] jouncebot: nowandnext [15:49:28] No deployments scheduled for the next 0 hour(s) and 10 minute(s) [15:49:28] In 0 hour(s) and 10 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1600) [15:50:12] (03CR) 10Urbanecm: [C:03+2] Growth: Remove now removed config variable [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325868 (owner: 10Urbanecm) [15:50:55] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325868 (owner: 10Urbanecm) [15:51:12] (03Merged) 10jenkins-bot: Growth: Remove now removed config variable [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325868 (owner: 10Urbanecm) [15:51:22] RESOLVED: [9x] GnmiInterfaceCountersDrop: cr2-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:51:37] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1325868|Growth: Remove now removed config variable]] [15:51:43] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: name=urldownloader[12]00[56].wikimedia.org [reason: depooling urldownloader trixie nodes] [15:53:01] (03PS1) 10Muehlenhoff: Add pki[12]003 to site.pp and preseed [puppet] - 10https://gerrit.wikimedia.org/r/1334986 (https://phabricator.wikimedia.org/T436903) [15:58:21] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12286650 (10Aklapper) @Geertivp: That likely means that you used a different email address, if you are sure that you registered on the website (!) https://lists.wikimedia.org [16:00:05] jhathaway and rzl: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) Puppet request window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1600). [16:00:05] No Gerrit patches in the queue for this window AFAICS. [16:01:06] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325868|Growth: Remove now removed config variable]] (duration: 09m 29s) [16:01:40] (03CR) 10JHathaway: [C:03+1] conftool::conftool2git: Switch to firewall::service [puppet] - 10https://gerrit.wikimedia.org/r/1334921 (owner: 10Muehlenhoff) [16:02:16] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.4 point update - https://phabricator.wikimedia.org/T420240#12286656 (10MoritzMuehlenhoff) [16:03:54] !log btullis@deploy1003 Started scap sync-world: Rebuilding to pick up new version of dump scripts in mediawiki-cli for T436913 [16:03:57] T436913: 7za is OOMkilled when dumps are running on a Bookworm-based image - https://phabricator.wikimedia.org/T436913 [16:04:16] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet, wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [16:04:16] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [16:06:16] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:06:16] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:07:38] !log ryankemper@puppetserver1001 conftool action : set/pooled=yes; selector: name=wdqs101[1-4].eqiad.wmnet [16:09:57] (03PS5) 10Arlolra: prv: Enable parsoid rendering for more wikisource wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334777 (https://phabricator.wikimedia.org/T436919) (owner: 10Jgiannelos) [16:11:36] (03CR) 10BCornwall: [V:04-1] "`" [puppet] - 10https://gerrit.wikimedia.org/r/1333533 (https://phabricator.wikimedia.org/T425216) (owner: 10Krinkle) [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:06] !log btullis@deploy1003 sync-world failed: Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.17,1.47.0-wmf.18,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/m [16:12:06] ediawiki-multiversion --singleversion-image-basename docker-registry.discovery.wmnet/restricted/mediawiki-singleversion --webserver-image-name docker-registry.discovery.wmnet/restricted/mediawiki-webserver --latest-tag latest --label vnd.wikimedia.builder.name=scap --label vnd.wikimedia.builder.version=4.288.0 --label vnd.wikimedia.scap.stage_dir=/srv/mediawiki-staging --label vnd.wikimedia.scap.build_state_dir=/srv/media [16:12:06] wiki-staging/scap/image-build --full' returned non-zero exit status 1. (scap version: 4.288.0) (duration: 16m 11s) [16:12:08] !log btullis@deploy1003 Started deploy [analytics/refinery@0e301af] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@0e301af2] [16:12:46] !log btullis@deploy1003 Finished deploy [analytics/refinery@0e301af] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@0e301af2] (duration: 00m 38s) [16:14:44] !log T430880 Stumbled across `wdqs2021` listed as inactive, looks like it was never fully re-pooled after a data xfer. Pooled. [16:14:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:14:47] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [16:14:59] !log ryankemper@puppetserver1001 conftool action : set/pooled=yes; selector: name=wdqs2021.codfw.wmnet,service=wdqs-main [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:30:22] FIRING: CertAlmostExpired: gNMI TLS certificate for ssw1-d1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:39:23] FIRING: [3x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [16:39:53] !log btullis@deploy1003 Started scap sync-world: Trying again for T436913 [16:39:57] T436913: 7za is OOMkilled when dumps are running on a Bookworm-based image - https://phabricator.wikimedia.org/T436913 [16:40:32] (03CR) 10BCornwall: [C:03+1] wmnet: Update x1-master alias [dns] - 10https://gerrit.wikimedia.org/r/1334814 (https://phabricator.wikimedia.org/T436923) (owner: 10Gerrit maintenance bot) [16:41:36] (03PS2) 10Samtar: IS/IS-labs: wgEnableWatchstarPopover default false, enable for enwiki beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307862 (https://phabricator.wikimedia.org/T431355) [16:41:44] FIRING: KubernetesDeploymentUnavailableReplicas: ... [16:41:44] Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ... [16:41:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:48:22] (03PS1) 10Samtar: IS: enable wgEnableWatchstarPopover on test.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335032 (https://phabricator.wikimedia.org/T436955) [16:49:10] (03PS2) 10Samtar: IS: enable wgEnableWatchstarPopover on test.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335032 (https://phabricator.wikimedia.org/T436955) [16:49:20] (03PS3) 10Samtar: IS/IS-labs: wgEnableWatchstarPopover default false, enable for enwiki beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307862 (https://phabricator.wikimedia.org/T431355) [16:50:27] !log andrew@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [16:51:09] !log andrew@cumin2003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [16:51:25] !log andrew@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [16:51:44] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [16:51:44] Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ... [16:51:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:51:49] !log andrew@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [16:52:40] 06SRE, 06Commons, 10MediaWiki-File-management, 06Traffic, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12286788 (10Bawolff) >>! In T425216#12284293, @Tgr wrote: > Possibly Varnish normalizes thumb URLs but do... [16:52:46] !log btullis@deploy1003 sync-world failed: Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.17,1.47.0-wmf.18,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/m [16:52:46] ediawiki-multiversion --singleversion-image-basename docker-registry.discovery.wmnet/restricted/mediawiki-singleversion --webserver-image-name docker-registry.discovery.wmnet/restricted/mediawiki-webserver --latest-tag latest --label vnd.wikimedia.builder.name=scap --label vnd.wikimedia.builder.version=4.288.0 --label vnd.wikimedia.scap.stage_dir=/srv/mediawiki-staging --label vnd.wikimedia.scap.build_state_dir=/srv/media [16:52:46] wiki-staging/scap/image-build --full' returned non-zero exit status 1. (scap version: 4.288.0) (duration: 13m 34s) [16:55:12] andrew@cumin2003 upgrade-firmware (PID 3115820) is awaiting input [16:55:59] (03Abandoned) 10Samtar: InitialiseSettings: Enable TemplateDiscovery for dewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1160126 (https://phabricator.wikimedia.org/T377975) (owner: 10Samtar) [16:58:30] !log Running scap clean-images on deploy1003 [16:58:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:59:49] !log andrew@cumin2003 START - Cookbook sre.hosts.reboot-single for host cloudcephosd1045.eqiad.wmnet [17:00:05] bd808: That opportune time for a Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1700). [17:00:06] swfrench-wmf: How many deployers does it take to do MediaWiki infrastructure (UTC late) deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1700). [17:00:19] o/ [17:00:45] (03CR) 10Scott French: [C:03+2] api-gateway: Support x-wikimedia-debug routing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311964 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:01:33] !log btullis@deploy1003 Started scap sync-world: Trying again for T436913 [17:01:37] T436913: 7za is OOMkilled when dumps are running on a Bookworm-based image - https://phabricator.wikimedia.org/T436913 [17:01:51] I'll be deploying some rest-gateway changes during this infra window. none of the new functionality being added will be enabled in production, though I will briefly enable / test it in staging. [17:03:25] (03Merged) 10jenkins-bot: api-gateway: Support x-wikimedia-debug routing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311964 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:04:21] (03PS1) 10Ssingh: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 [17:04:33] (03PS7) 10Scott French: api-gateway: Support Host-based diversion in the mw-api plugin [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322140 (https://phabricator.wikimedia.org/T433752) [17:04:33] (03PS3) 10Scott French: api-gateway: Allow auto_host_rewrite to be overridden [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334947 (https://phabricator.wikimedia.org/T427666) [17:07:29] (03CR) 10CI reject: [V:04-1] sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 (owner: 10Ssingh) [17:09:29] !log btullis@deploy1003 sync-world failed: Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.17,1.47.0-wmf.18,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted/m [17:09:29] ediawiki-multiversion --singleversion-image-basename docker-registry.discovery.wmnet/restricted/mediawiki-singleversion --webserver-image-name docker-registry.discovery.wmnet/restricted/mediawiki-webserver --latest-tag latest --label vnd.wikimedia.builder.name=scap --label vnd.wikimedia.builder.version=4.288.0 --label vnd.wikimedia.scap.stage_dir=/srv/mediawiki-staging --label vnd.wikimedia.scap.build_state_dir=/srv/media [17:09:29] wiki-staging/scap/image-build --full' returned non-zero exit status 1. (scap version: 4.288.0) (duration: 08m 30s) [17:10:15] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:10:28] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:11:15] (03PS2) 10Ssingh: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 [17:13:15] !log gmodena@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [17:14:15] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a2-eqiad [17:15:19] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a2-eqiad [17:15:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:16:06] (03PS3) 10Ssingh: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 [17:16:28] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:16:53] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:17:06] (03PS4) 10Ssingh: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 [17:17:29] (03PS5) 10Ssingh: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 [17:22:44] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:22:54] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:23:41] (03CR) 10Scott French: [C:03+2] api-gateway: Support Host-based diversion in the mw-api plugin [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322140 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:24:31] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a2-eqiad [17:24:52] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a2-eqiad [17:25:45] FIRING: WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [17:26:07] (03Merged) 10jenkins-bot: api-gateway: Support Host-based diversion in the mw-api plugin [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322140 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:26:29] !log dancy@deploy1003 Installing scap version "4.289.0" for 3 host(s) [17:26:31] puppetserver acting up [17:27:49] yeah this can't be ignored [17:28:20] (03PS1) 10Ottomata: refine jobs - explicitly set smtp_uri everywhere [puppet] - 10https://gerrit.wikimedia.org/r/1335061 (https://phabricator.wikimedia.org/T393202) [17:28:24] !log dancy@deploy1003 Installation of scap version "4.289.0" completed for 3 hosts [17:28:47] ok it seems to be getting better so I guess we wait to see how it plays out [17:28:54] (03PS2) 10Ottomata: refine jobs - explicitly set smtp_uri everywhere [puppet] - 10https://gerrit.wikimedia.org/r/1335061 (https://phabricator.wikimedia.org/T393202) [17:28:56] (03CR) 10CI reject: [V:04-1] refine jobs - explicitly set smtp_uri everywhere [puppet] - 10https://gerrit.wikimedia.org/r/1335061 (https://phabricator.wikimedia.org/T393202) (owner: 10Ottomata) [17:30:34] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:30:45] FIRING: [2x] WidespreadPuppetFailure: Puppet has failed in eqiad - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [17:30:49] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:31:34] !log andrew@cumin2003 END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host cloudcephosd1045.eqiad.wmnet [17:34:25] (03PS6) 10Ssingh: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 [17:36:11] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:36:25] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:37:23] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [17:37:48] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [17:38:03] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-08-19-165650 to 2026-09-03-172537 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335068 (https://phabricator.wikimedia.org/T426337) [17:38:24] (03PS7) 10Ssingh: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 [17:39:06] !log T421642 [WDQS] requestctl changes: `2026-09-03 16:23-17:33` UTC: added hard-deny pair `cache-text/wdqs_futile_sparql_sep_2026_deny(+_bots)`; extended pattern `ua/wdqs_heavy_sparql_bots_2026` and added default-scope twin `wdqs_heavy_sparql_bots_jul_2026_ratelimit_default`; added ipblock `abuse/wdqs_sparql_scanners_sep_2026` + throttle `wdqs_sparql_scanners_sep_2026_ratelimit` (needed manual `requestctl update-provenance-map`) [17:39:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:39:09] T421642: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642 [17:39:23] !log [WDQS] Service looks healthy again, CPU load and thread count have dropped considerably over the last hour [17:39:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:40:04] (03PS8) 10Ssingh: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 [17:40:40] (03CR) 10Ryan Kemper: [C:03+2] refine jobs - explicitly set smtp_uri everywhere [puppet] - 10https://gerrit.wikimedia.org/r/1335061 (https://phabricator.wikimedia.org/T393202) (owner: 10Ottomata) [17:40:41] !log andrew@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [17:40:44] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-08-19-165650 to 2026-09-03-172537 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335068 (https://phabricator.wikimedia.org/T426337) (owner: 10Jforrester) [17:40:45] FIRING: [2x] WidespreadPuppetFailure: Puppet has failed in eqiad - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [17:40:53] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: Q1:rack/setup/install dse-k8s-worker10[29-38] - https://phabricator.wikimedia.org/T436964 (10RobH) 03NEW [17:41:10] (03CR) 10Scott French: "Moving this back to WIP while reevaluating whether we need this at all. What I failed to realize before is that there's no other relevant " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334947 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [17:41:12] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: Q1:rack/setup/install dse-k8s-worker10[29-38] - https://phabricator.wikimedia.org/T436964#12287021 (10RobH) [17:41:42] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: Q1:rack/setup/install dse-k8s-worker10[29-38] - https://phabricator.wikimedia.org/T436964#12287023 (10RobH) a:03BTullis [17:43:14] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-08-19-165650 to 2026-09-03-172537 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335068 (https://phabricator.wikimedia.org/T426337) (owner: 10Jforrester) [17:44:45] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [17:44:57] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [17:45:17] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [17:45:29] !log jforrester@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [17:45:30] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [17:46:01] jforrester@deploy1003: Failed to log message to wiki. Somebody should check the error logs. [17:46:10] !log jforrester@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [17:46:17] !log jforrester@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [17:46:49] !log jforrester@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [17:47:24] FYI, I'm done with rest-gateway related work. my tests in staging have been reverted and everything should be back to normal. [17:48:34] (03PS1) 10Catrope: Instrumentation for email confirmation upfront enforcement A/A test [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335073 (https://phabricator.wikimedia.org/T435135) [17:49:01] (03PS1) 10Catrope: Email confirmation A/A: check creation wiki, centralize logic [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335074 (https://phabricator.wikimedia.org/T435135) [17:49:20] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335073 (https://phabricator.wikimedia.org/T435135) (owner: 10Catrope) [17:49:32] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335074 (https://phabricator.wikimedia.org/T435135) (owner: 10Catrope) [17:50:14] (03CR) 10BBlack: "+1 LGTM!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 (owner: 10Ssingh) [17:52:08] !log andrew@cumin1003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [17:53:34] !log andrew@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [17:54:04] !log andrew@cumin1003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [17:54:38] !log andrew@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [17:54:41] !log andrew@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [17:55:33] !log andrew@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [17:55:36] !log andrew@cumin1003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [18:00:05] dancy and hashar: Deploy window MediaWiki train - Utc-7+Utc-0 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T1800) [18:00:11] o/ [18:00:33] 06SRE, 10SRE-Access-Requests: Requesting access to production for derenrich - https://phabricator.wikimedia.org/T436965 (10derenrich) 03NEW [18:00:36] (03PS1) 10TrainBranchBot: group2 to 1.47.0-wmf.18 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335077 (https://phabricator.wikimedia.org/T430837) [18:00:39] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dancy@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335077 (https://phabricator.wikimedia.org/T430837) (owner: 10TrainBranchBot) [18:00:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [18:01:36] (03Merged) 10jenkins-bot: group2 to 1.47.0-wmf.18 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335077 (https://phabricator.wikimedia.org/T430837) (owner: 10TrainBranchBot) [18:03:00] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:16:53] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cassandra-dev2002.codfw.wmnet with OS bookworm [18:20:57] !log dancy@deploy1003 rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.18 refs T430837 [18:21:00] T430837: 1.47.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T430837 [18:27:17] (03PS1) 10Bking: Druid: Add alerts for memory and I/O stalls [alerts] - 10https://gerrit.wikimedia.org/r/1335090 (https://phabricator.wikimedia.org/T436968) [18:27:39] (03PS2) 10Bking: Druid: Add alerts for memory and I/O stalls [alerts] - 10https://gerrit.wikimedia.org/r/1335090 (https://phabricator.wikimedia.org/T436968) [18:30:04] (03CR) 10CI reject: [V:04-1] Druid: Add alerts for memory and I/O stalls [alerts] - 10https://gerrit.wikimedia.org/r/1335090 (https://phabricator.wikimedia.org/T436968) (owner: 10Bking) [18:34:43] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2002.codfw.wmnet with reason: host reimage [18:39:52] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2002.codfw.wmnet with reason: host reimage [18:45:06] (03PS3) 10Bking: Druid: Add alerts for memory and I/O stalls [alerts] - 10https://gerrit.wikimedia.org/r/1335090 (https://phabricator.wikimedia.org/T436968) [18:50:02] (03PS1) 10Tsevener: Add exclusions to Apple app site association file [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335109 (https://phabricator.wikimedia.org/T435363) [18:55:06] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335109 (https://phabricator.wikimedia.org/T435363) (owner: 10Tsevener) [18:57:56] !log andrew@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [18:57:59] !log andrew@cumin1003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [19:00:10] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2002.codfw.wmnet with OS bookworm [19:01:43] !log andrew@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [19:01:46] !log andrew@cumin1003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cloudcephosd1045.eqiad.wmnet [19:03:01] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:05:51] dancy: once the dust settles after rolling the train, perhaps I could merge and test your [0]. any concerns about doing that? if not, please let me know when I should do so :) [19:05:52] [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1333296 [19:06:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:06:43] swfrench-wmf: train is done. Go for it [19:07:01] dancy: awesome, will do [19:08:31] (03CR) 10Scott French: [C:03+2] "Thanks, Ahmon!" [puppet] - 10https://gerrit.wikimedia.org/r/1333296 (https://phabricator.wikimedia.org/T435419) (owner: 10Ahmon Dancy) [19:16:47] !log swfrench@deploy1003 Started scap sync-world: Noop deployment to validate pretrain logstash check configuration - T435419 [19:16:50] T435419: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419 [19:19:46] !log swfrench@deploy1003 Finished scap sync-world: Noop deployment to validate pretrain logstash check configuration - T435419 (duration: 02m 59s) [19:20:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:25:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:31:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:33:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:41:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:45:36] (03PS4) 10Milazg: Remove mode from RestModuleOverrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328215 (https://phabricator.wikimedia.org/T434267) [19:47:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:49:13] (03Abandoned) 10Krinkle: Switch changeprop to native prometheus metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/642009 (owner: 10Ppchelko) [19:49:58] !log eevans@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs1021.eqiad.wmnet [19:52:18] (03PS1) 10Bking: matomo: Restore GeoIP functionality [puppet] - 10https://gerrit.wikimedia.org/r/1335153 (https://phabricator.wikimedia.org/T431608) [19:53:51] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335153 (https://phabricator.wikimedia.org/T431608) (owner: 10Bking) [19:55:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: gettimeofday() says it's time for UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T2000) [20:00:05] RoanKattouw, arlolra, and toni_: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:44] !log eevans@cumin1003 START - Cookbook sre.hosts.reboot-single for host aqs1021.eqiad.wmnet [20:03:49] I'm here, I'll do the deployment [20:04:07] FIRING: [2x] ProbeDown: Service aqs1021-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:04:48] (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1334268 (https://phabricator.wikimedia.org/T435135) (owner: 10Catrope) [20:04:49] (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335073 (https://phabricator.wikimedia.org/T435135) (owner: 10Catrope) [20:04:49] (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335074 (https://phabricator.wikimedia.org/T435135) (owner: 10Catrope) [20:09:07] FIRING: [4x] ProbeDown: Service aqs1021-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:09:15] (03Merged) 10jenkins-bot: Email confirmation A/A test: make registration cutoff consistent [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1334268 (https://phabricator.wikimedia.org/T435135) (owner: 10Catrope) [20:09:17] (03Merged) 10jenkins-bot: Instrumentation for email confirmation upfront enforcement A/A test [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335073 (https://phabricator.wikimedia.org/T435135) (owner: 10Catrope) [20:09:19] (03Merged) 10jenkins-bot: Email confirmation A/A: check creation wiki, centralize logic [extensions/WikimediaEvents] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335074 (https://phabricator.wikimedia.org/T435135) (owner: 10Catrope) [20:09:41] !log catrope@deploy1003 Started scap sync-world: Backport for [[gerrit:1334268|Email confirmation A/A test: make registration cutoff consistent (T435135)]], [[gerrit:1335073|Instrumentation for email confirmation upfront enforcement A/A test (T435135)]], [[gerrit:1335074|Email confirmation A/A: check creation wiki, centralize logic (T435135)]] [20:09:45] T435135: Create A/A test for page edit - https://phabricator.wikimedia.org/T435135 [20:10:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:11:15] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12287602 (10Geertivp) I can confirm that the login account with wich I was admin in the past does not exist any more on the system, so I can't login, but it is still registered as having admin... [20:11:29] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1021.eqiad.wmnet [20:11:30] !log eevans@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs1021.eqiad.wmnet [20:13:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:13:56] !log catrope@deploy1003 catrope: Backport for [[gerrit:1334268|Email confirmation A/A test: make registration cutoff consistent (T435135)]], [[gerrit:1335073|Instrumentation for email confirmation upfront enforcement A/A test (T435135)]], [[gerrit:1335074|Email confirmation A/A: check creation wiki, centralize logic (T435135)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be [20:13:56] verified there. [20:14:07] RESOLVED: [4x] ProbeDown: Service aqs1021-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:14:24] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host aqs1021.eqiad.wmnet with OS bookworm [20:16:38] !log catrope@deploy1003 catrope: Continuing with deployment [20:18:32] o/ [20:19:07] FIRING: [4x] ProbeDown: Service aqs1021-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:19:07] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Attempt to use a VM as Kerberos replica - https://phabricator.wikimedia.org/T435873#12287629 (10bking) [[ https://grafana.wikimedia.org/d/000000342/node-exporter-server-metrics?orgId=1&from=now-7d&to=now&timezone=utc&var-datasourc... [20:20:08] o/ [20:20:49] !log andrew@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephosd1046.eqiad.wmnet [20:23:23] !log catrope@deploy1003 Finished scap sync-world: Backport for [[gerrit:1334268|Email confirmation A/A test: make registration cutoff consistent (T435135)]], [[gerrit:1335073|Instrumentation for email confirmation upfront enforcement A/A test (T435135)]], [[gerrit:1335074|Email confirmation A/A: check creation wiki, centralize logic (T435135)]] (duration: 13m 41s) [20:23:27] T435135: Create A/A test for page edit - https://phabricator.wikimedia.org/T435135 [20:24:07] FIRING: [4x] ProbeDown: Service aqs1021-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:26:21] (03PS1) 10Bking: Kerberos: remove krb1003 from list presented to clients [puppet] - 10https://gerrit.wikimedia.org/r/1335159 (https://phabricator.wikimedia.org/T435873) [20:26:52] RoanKattouw: Can I go? [20:27:01] arlolra: Yes sorry, go ahead [20:27:06] Thanks [20:27:26] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334777 (https://phabricator.wikimedia.org/T436919) (owner: 10Jgiannelos) [20:27:37] toni_: Are you planning to self-serve, or do you want me to do yours after arlolra is done? [20:28:12] sorry yeah I will need someone else to deploy, but happy to go last [20:28:23] (03Merged) 10jenkins-bot: prv: Enable parsoid rendering for more wikisource wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334777 (https://phabricator.wikimedia.org/T436919) (owner: 10Jgiannelos) [20:28:36] !log arlolra@deploy1003 Started scap sync-world: Backport for [[gerrit:1334777|prv: Enable parsoid rendering for more wikisource wikis (T436919)]] [20:28:39] T436919: Rollout parsoid read views on wikisource - week of 28 Aug - https://phabricator.wikimedia.org/T436919 [20:28:45] (03PS2) 10Bking: Kerberos: remove krb1003 from list presented to clients [puppet] - 10https://gerrit.wikimedia.org/r/1335159 (https://phabricator.wikimedia.org/T435873) [20:28:48] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335159 (https://phabricator.wikimedia.org/T435873) (owner: 10Bking) [20:28:57] !log eevans@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1021.eqiad.wmnet with reason: host reimage [20:29:07] FIRING: [4x] ProbeDown: Service aqs1021-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:29:22] FIRING: [4x] ProbeDown: Service aqs1021-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:30:03] silly question about process here but when you all do o/, does that just mean "I'm here and paying attention" or does that mean "I will deploy my own patch". I've been raising my hand to indicate I'm here but I might just be confusing everyone [20:30:38] FIRING: CertAlmostExpired: gNMI TLS certificate for ssw1-d1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [20:31:15] !log andrew@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1046.eqiad.wmnet [20:31:39] toni_: the former [20:31:45] you're doing it right [20:31:50] haha ok cool [20:32:43] !log arlolra@deploy1003 arlolra, jgiannelos: Backport for [[gerrit:1334777|prv: Enable parsoid rendering for more wikisource wikis (T436919)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:33:08] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1021.eqiad.wmnet with reason: host reimage [20:33:40] !log andrew@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephosd1047.eqiad.wmnet [20:34:07] FIRING: [4x] ProbeDown: Service aqs1021-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:34:25] !log arlolra@deploy1003 arlolra, jgiannelos: Continuing with deployment [20:38:59] !log arlolra@deploy1003 Finished scap sync-world: Backport for [[gerrit:1334777|prv: Enable parsoid rendering for more wikisource wikis (T436919)]] (duration: 10m 23s) [20:39:03] T436919: Rollout parsoid read views on wikisource - week of 28 Aug - https://phabricator.wikimedia.org/T436919 [20:39:07] FIRING: [4x] ProbeDown: Service aqs1021-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:39:30] RoanKattouw: back to you, or I could deploy the last patch if you want [20:39:42] FIRING: [3x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [20:39:58] arlolra: If you wouldn't mind deploying that would be great [20:40:05] no problem [20:40:16] I had an interview over my lunch hour so now I need to eat :) [20:40:29] enjoy [20:40:46] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a3-eqiad [20:41:03] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a3-eqiad [20:41:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335109 (https://phabricator.wikimedia.org/T435363) (owner: 10Tsevener) [20:42:29] (03Merged) 10jenkins-bot: Add exclusions to Apple app site association file [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335109 (https://phabricator.wikimedia.org/T435363) (owner: 10Tsevener) [20:42:43] !log arlolra@deploy1003 Started scap sync-world: Backport for [[gerrit:1335109|Add exclusions to Apple app site association file (T435363)]] [20:42:46] T435363: [Eng] Visual Editor: Copy test wiki association file over to production language endpoints (iOS) - https://phabricator.wikimedia.org/T435363 [20:44:14] !log andrew@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1047.eqiad.wmnet [20:45:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1047:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1047 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [20:46:56] !log arlolra@deploy1003 arlolra, tsev: Backport for [[gerrit:1335109|Add exclusions to Apple app site association file (T435363)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:47:30] toni_: time to test [20:48:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.49% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:49:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:50:19] looks good [20:50:30] !log arlolra@deploy1003 arlolra, tsev: Continuing with deployment [20:52:00] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1021.eqiad.wmnet with OS bookworm [20:52:24] !log eevans@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs1021.eqiad.wmnet [20:54:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:55:07] !log arlolra@deploy1003 Finished scap sync-world: Backport for [[gerrit:1335109|Add exclusions to Apple app site association file (T435363)]] (duration: 12m 24s) [20:55:10] T435363: [Eng] Visual Editor: Copy test wiki association file over to production language endpoints (iOS) - https://phabricator.wikimedia.org/T435363 [20:55:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.19% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:55:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1047:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1047 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [20:55:48] done with the backport window, readers on to you [20:56:20] thank you! [20:59:07] RESOLVED: [4x] ProbeDown: Service aqs1021-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260903T2100) [21:03:56] !log eevans@cumin1003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts aqs1021.eqiad.wmnet [21:10:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.63% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:21:28] 06SRE, 10fundraising-tech-ops: donate.wikimedia.org 2fa reset? - https://phabricator.wikimedia.org/T436944#12287821 (10JLam-WMF) Hi all, Just a note. @Pcoombe resolved this by deactivating Noah's accounts and creating a new one. Content will be moved off donate wiki in the future, which will make a repeat of... [21:21:41] 06SRE, 10fundraising-tech-ops: donate.wikimedia.org 2fa reset? - https://phabricator.wikimedia.org/T436944#12287822 (10JLam-WMF) 05Open→03Resolved [21:21:51] (03PS2) 10Ryan Kemper: matomo: Restore GeoIP functionality [puppet] - 10https://gerrit.wikimedia.org/r/1335153 (https://phabricator.wikimedia.org/T431608) (owner: 10Bking) [21:23:05] (03CR) 10Ryan Kemper: [C:03+1] matomo: Restore GeoIP functionality [puppet] - 10https://gerrit.wikimedia.org/r/1335153 (https://phabricator.wikimedia.org/T431608) (owner: 10Bking) [21:46:31] !log tsev@deploy1003 mwscript-k8s job started: purgeList.php # T435363 [21:46:34] T435363: [Eng] Visual Editor: Copy test wiki association file over to production language endpoints (iOS) - https://phabricator.wikimedia.org/T435363 [22:20:29] (03PS2) 10Andrea Denisse: klaxon: Add new GitLab origin [puppet] - 10https://gerrit.wikimedia.org/r/1335233 (https://phabricator.wikimedia.org/T436694) [22:20:29] (03CR) 10Andrea Denisse: [V:03+1] "PCC results: https://puppet-compiler.wmflabs.org/output/1335233/9366/" [puppet] - 10https://gerrit.wikimedia.org/r/1335233 (https://phabricator.wikimedia.org/T436694) (owner: 10Andrea Denisse) [22:35:14] (03CR) 10Andrea Denisse: [V:03+1] klaxon: Add new GitLab origin (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1335233 (https://phabricator.wikimedia.org/T436694) (owner: 10Andrea Denisse) [23:33:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed