[00:17:27] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [00:25:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:26:27] 10SRE-SLO: Support SLOs based on Test Kitchen instruments - https://phabricator.wikimedia.org/T438274 (10RLazarus) 03NEW p:05Triage→03Medium [00:40:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:42:44] RECOVERY - snapshot of s4 in codfw on backupmon1001 is OK: Last snapshot for s4 at codfw (db2239) taken on 2026-09-16 23:51:36 (1180 GiB, +0.1 %) https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [00:44:56] FIRING: TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [00:47:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:05:24] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-ml_staging_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:11:00] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1342392 [01:11:00] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1342392 (owner: 10TrainBranchBot) [01:19:35] (03CR) 10CI reject: [V:04-1] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1342392 (owner: 10TrainBranchBot) [01:27:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:29:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:34:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:34:25] (03PS1) 10Jdlrobson: Localisation updates from https://translatewiki.net. [skins/MinervaNeue] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342395 [01:34:55] (03PS1) 10Jdlrobson: Localisation updates from https://translatewiki.net. [extensions/MobileFrontend] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342396 [01:37:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:54:42] (03PS1) 10Jdlrobson: Localisation updates from https://translatewiki.net. [extensions/MobileFrontend] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342398 [02:00:04] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T0200) [02:04:56] RESOLVED: TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [02:05:24] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:07:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.18% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:08:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:12:12] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:14:42] 10ops-codfw, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438282 (10phaultfinder) 03NEW [02:17:12] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:29:52] (03PS1) 10Ryan Kemper: druid: (step 3) coordinator LVS to lvs_setup [puppet] - 10https://gerrit.wikimedia.org/r/1342400 (https://phabricator.wikimedia.org/T432080) [02:29:55] (03PS1) 10Ryan Kemper: druid: (step 4) coordinator LVS to service_setup [puppet] - 10https://gerrit.wikimedia.org/r/1342401 (https://phabricator.wikimedia.org/T432080) [02:29:57] (03PS1) 10Ryan Kemper: druid: (steps 5-6) remove coordinator LVS service [puppet] - 10https://gerrit.wikimedia.org/r/1342402 (https://phabricator.wikimedia.org/T432080) [02:32:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:32:31] FIRING: [2x] Traffic bill over quota: Alert for device cr1-eqiad.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [02:33:04] (03PS1) 10Ryan Kemper: druid: (step 2) remove coordinator LVS DNS records [dns] - 10https://gerrit.wikimedia.org/r/1342403 (https://phabricator.wikimedia.org/T432080) [02:33:13] (03CR) 10CI reject: [V:04-1] druid: (step 4) coordinator LVS to service_setup [puppet] - 10https://gerrit.wikimedia.org/r/1342401 (https://phabricator.wikimedia.org/T432080) (owner: 10Ryan Kemper) [02:33:56] FIRING: TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [02:35:24] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:37:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:37:31] FIRING: [3x] Traffic bill over quota: Alert for device cr1-eqiad.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [02:52:31] FIRING: [3x] Traffic bill over quota: Alert for device cr1-eqiad.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [02:57:31] RESOLVED: Traffic bill over quota: Alert for device cr2-codfw.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [03:09:23] (03CR) 10Jasmine: [C:03+1] hieradata: update deployment_server to deploy2003 [puppet] - 10https://gerrit.wikimedia.org/r/1319826 (https://phabricator.wikimedia.org/T436630) (owner: 10Slyngshede) [03:10:05] (03CR) 10Jasmine: [C:03+1] wmnet: update deployment CNAME record to deploy2003 [dns] - 10https://gerrit.wikimedia.org/r/1319825 (https://phabricator.wikimedia.org/T436630) (owner: 10Slyngshede) [03:10:24] FIRING: [3x] SystemdUnitFailed: docker-system-prune-dangling.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:16:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:21:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:22:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.63% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:47:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:12:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:13:56] RESOLVED: TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [04:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [04:24:50] !log aokoth@cumin1004 START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet [04:26:29] !log aokoth@cumin1004 END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet [04:27:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:30:02] PROBLEM - Backup freshness on backup1014 is CRITICAL: Stale: 1 (arclamp2001), Fresh: 142 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [04:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.11% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:43:56] FIRING: TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [04:47:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:54:16] (03CR) 10Ryan Kemper: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1342401 (https://phabricator.wikimedia.org/T432080) (owner: 10Ryan Kemper) [05:05:25] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-ml_serve_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:11:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:16:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:42:42] (03PS1) 10Marostegui: mariadb: Decommission db1180 [puppet] - 10https://gerrit.wikimedia.org/r/1342540 (https://phabricator.wikimedia.org/T437222) [05:43:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:43:58] !log marostegui@cumin1004 START - Cookbook sre.mysql.decommission [05:44:27] !log marostegui@cumin1004 START - Cookbook sre.hosts.decommission for hosts db1180.eqiad.wmnet [05:48:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:49:15] !log marostegui@cumin1004 START - Cookbook sre.dns.netbox [05:49:46] (03CR) 10Marostegui: [C:03+2] mariadb: Decommission db1180 [puppet] - 10https://gerrit.wikimedia.org/r/1342540 (https://phabricator.wikimedia.org/T437222) (owner: 10Marostegui) [05:53:05] !log marostegui@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1180.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" [05:53:21] !log marostegui@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1180.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" [05:53:21] !log marostegui@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [05:53:22] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1180.eqiad.wmnet [05:53:27] !log marostegui@cumin1004 Removing db1180 from zarcillo T437222 [05:53:27] !log marostegui@cumin1004 END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) [05:53:30] T437222: decommission db1180.eqiad.wmnet - https://phabricator.wikimedia.org/T437222 [05:54:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.72% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:55:51] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1180.eqiad.wmnet - https://phabricator.wikimedia.org/T437222#12331934 (10Marostegui) a:05Marostegui→03None [05:57:22] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1180.eqiad.wmnet - https://phabricator.wikimedia.org/T437222#12331941 (10Marostegui) This is ready for DCOps [05:59:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.59% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:59:54] (03PS1) 10Filippo Giunchedi: cloudnfs: report zero clients in nfsd-exporter when none are connected [puppet] - 10https://gerrit.wikimedia.org/r/1342541 (https://phabricator.wikimedia.org/T436252) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T0600) [06:00:05] marostegui, cezmunsta, and federico3: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Primary database switchover . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T0600). [06:05:25] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-ml_serve_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:07:04] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [06:11:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:14:29] 10SRE-SLO: Support SLOs based on Test Kitchen instruments - https://phabricator.wikimedia.org/T438274#12331964 (10tappof) [06:17:12] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:42:24] (03PS5) 10Federico Ceratto: mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) [06:44:34] (03PS1) 10Muehlenhoff: Remove access for jmoore111 [puppet] - 10https://gerrit.wikimedia.org/r/1342544 [06:45:35] (03CR) 10CI reject: [V:04-1] Remove access for jmoore111 [puppet] - 10https://gerrit.wikimedia.org/r/1342544 (owner: 10Muehlenhoff) [06:46:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.8% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:47:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:48:21] (03PS1) 10Matthias Mullie: Localisation updates from https://translatewiki.net. [extensions/MobileFrontend] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342545 [06:48:36] (03PS1) 10Matthias Mullie: Localisation updates from https://translatewiki.net. [skins/MinervaNeue] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342546 [06:48:42] (03PS2) 10Matthias Mullie: Localisation updates from https://translatewiki.net. [extensions/MobileFrontend] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342396 (owner: 10Jdlrobson) [06:49:00] (03PS2) 10Matthias Mullie: Localisation updates from https://translatewiki.net. [skins/MinervaNeue] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342395 (owner: 10Jdlrobson) [06:50:23] (03PS1) 10Matthias Mullie: Localisation updates from https://translatewiki.net. [extensions/MobileFrontend] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342547 [06:51:13] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/MobileFrontend] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342398 (owner: 10Jdlrobson) [06:51:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/MobileFrontend] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342396 (owner: 10Jdlrobson) [06:51:37] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [skins/MinervaNeue] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342395 (owner: 10Jdlrobson) [06:51:47] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/MobileFrontend] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342545 (owner: 10Matthias Mullie) [06:51:57] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [skins/MinervaNeue] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342546 (owner: 10Matthias Mullie) [06:52:05] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/MobileFrontend] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342547 (owner: 10Matthias Mullie) [06:52:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:53:30] (03PS2) 10Muehlenhoff: Remove access for jmoore111 [puppet] - 10https://gerrit.wikimedia.org/r/1342544 [06:54:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:54:36] (03CR) 10CI reject: [V:04-1] Remove access for jmoore111 [puppet] - 10https://gerrit.wikimedia.org/r/1342544 (owner: 10Muehlenhoff) [07:00:05] Amir1, urbanecm, and awight: It is that lovely time of the day again! You are hereby commanded to deploy UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T0700). [07:00:05] matthiasmullie: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:14] o/ [07:00:32] (03CR) 10Marostegui: "This only covers database backups, no mediabackups. Which is fine and I assume it is going to be done in a different patch, right? I am ok" [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [07:01:38] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MobileFrontend] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342398 (owner: 10Jdlrobson) [07:01:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MobileFrontend] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342396 (owner: 10Jdlrobson) [07:01:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [skins/MinervaNeue] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342395 (owner: 10Jdlrobson) [07:01:40] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MobileFrontend] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342545 (owner: 10Matthias Mullie) [07:01:40] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [skins/MinervaNeue] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342546 (owner: 10Matthias Mullie) [07:01:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MobileFrontend] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342547 (owner: 10Matthias Mullie) [07:02:54] (03PS3) 10Muehlenhoff: Remove access for jmoore111 [puppet] - 10https://gerrit.wikimedia.org/r/1342544 [07:03:48] (03CR) 10CWilliams: mariadb: Deploy backup_metrics.py (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [07:03:57] (03Merged) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/MobileFrontend] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342398 (owner: 10Jdlrobson) [07:06:35] (03CR) 10Muehlenhoff: [C:03+2] Remove access for jmoore111 [puppet] - 10https://gerrit.wikimedia.org/r/1342544 (owner: 10Muehlenhoff) [07:08:12] (03CR) 10Ayounsi: [C:03+2] Add Arelion Prometheus integration [puppet] - 10https://gerrit.wikimedia.org/r/1341098 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [07:09:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:09:38] (03Merged) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/MobileFrontend] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342396 (owner: 10Jdlrobson) [07:09:39] (03Merged) 10jenkins-bot: Localisation updates from https://translatewiki.net. [skins/MinervaNeue] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342395 (owner: 10Jdlrobson) [07:09:41] (03Merged) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/MobileFrontend] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342545 (owner: 10Matthias Mullie) [07:09:43] (03Merged) 10jenkins-bot: Localisation updates from https://translatewiki.net. [skins/MinervaNeue] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342546 (owner: 10Matthias Mullie) [07:09:44] (03Merged) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/MobileFrontend] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342547 (owner: 10Matthias Mullie) [07:10:11] !log jmm@cumin2003 DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Jmoore111 out of all services on: 2444 hosts [07:10:40] FIRING: [3x] SystemdUnitFailed: docker-system-prune-dangling.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:11:10] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1342398|Localisation updates from https://translatewiki.net.]], [[gerrit:1342396|Localisation updates from https://translatewiki.net.]], [[gerrit:1342395|Localisation updates from https://translatewiki.net.]], [[gerrit:1342545|Localisation updates from https://translatewiki.net.]], [[gerrit:1342546|Localisation updates from https://translatewiki.net.]], [07:11:10] [[gerrit:1342547|Localisation updates from https://translatewiki.net.]] [07:12:06] (03PS6) 10Federico Ceratto: mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) [07:13:02] (03CR) 10CI reject: [V:04-1] mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [07:14:13] (03CR) 10CWilliams: "Yep, I will do that." [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [07:17:56] (03CR) 10Federico Ceratto: "Maybe we can run it on the 3 hosts on port 3306 - or maybe it's safer to run it only on the 2 multiinstance hosts (setting the socket for " [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [07:18:22] !log mlitn@deploy1003 mlitn, jdlrobson: Backport for [[gerrit:1342398|Localisation updates from https://translatewiki.net.]], [[gerrit:1342396|Localisation updates from https://translatewiki.net.]], [[gerrit:1342395|Localisation updates from https://translatewiki.net.]], [[gerrit:1342545|Localisation updates from https://translatewiki.net.]], [[gerrit:1342546|Localisation updates from https://translatewiki.net.]], [[gerri [07:18:22] t:1342547|Localisation updates from https://translatewiki.net.]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:18:59] (03PS1) 10Ayounsi: prometheus-arelion-exporter: add missing shebang [puppet] - 10https://gerrit.wikimedia.org/r/1342552 [07:19:05] !log mlitn@deploy1003 mlitn, jdlrobson: Continuing with deployment [07:20:25] FIRING: [4x] SystemdUnitFailed: docker-system-prune-dangling.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:22:35] (03PS7) 10Federico Ceratto: mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) [07:24:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 18.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:24:21] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1342398|Localisation updates from https://translatewiki.net.]], [[gerrit:1342396|Localisation updates from https://translatewiki.net.]], [[gerrit:1342395|Localisation updates from https://translatewiki.net.]], [[gerrit:1342545|Localisation updates from https://translatewiki.net.]], [[gerrit:1342546|Localisation updates from https://translatewiki.net.]], [07:24:21] [[gerrit:1342547|Localisation updates from https://translatewiki.net.]] (duration: 13m 11s) [07:24:39] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1342552 (owner: 10Ayounsi) [07:25:12] !log Manually install gnmic 0.49.0 on netflow1004 - T438291 [07:25:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:25:16] T438291: Upgrade gNMIc to 0.49.0 - https://phabricator.wikimedia.org/T438291 [07:25:46] (03CR) 10Ayounsi: [C:03+2] prometheus-arelion-exporter: add missing shebang [puppet] - 10https://gerrit.wikimedia.org/r/1342552 (owner: 10Ayounsi) [07:34:47] (03CR) 10Kevin Bazira: "Thank you for working on this David. I've left a small comment : )" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341259 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [07:39:12] (03PS3) 10Tiziano Fogli: kafka-logging100[78]: disable icinga notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342553 (https://phabricator.wikimedia.org/T432444) [07:39:16] (03PS1) 10Tiziano Fogli: kafka-logging: remove kafka-logging100[12] [puppet] - 10https://gerrit.wikimedia.org/r/1342557 (https://phabricator.wikimedia.org/T432444) [07:39:18] (03PS9) 10Tiziano Fogli: kafka-logging: remove kafka-logging1003 [puppet] - 10https://gerrit.wikimedia.org/r/1329299 (https://phabricator.wikimedia.org/T432444) [07:39:21] (03PS13) 10Tiziano Fogli: kafka-logging: add kafka-logging100[7-8] to eqiad cluster [puppet] - 10https://gerrit.wikimedia.org/r/1327496 (https://phabricator.wikimedia.org/T432444) [07:39:23] (03PS3) 10DCausse: search: add semantic-highlighter to liftwing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341259 (https://phabricator.wikimedia.org/T433872) [07:39:24] (03PS4) 10Tiziano Fogli: kafka-logging1006: disable icinga notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342257 (https://phabricator.wikimedia.org/T432444) [07:39:26] (03PS9) 10Tiziano Fogli: kafka-logging: bring up kafka-logging1006 with node id 1006 [puppet] - 10https://gerrit.wikimedia.org/r/1329302 (https://phabricator.wikimedia.org/T432444) [07:39:27] (03CR) 10Marostegui: "> Maybe we can run it on the 3 hosts on port 3306 - or maybe it's safer to run it only on the 2 multiinstance hosts (setting the socket fo" [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [07:39:32] (03CR) 10DCausse: search: add semantic-highlighter to liftwing (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341259 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [07:39:53] (03CR) 10CI reject: [V:04-1] kafka-logging: add kafka-logging100[7-8] to eqiad cluster [puppet] - 10https://gerrit.wikimedia.org/r/1327496 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [07:44:50] (03PS5) 10Tiziano Fogli: kafka-logging1006: disable icinga notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342257 (https://phabricator.wikimedia.org/T432444) [07:44:50] (03PS10) 10Tiziano Fogli: kafka-logging: bring up kafka-logging1006 with node id 1006 [puppet] - 10https://gerrit.wikimedia.org/r/1329302 (https://phabricator.wikimedia.org/T432444) [07:44:50] (03PS4) 10Tiziano Fogli: kafka-logging100[78]: disable icinga notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342553 (https://phabricator.wikimedia.org/T432444) [07:44:51] (03PS14) 10Tiziano Fogli: kafka-logging: add kafka-logging100[7-8] to eqiad cluster [puppet] - 10https://gerrit.wikimedia.org/r/1327496 (https://phabricator.wikimedia.org/T432444) [07:44:52] (03PS2) 10Tiziano Fogli: kafka-logging: remove kafka-logging100[12] [puppet] - 10https://gerrit.wikimedia.org/r/1342557 (https://phabricator.wikimedia.org/T432444) [07:48:57] !log jnuche@deploy1003 Started deploy [releng/jenkins-deploy@8eaca67] (releasing): T438205 to backup host [07:49:25] !log jnuche@deploy1003 Finished deploy [releng/jenkins-deploy@8eaca67] (releasing): T438205 to backup host (duration: 00m 47s) [07:51:59] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12332140 (10isarantopoulos) [07:52:20] !log jnuche@deploy1003 Started deploy [releng/jenkins-deploy@8eaca67] (releasing): T438205 to prod host [07:52:21] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12332141 (10isarantopoulos) [07:52:36] (03CR) 10Marostegui: [C:03+2] Revert "dbbackups: Pool db2250:s5 to replace db2201:s5 for backups" [puppet] - 10https://gerrit.wikimedia.org/r/1342197 (owner: 10Marostegui) [07:52:58] !log jnuche@deploy1003 Finished deploy [releng/jenkins-deploy@8eaca67] (releasing): T438205 to prod host (duration: 00m 44s) [07:54:24] (03PS1) 10Marostegui: db2201: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342596 (https://phabricator.wikimedia.org/T437411) [07:54:42] (03CR) 10Federico Ceratto: "There is also one replica on 3306, db1164" [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [07:55:17] (03CR) 10Marostegui: "That host will be decommissioned (https://phabricator.wikimedia.org/T433475), you can ignore it." [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [07:55:59] (03PS2) 10Marostegui: db2201: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342596 (https://phabricator.wikimedia.org/T437411) [07:56:52] (03CR) 10Muehlenhoff: [C:03+2] Create repository components and sync definitions for nodejs 26 [puppet] - 10https://gerrit.wikimedia.org/r/1342277 (https://phabricator.wikimedia.org/T437510) (owner: 10Muehlenhoff) [07:57:09] (03CR) 10Marostegui: [C:03+2] db2201: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342596 (https://phabricator.wikimedia.org/T437411) (owner: 10Marostegui) [07:57:35] !log dropped non-links tables from db2245 (T437278) [07:57:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:57:38] T437278: Drop unneeded tables from x4 and s4 - https://phabricator.wikimedia.org/T437278 [07:59:24] !log dropped non-links tables from db1262 (T437278) [07:59:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:59:59] (03PS1) 10Marostegui: db2250: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342598 (https://phabricator.wikimedia.org/T437411) [08:00:04] jnuche and dduvall: OwO what's this, a deployment window?? MediaWiki train - Utc-0+Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T0800). nyaa~ [08:00:20] morning, train rolling out in a few minutes [08:01:27] !log dropped links tables from db2236 (T437278) [08:01:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:01:48] (03CR) 10Marostegui: [C:03+2] db2250: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342598 (https://phabricator.wikimedia.org/T437411) (owner: 10Marostegui) [08:02:11] 06SRE, 10Data-Persistence-Backup, 10database-backups, 13Patch-For-Review: Put db2201 back into backup production as a backup source - https://phabricator.wikimedia.org/T437411#12332165 (10Marostegui) db2201 has been added back in the backups rotation for s5. Once confirmed backups work fine, I have to remo... [08:02:54] (03PS8) 10Federico Ceratto: mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) [08:03:19] (03PS1) 10TrainBranchBot: group2 to 1.47.0-wmf.20 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342600 (https://phabricator.wikimedia.org/T430839) [08:03:22] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by jnuche@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342600 (https://phabricator.wikimedia.org/T430839) (owner: 10TrainBranchBot) [08:03:36] (03CR) 10Federico Ceratto: "Ok, I updated the script to run only on hosts with /run/mysqld/mysqld.m1.sock" [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [08:03:53] (03CR) 10Raymond Ndibe: [C:03+2] webservice-runner: default to 8000 only if PORT and TOOL_WEB_PORT has no value [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1310212 (https://phabricator.wikimedia.org/T432078) (owner: 10Raymond Ndibe) [08:03:55] !log dropped links tables from db2219 (T437278) [08:03:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:03:59] T437278: Drop unneeded tables from x4 and s4 - https://phabricator.wikimedia.org/T437278 [08:04:02] (03CR) 10Federico Ceratto: mariadb: Deploy backup_metrics.py (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [08:04:27] (03Merged) 10jenkins-bot: group2 to 1.47.0-wmf.20 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342600 (https://phabricator.wikimedia.org/T430839) (owner: 10TrainBranchBot) [08:04:37] (03Merged) 10jenkins-bot: webservice-runner: default to 8000 only if PORT and TOOL_WEB_PORT has no value [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1310212 (https://phabricator.wikimedia.org/T432078) (owner: 10Raymond Ndibe) [08:07:42] !log dropped links tables from db2206 (T437278) [08:07:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:08:52] that's good enough for now [08:10:35] !log imported nodejs_26.8.2-1nodesource1 to thirdparty/node26 for trixie-wikimedia T437510 [08:10:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:10:39] T437510: Add Nodejs 26 from Nodesource to Wikimedia apt - https://phabricator.wikimedia.org/T437510 [08:13:20] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Add Nodejs 26 from Nodesource to Wikimedia apt - https://phabricator.wikimedia.org/T437510#12332190 (10MoritzMuehlenhoff) 05Open→03Resolved A sync definition has been setup for future updates within the 26.x branch and 26.8.2 imported as the in... [08:13:57] !log jnuche@deploy1003 rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.20 refs T430839 [08:14:00] T430839: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839 [08:15:06] (03PS3) 10Hasan Akgün (WMDE): wikidata-query-builder: bump image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342195 (https://phabricator.wikimedia.org/T436532) [08:17:32] (03PS4) 10Hasan Akgün (WMDE): wikidata-query-builder: bump image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342195 [08:18:06] (03PS1) 10Muehlenhoff: Add config to build node26 images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1342602 (https://phabricator.wikimedia.org/T437509) [08:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [08:21:53] (03PS1) 10Brouberol: Only serve opensearch-{ipoid,ttmserver} traffic from dse-k8s-eqiad [dns] - 10https://gerrit.wikimedia.org/r/1342604 (https://phabricator.wikimedia.org/T438294) [08:24:42] (03PS2) 10Brouberol: Only serve opensearch-{ipoid,ttmserver} traffic from dse-k8s-eqiad [dns] - 10https://gerrit.wikimedia.org/r/1342604 (https://phabricator.wikimedia.org/T438294) [08:24:53] (03CR) 10Btullis: [C:03+1] Only serve opensearch-{ipoid,ttmserver} traffic from dse-k8s-eqiad [dns] - 10https://gerrit.wikimedia.org/r/1342604 (https://phabricator.wikimedia.org/T438294) (owner: 10Brouberol) [08:24:57] (03CR) 10Ozge: [C:03+1] search: add semantic-highlighter to liftwing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341259 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [08:26:16] (03CR) 10Brouberol: [C:03+2] Only serve opensearch-{ipoid,ttmserver} traffic from dse-k8s-eqiad [dns] - 10https://gerrit.wikimedia.org/r/1342604 (https://phabricator.wikimedia.org/T438294) (owner: 10Brouberol) [08:26:22] FIRING: GnmiInterfaceCountersDrop: ... [08:26:22] fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=fasw2-e16a-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [08:26:31] !log brouberol@dns1004 START - running authdns-update [08:26:59] (03CR) 10JMeybohm: [C:03+1] Remove obsolete nutcracker container image based on bullseye [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1341665 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [08:28:51] !log brouberol@dns1004 END - running authdns-update [08:29:28] (03CR) 10Kevin Bazira: [C:03+1] search: add semantic-highlighter to liftwing (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341259 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [08:34:26] !log Manually install gnmic 0.49.0 on netflow2005 - T438291 [08:34:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:34:29] T438291: Upgrade gNMIc to 0.49.0 - https://phabricator.wikimedia.org/T438291 [08:34:46] (03CR) 10Klausman: [C:03+2] service::catalog: Add tool-server [puppet] - 10https://gerrit.wikimedia.org/r/1337900 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [08:35:49] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete nutcracker container image based on bullseye [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1341665 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [08:37:56] !log pruned obsolete Bullseye image prometheus-nutcracker-exporter from the docker registry T416452 [08:37:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:37:59] T416452: Migrate Docker images running in Production away from Bullseye - https://phabricator.wikimedia.org/T416452 [08:38:08] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] "prometheus-nutcracker-exporter has been pruned from the registry." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1341665 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [08:39:15] (03CR) 10Marostegui: mariadb: Deploy backup_metrics.py (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [08:41:09] (03PS1) 10Klausman: role/kubernetes: Add tool-server stanza for LW tool-server [puppet] - 10https://gerrit.wikimedia.org/r/1342605 (https://phabricator.wikimedia.org/T436892) [08:41:56] (03PS8) 10CWilliams: mediabackups: Update versitygw systemd unit file to SIGHUP on reload [puppet] - 10https://gerrit.wikimedia.org/r/1321123 (https://phabricator.wikimedia.org/T430023) (owner: 10Jcrespo) [08:43:05] (03CR) 10Jelto: [C:03+2] trafficserver: add mapping for etherpad-next [puppet] - 10https://gerrit.wikimedia.org/r/1341893 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [08:43:16] (03CR) 10Btullis: [C:03+1] role/kubernetes: Add tool-server stanza for LW tool-server [puppet] - 10https://gerrit.wikimedia.org/r/1342605 (https://phabricator.wikimedia.org/T436892) (owner: 10Klausman) [08:43:36] (03CR) 10CWilliams: [C:03+2] mediabackups: Update versitygw systemd unit file to SIGHUP on reload [puppet] - 10https://gerrit.wikimedia.org/r/1321123 (https://phabricator.wikimedia.org/T430023) (owner: 10Jcrespo) [08:44:11] FIRING: TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [08:44:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:44:20] (03CR) 10Klausman: [C:03+2] role/kubernetes: Add tool-server stanza for LW tool-server [puppet] - 10https://gerrit.wikimedia.org/r/1342605 (https://phabricator.wikimedia.org/T436892) (owner: 10Klausman) [08:47:25] (03CR) 10Federico Ceratto: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [08:50:56] (03PS1) 10Dpogorzelski: mesh: Accept websocket upgrades in configuration 1.15.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342606 (https://phabricator.wikimedia.org/T437706) [08:50:59] (03PS1) 10Dpogorzelski: liftwing-studio: accept websocket upgrades at the tls terminator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342607 (https://phabricator.wikimedia.org/T437706) [08:51:17] (03PS9) 10Federico Ceratto: mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) [08:51:28] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12332382 (10Aklapper) Please try to connect to the server via `ssh -vvv` and include the commands being used. Thanks. [08:53:23] (03CR) 10Elukey: [C:03+1] Add config to build node26 images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1342602 (https://phabricator.wikimedia.org/T437509) (owner: 10Muehlenhoff) [08:56:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.17% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:56:41] (03PS10) 10Federico Ceratto: mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) [08:59:24] (03CR) 10Federico Ceratto: mariadb: Deploy backup_metrics.py (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [09:01:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:03:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:06:22] (03PS1) 10Dpogorzelski: Copy mesh.configuration 1.15.3 -> 1.15.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342611 (https://phabricator.wikimedia.org/T437706) [09:06:25] (03PS1) 10Dpogorzelski: mesh.configuration: Accept websocket upgrades [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342612 (https://phabricator.wikimedia.org/T437706) [09:06:27] (03PS1) 10Dpogorzelski: liftwing-studio: accept websocket upgrades at the tls terminator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342613 (https://phabricator.wikimedia.org/T437706) [09:08:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:12:15] (03CR) 10Federico Ceratto: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [09:23:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.72% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:29:19] (03Abandoned) 10Dpogorzelski: mesh: Accept websocket upgrades in configuration 1.15.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342606 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [09:35:28] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [09:35:57] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [09:36:19] (03CR) 10Arthur taylor: [C:03+2] "let's go!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342195 (owner: 10Hasan Akgün (WMDE)) [09:36:22] FIRING: [3x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [09:38:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.41% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:38:50] (03Abandoned) 10Dpogorzelski: liftwing-studio: accept websocket upgrades at the tls terminator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342607 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [09:38:55] (03Merged) 10jenkins-bot: wikidata-query-builder: bump image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342195 (owner: 10Hasan Akgün (WMDE)) [09:40:18] !log arthurtaylor@deploy1003 helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply [09:40:39] !log arthurtaylor@deploy1003 helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply [09:40:54] !log arthurtaylor@deploy1003 helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply [09:41:13] !log arthurtaylor@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply [09:41:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:41:20] !log arthurtaylor@deploy1003 helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply [09:41:41] !log arthurtaylor@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply [09:43:02] (03PS2) 10Dpogorzelski: mesh.configuration: Copy 1.15.3 -> 1.15.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342611 (https://phabricator.wikimedia.org/T437706) [09:43:02] (03PS2) 10Dpogorzelski: mesh.configuration: Accept websocket upgrades [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342612 (https://phabricator.wikimedia.org/T437706) [09:43:02] (03PS2) 10Dpogorzelski: liftwing-studio: accept websocket upgrades at the tls terminator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342613 (https://phabricator.wikimedia.org/T437706) [09:43:44] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 487687320 and 46 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [09:51:32] !log marostegui@cumin1004 dbctl commit (dc=all): 'Fix weights T436496', diff saved to https://phabricator.wikimedia.org/P96465 and previous config saved to /var/cache/conftool/dbconfig/20260917-095131-marostegui.json [09:51:36] T436496: Pre switchover: Database preparations - https://phabricator.wikimedia.org/T436496 [09:51:40] (03CR) 10Ilias Sarantopoulos: [C:03+1] "lgtm! let's go!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342199 (https://phabricator.wikimedia.org/T436644) (owner: 10Gkyziridis) [09:52:36] !log marostegui@cumin1004 dbctl commit (dc=all): 'Fix weights T436496', diff saved to https://phabricator.wikimedia.org/P96466 and previous config saved to /var/cache/conftool/dbconfig/20260917-095235-marostegui.json [09:54:30] 06SRE, 10SRE-swift-storage, 10Pontoon, 06Traffic: /etc/envoy/envoy.yaml empty on pontoon hosts (and thus envoy doesn't work) - https://phabricator.wikimedia.org/T438309 (10MatthewVernon) 03NEW [09:54:44] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 0 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [09:59:10] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12332678 (10MoritzMuehlenhoff) >>! In T435354#12329492, @VRiley-WMF wrote: > /dev/mapper/vg0-srv > /dev/mapper/vg0-swap > /dev/ttyS1 > The logs do not indicate a failure of these devices, but the delay... [09:59:56] RECOVERY - LDAP -writable server- on ldap-rw1001 is OK: LDAP OK - 0.010 seconds response time https://wikitech.wikimedia.org/wiki/LDAP%23Troubleshooting [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1000) [10:00:42] RECOVERY - LDAP -writable server- on ldap-rw2001 is OK: LDAP OK - 0.102 seconds response time https://wikitech.wikimedia.org/wiki/LDAP%23Troubleshooting [10:01:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:01:31] 06SRE, 06Infrastructure-Foundations: Provide a live Debian image to act as a console - https://phabricator.wikimedia.org/T78135#12332682 (10fgiunchedi) [10:02:20] 06SRE, 06Infrastructure-Foundations: Provide a live Debian image to act as a console - https://phabricator.wikimedia.org/T78135#12332687 (10fgiunchedi) >>! In T78135#10591609, @jhathaway wrote: > I found this task while pondering similar functionality, as I have been using SystemRescue to troubleshoot some iss... [10:02:28] (03CR) 10Blake: [C:03+2] mediawiki: remove lamp.deployment. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342304 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:03:03] going to do a helm-only deployment in a few minutes to pick up the cleanup above, should be a no-op [10:04:32] (03PS2) 10Gmodena: wdqs: extend envoy rq_time histogram buckets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) [10:04:48] (03CR) 10Gmodena: wdqs: extend envoy rq_time histogram buckets (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) (owner: 10Gmodena) [10:05:40] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-ml_serve_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:05:46] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 271986184 and 6 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [10:06:37] (03Merged) 10jenkins-bot: mediawiki: remove lamp.deployment. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342304 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:07:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.49% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:07:26] !log blake@deploy1003 Started scap sync-world: cleanup for T417800 [10:07:29] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [10:09:38] 06SRE, 06Infrastructure-Foundations: Provide an option menu when booting via PXE - https://phabricator.wikimedia.org/T191018#12332697 (10fgiunchedi) 05Open→03Declined Declining, we're going another route in {T78135} to select the kernel/initramfs to boot [10:10:53] (03CR) 10Trueg: [C:03+1] wdqs: extend envoy rq_time histogram buckets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) (owner: 10Gmodena) [10:10:56] !log blake@deploy1003 Finished scap sync-world: cleanup for T417800 (duration: 03m 57s) [10:11:53] (03PS1) 10Elukey: Deploy kubernetes credentials on build2004 for docker-report [puppet] - 10https://gerrit.wikimedia.org/r/1342620 [10:13:53] (03CR) 10Trueg: [C:03+1] "lgtm." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342298 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [10:15:32] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1342620 (owner: 10Elukey) [10:16:15] (03CR) 10Elukey: [C:03+2] Deploy kubernetes credentials on build2004 for docker-report [puppet] - 10https://gerrit.wikimedia.org/r/1342620 (owner: 10Elukey) [10:17:27] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:18:44] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 255296 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [10:24:21] (03CR) 10Gkyziridis: [C:03+2] ml-services: Deployment of Qwen3.8-27B-FP8 model under llm namespace. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342199 (https://phabricator.wikimedia.org/T436644) (owner: 10Gkyziridis) [10:25:25] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:26:20] (03PS3) 10Jelto: cache-text: set caching for etherpad-next "websockets" [puppet] - 10https://gerrit.wikimedia.org/r/1341894 (https://phabricator.wikimedia.org/T435509) [10:26:45] (03Merged) 10jenkins-bot: ml-services: Deployment of Qwen3.8-27B-FP8 model under llm namespace. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342199 (https://phabricator.wikimedia.org/T436644) (owner: 10Gkyziridis) [10:28:57] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:31:44] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 108482248 and 14 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [10:32:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:37:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:37:47] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [10:41:44] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 27776 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [10:42:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:46:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:51:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.11% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:53:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:58:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:59:32] (03PS1) 10Krinkle: MathMathML: Simplify Mathoid fallback/a11y class logic [extensions/Math] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342626 (https://phabricator.wikimedia.org/T436026) [10:59:50] (03PS1) 10Krinkle: ext.math.mathjax: Implement mwe-math-mathml-a11y for client-side MathJax [extensions/Math] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342627 (https://phabricator.wikimedia.org/T436026) [11:09:19] 06SRE, 06ServiceOps, 10ServiceOps-MediaWiki: page error 404 missing when url ends with .php - https://phabricator.wikimedia.org/T425239#12332919 (10MLechvien-WMF) [11:43:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:47:25] 06SRE, 06ServiceOps, 06serviceops-deprecated, 07Datacenter-Switchover: Find a way to verify mediawiki-config IPs ahead of datacenter switchovers - https://phabricator.wikimedia.org/T163354#12333121 (10MLechvien-WMF) [11:47:26] 06SRE, 10Observability-Logging, 06ServiceOps, 06serviceops-deprecated, 07Datacenter-Switchover: Figure out switchover steps for mwlog hosts - https://phabricator.wikimedia.org/T261274#12333122 (10MLechvien-WMF) [11:47:28] 06SRE, 06ServiceOps, 06serviceops-deprecated, 07Datacenter-Switchover: Services without a service IP cannot automatically be switched by the switchdc cookbook - https://phabricator.wikimedia.org/T285707#12333124 (10MLechvien-WMF) [11:47:31] 06SRE, 06ServiceOps, 06serviceops-deprecated, 07Datacenter-Switchover: Updates to warmup script (2020-2021) - https://phabricator.wikimedia.org/T269179#12333125 (10MLechvien-WMF) [11:48:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:54:20] 06SRE, 06ServiceOps, 10ServiceOps-MediaWiki: page error 404 missing when url ends with .php - https://phabricator.wikimedia.org/T425239#12333138 (10Krinkle) > if is without the /w it will always redirect to a blank page with just a tiny "File not found." If it has the /w/ the index.php will redirect to main... [11:54:50] 06SRE, 06ServiceOps, 10ServiceOps-MediaWiki, 10WMF-General-or-Unknown, 07Regression: Custom 404 error page should apply to *.php - https://phabricator.wikimedia.org/T91474#12333139 (10Krinkle) [11:54:59] 06SRE, 06ServiceOps, 10ServiceOps-MediaWiki, 10WMF-General-or-Unknown, 07Regression: Custom 404 error page should apply to *.php - https://phabricator.wikimedia.org/T91474#12333142 (10Krinkle) [11:55:40] 06SRE, 06ServiceOps, 10ServiceOps-MediaWiki, 10WMF-General-or-Unknown, 07Regression: Custom 404 error page should apply to *.php - https://phabricator.wikimedia.org/T91474#12333146 (10Krinkle) [11:57:09] (03PS1) 10Majavah: P:wmcs::kubeadm::control: Remove maintain-kubeusers clone [puppet] - 10https://gerrit.wikimedia.org/r/1342634 [11:57:09] (03PS1) 10Majavah: kubeadm: Drop support for tuning stacked etcd instances [puppet] - 10https://gerrit.wikimedia.org/r/1342635 (https://phabricator.wikimedia.org/T437219) [11:57:12] (03PS1) 10Majavah: P:wmcs: Make kubeadm etcd profile more generic [puppet] - 10https://gerrit.wikimedia.org/r/1342636 (https://phabricator.wikimedia.org/T437219) [11:57:15] (03PS1) 10Majavah: P:wmcs::etcd: Remove wide etcd metrics firewall term [puppet] - 10https://gerrit.wikimedia.org/r/1342637 [12:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1200) [12:03:46] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [extensions/Math] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342626 (https://phabricator.wikimedia.org/T436026) (owner: 10Krinkle) [12:03:47] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [extensions/Math] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342627 (https://phabricator.wikimedia.org/T436026) (owner: 10Krinkle) [12:05:20] (03Merged) 10jenkins-bot: MathMathML: Simplify Mathoid fallback/a11y class logic [extensions/Math] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342626 (https://phabricator.wikimedia.org/T436026) (owner: 10Krinkle) [12:05:41] (03Merged) 10jenkins-bot: ext.math.mathjax: Implement mwe-math-mathml-a11y for client-side MathJax [extensions/Math] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342627 (https://phabricator.wikimedia.org/T436026) (owner: 10Krinkle) [12:05:56] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1342626|MathMathML: Simplify Mathoid fallback/a11y class logic (T436026)]], [[gerrit:1342627|ext.math.mathjax: Implement mwe-math-mathml-a11y for client-side MathJax (T436026)]] [12:06:01] T436026: Include a11y MathML in client MathJax mode - https://phabricator.wikimedia.org/T436026 [12:06:04] (03PS1) 10Btullis: ceph: Add the ceph::admin role and its two hosts [puppet] - 10https://gerrit.wikimedia.org/r/1342644 (https://phabricator.wikimedia.org/T435608) [12:07:27] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9446/co" [puppet] - 10https://gerrit.wikimedia.org/r/1342635 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [12:08:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:10:25] RESOLVED: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:10:25] RESOLVED: [3x] SystemdUnitFailed: docker-reporter-kubernetes-ml_serve_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:10:33] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1342626|MathMathML: Simplify Mathoid fallback/a11y class logic (T436026)]], [[gerrit:1342627|ext.math.mathjax: Implement mwe-math-mathml-a11y for client-side MathJax (T436026)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:11:23] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342653 [12:11:24] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:11:32] (03PS9) 10Elukey: docker_registry: remove the Swift support [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) [12:13:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:14:31] !log krinkle@deploy1003 krinkle: Continuing with deployment [12:14:42] (03PS1) 10Muehlenhoff: Inline profile::docker::reporter::credentials and use the same Hiera option [puppet] - 10https://gerrit.wikimedia.org/r/1342656 (https://phabricator.wikimedia.org/T435314) [12:17:29] 06SRE, 06Infrastructure-Foundations: Segfault for systemd-sysusers.service on stat1007 - https://phabricator.wikimedia.org/T256098#12333282 (10elukey) 05Open→03Invalid [12:17:57] (03CR) 10DCausse: [C:03+2] search: add semantic-highlighter to liftwing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341259 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [12:19:01] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1342626|MathMathML: Simplify Mathoid fallback/a11y class logic (T436026)]], [[gerrit:1342627|ext.math.mathjax: Implement mwe-math-mathml-a11y for client-side MathJax (T436026)]] (duration: 13m 04s) [12:19:05] T436026: Include a11y MathML in client MathJax mode - https://phabricator.wikimedia.org/T436026 [12:19:11] 06SRE, 06Infrastructure-Foundations, 10netops: Errors for ifup@ens5.service after rebooting Ganeti VMs - https://phabricator.wikimedia.org/T273026#12333292 (10elukey) 05Open→03Invalid [12:19:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.54% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:20:01] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1180.eqiad.wmnet - https://phabricator.wikimedia.org/T437222#12333300 (10Jclark-ctr) a:03Jclark-ctr [12:20:23] (03Merged) 10jenkins-bot: search: add semantic-highlighter to liftwing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341259 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [12:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [12:23:56] RESOLVED: TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [12:25:24] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1180.eqiad.wmnet - https://phabricator.wikimedia.org/T437222#12333333 (10Jclark-ctr) 05Open→03Resolved [12:26:51] (03CR) 10Filippo Giunchedi: "LGTM overall, what is the plan/intention with respect to severity? I see all alerts are warnings now, are some/all going to be promoted to" [alerts] - 10https://gerrit.wikimedia.org/r/1342349 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [12:27:58] (03CR) 10Filippo Giunchedi: "Relatively minor though for the runbook at https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Magnum please consider shi" [alerts] - 10https://gerrit.wikimedia.org/r/1342349 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [12:28:45] (03CR) 10FNegri: [C:03+1] P:wmcs::kubeadm::control: Remove maintain-kubeusers clone [puppet] - 10https://gerrit.wikimedia.org/r/1342634 (owner: 10Majavah) [12:29:29] (03CR) 10FNegri: [C:03+1] "LGTM. the PCC diff is related to the previous patch in the stack." [puppet] - 10https://gerrit.wikimedia.org/r/1342635 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [12:30:22] (03PS1) 10Muehlenhoff: Add explicit hostnames to LDAP cert SNIs [puppet] - 10https://gerrit.wikimedia.org/r/1342660 (https://phabricator.wikimedia.org/T331699) [12:30:33] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342656 (https://phabricator.wikimedia.org/T435314) (owner: 10Muehlenhoff) [12:30:43] (03CR) 10Majavah: [C:03+2] P:wmcs::kubeadm::control: Remove maintain-kubeusers clone [puppet] - 10https://gerrit.wikimedia.org/r/1342634 (owner: 10Majavah) [12:31:14] (03CR) 10Majavah: [V:03+1 C:03+2] kubeadm: Drop support for tuning stacked etcd instances [puppet] - 10https://gerrit.wikimedia.org/r/1342635 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [12:32:49] 06SRE, 10ServiceOps-Upgrades-Hardware, 07Essential-Work, 06ServiceOps (Next quarter): mc20[56-73] implementation tracking - https://phabricator.wikimedia.org/T436273#12333403 (10MLechvien-WMF) [12:34:56] (03PS2) 10Muehlenhoff: Add explicit hostnames to LDAP cert SNIs [puppet] - 10https://gerrit.wikimedia.org/r/1342660 (https://phabricator.wikimedia.org/T331699) [12:35:42] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [12:38:55] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (NOOP 2 CORE_DIFF 3): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1342636 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [12:39:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:39:37] (03PS2) 10Muehlenhoff: Inline profile::docker::reporter::credentials and use the same Hiera option [puppet] - 10https://gerrit.wikimedia.org/r/1342656 (https://phabricator.wikimedia.org/T435314) [12:40:57] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [12:40:59] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [12:41:31] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342656 (https://phabricator.wikimedia.org/T435314) (owner: 10Muehlenhoff) [12:42:33] !log jclark@cumin1004 START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [12:44:02] (03PS3) 10Muehlenhoff: Inline profile::docker::reporter::credentials and use the same Hiera option [puppet] - 10https://gerrit.wikimedia.org/r/1342656 (https://phabricator.wikimedia.org/T435314) [12:44:58] (03CR) 10Elukey: [C:03+1] transports.clustershell: report also empty outputs [software/cumin] - 10https://gerrit.wikimedia.org/r/1341027 (owner: 10Volans) [12:47:04] (03CR) 10Marostegui: mariadb: Deploy backup_metrics.py (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [12:47:43] 06SRE, 10LDAP-Access-Requests: Grant Access to for - https://phabricator.wikimedia.org/T438338 (10Ricardomota) 03NEW [12:47:52] (03PS1) 10DCausse: search: fine-tune semantic-highlighting resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342668 (https://phabricator.wikimedia.org/T433872) [12:48:51] (03PS1) 10Brouberol: Revert "Only serve opensearch-{ipoid,ttmserver} traffic from dse-k8s-eqiad" [dns] - 10https://gerrit.wikimedia.org/r/1342669 [12:49:02] 06SRE, 10LDAP-Access-Requests: Grant Access to Grafana for Ricardomota - https://phabricator.wikimedia.org/T438338#12333464 (10Ricardomota) [12:49:02] (03CR) 10Ozge: [C:03+1] search: fine-tune semantic-highlighting resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342668 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [12:49:59] (03CR) 10DCausse: [C:03+2] search: fine-tune semantic-highlighting resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342668 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [12:50:01] jclark@cumin1004 provision (PID 3225527) is awaiting input [12:50:42] (03CR) 10Elukey: [C:03+1] puppetdb backend: JSON-encode query interpolations [software/cumin] - 10https://gerrit.wikimedia.org/r/1341029 (owner: 10Volans) [12:51:48] (03CR) 10Elukey: [C:03+1] query: detect circular alias references [software/cumin] - 10https://gerrit.wikimedia.org/r/1341030 (owner: 10Volans) [12:52:24] (03CR) 10Elukey: [C:03+1] puppetdb backend: reset endpoint on each build [software/cumin] - 10https://gerrit.wikimedia.org/r/1341031 (owner: 10Volans) [12:52:27] (03Merged) 10jenkins-bot: search: fine-tune semantic-highlighting resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342668 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [12:53:30] (03CR) 10Elukey: [C:03+1] backends: use pyparsing PEP8 names [software/cumin] - 10https://gerrit.wikimedia.org/r/1341032 (owner: 10Volans) [12:53:40] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [12:55:27] (03CR) 10Elukey: [C:03+1] transports: disable Tqdm monitor thread (031 comment) [software/cumin] - 10https://gerrit.wikimedia.org/r/1341033 (owner: 10Volans) [12:55:44] (03CR) 10Elukey: [C:03+1] tests: disable deprecation warning [software/cumin] - 10https://gerrit.wikimedia.org/r/1341034 (owner: 10Volans) [12:55:48] !log jclark@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [12:59:33] (03CR) 10Jforrester: [C:03+1] Add config to build node26 images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1342602 (https://phabricator.wikimedia.org/T437509) (owner: 10Muehlenhoff) [12:59:38] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations: Raise the max bucket size limit on apus for the Docker Registry - https://phabricator.wikimedia.org/T438339 (10elukey) 03NEW [12:59:47] jouncebot: refresh [12:59:48] I refreshed my knowledge about deployments. [12:59:52] jouncebot: nowandnext [12:59:52] For the next 0 hour(s) and 0 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1200) [12:59:52] In 0 hour(s) and 0 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1300) [13:00:04] Lucas_WMDE, urbanecm, and TheresNoTime: How many deployers does it take to do UTC afternoon backport window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1300). [13:00:05] cscott: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:01:55] 06SRE, 10LDAP-Access-Requests: Grant Access to Grafana for Ricardomota - https://phabricator.wikimedia.org/T438338#12333512 (10Ricardomota) [13:02:44] (03PS1) 10Elukey: sre.hosts.provision: fix ipmi warning [cookbooks] - 10https://gerrit.wikimedia.org/r/1342672 [13:03:48] I am going to restart the CI Jenkins to update plugins [13:08:34] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12333559 (10isarantopoulos) a:05isarantopoulos→03klausman thanks for creating this task! re-assigning Tobias as the technical contact for this. ML... [13:08:55] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12333562 (10isarantopoulos) [13:09:55] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:09:57] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:10:22] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12333566 (10klausman) Racking details look good, Rob. I think the power/cooling constraints are already tighter than what we care about placement-wise... [13:11:55] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:11:57] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:13:41] (03PS2) 10Elukey: sre.hosts.provision: fix ipmi warning [cookbooks] - 10https://gerrit.wikimedia.org/r/1342672 [13:15:52] (03PS1) 10Ozge: ml-services: Lower cpu for llm namespace applications [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342675 (https://phabricator.wikimedia.org/T438342) [13:16:36] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to analytics-privatedata-users level 3 for derenrich - https://phabricator.wikimedia.org/T437668#12333591 (10tappof) [13:18:44] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to analytics-privatedata-users level 3 for derenrich - https://phabricator.wikimedia.org/T437668#12333611 (10tappof) ` $ sudo manage_principals.py create derenrich --email=derenrich@wikimedia... [13:19:14] (03CR) 10Kamila Součková: [C:03+1] deployment_server: run scap clean-images daily [puppet] - 10https://gerrit.wikimedia.org/r/1342337 (https://phabricator.wikimedia.org/T438148) (owner: 10Ahmon Dancy) [13:21:46] (03CR) 10Gmodena: [C:03+2] wdqs: set QLEVER_NUM_QUERIES to 2x allocated CPU cores [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342298 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [13:22:09] (03PS1) 10Majavah: Add cloudinfra backup pass [labs/private] - 10https://gerrit.wikimedia.org/r/1342676 (https://phabricator.wikimedia.org/T301640) [13:22:37] (03CR) 10Majavah: [V:03+2 C:03+2] Add cloudinfra backup pass [labs/private] - 10https://gerrit.wikimedia.org/r/1342676 (https://phabricator.wikimedia.org/T301640) (owner: 10Majavah) [13:22:53] (03CR) 10Elukey: [C:03+2] sre.hosts.provision: fix ipmi warning [cookbooks] - 10https://gerrit.wikimedia.org/r/1342672 (owner: 10Elukey) [13:24:15] (03Merged) 10jenkins-bot: wdqs: set QLEVER_NUM_QUERIES to 2x allocated CPU cores [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342298 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [13:24:40] !log jclark@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1016 [13:24:48] !log jclark@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1016 [13:25:40] (03CR) 10Lerickson: [C:03+2] wdqs: extend envoy rq_time histogram buckets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) (owner: 10Gmodena) [13:26:25] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:26:31] (03PS1) 10Medelius: enwiki desktop VE: add education popup for switching to source editor [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342678 (https://phabricator.wikimedia.org/T434249) [13:26:54] (03CR) 10Lerickson: [V:03+2 C:03+2] wdqs: extend envoy rq_time histogram buckets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) (owner: 10Gmodena) [13:27:22] (03CR) 10Lerickson: [V:03+2 C:03+1] "Sorry, meant to +1 so you can control when to merge." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) (owner: 10Gmodena) [13:27:40] (03CR) 10Brouberol: [C:03+1] "Nice and tidy" [puppet] - 10https://gerrit.wikimedia.org/r/1342644 (https://phabricator.wikimedia.org/T435608) (owner: 10Btullis) [13:28:31] (03CR) 10Gmodena: [C:03+2] wdqs: extend envoy rq_time histogram buckets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) (owner: 10Gmodena) [13:29:37] !log apus - radosgw-admin quota set --quota-scope=user --uid=docker-registry --max-size=5T T438339 [13:29:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:29:40] T438339: Raise the max bucket size limit on apus for the Docker Registry - https://phabricator.wikimedia.org/T438339 [13:30:14] (03CR) 10DCausse: [C:03+1] ml-services: Lower cpu for llm namespace applications [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342675 (https://phabricator.wikimedia.org/T438342) (owner: 10Ozge) [13:30:19] (03PS1) 10Elukey: sre.hosts.provision: allow re-provisioning ml-serve1016 [cookbooks] - 10https://gerrit.wikimedia.org/r/1342679 (https://phabricator.wikimedia.org/T426180) [13:30:44] (03PS2) 10Tiziano Fogli: admin: add derenrich to analytics-privatedata-users (krb) [puppet] - 10https://gerrit.wikimedia.org/r/1339911 (https://phabricator.wikimedia.org/T437668) (owner: 10Ssingh) [13:30:52] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:31:01] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:31:02] 06SRE, 10SRE-swift-storage, 10Ceph, 06Data-Persistence: Raise the max bucket size limit on apus for the Docker Registry - https://phabricator.wikimedia.org/T438339#12333713 (10MatthewVernon) 05Open→03Resolved a:03MatthewVernon [it's best to use sre-swift-storage for these requests] I've bumped t... [13:31:12] (03Merged) 10jenkins-bot: wdqs: extend envoy rq_time histogram buckets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) (owner: 10Gmodena) [13:31:40] (03CR) 10Ozge: [C:03+2] ml-services: Lower cpu for llm namespace applications [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342675 (https://phabricator.wikimedia.org/T438342) (owner: 10Ozge) [13:33:32] (03PS2) 10Elukey: sre.hosts.provision: allow re-provisioning ml-serve1016 [cookbooks] - 10https://gerrit.wikimedia.org/r/1342679 (https://phabricator.wikimedia.org/T426180) [13:33:54] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:34:02] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:34:09] (03Merged) 10jenkins-bot: ml-services: Lower cpu for llm namespace applications [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342675 (https://phabricator.wikimedia.org/T438342) (owner: 10Ozge) [13:34:54] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:35:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:36:01] 06SRE, 10LDAP-Access-Requests: Grant Access to Grafana for Ricardomota - https://phabricator.wikimedia.org/T438338#12333754 (10Aklapper) 05Open→03Invalid Hi @Ricardomota. Please follow https://phabricator.wikimedia.org/project/profile/1564/ for `ldap/wmf` and `logstash-access` membership. And per https... [13:36:18] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342678 (https://phabricator.wikimedia.org/T434249) (owner: 10Medelius) [13:36:28] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342362 (https://phabricator.wikimedia.org/T436574) (owner: 10Medelius) [13:36:37] FIRING: [3x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [13:37:13] would it be alright if I backport two patches? i can deploy! [13:37:18] happy to wait til next window too [13:37:45] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [13:43:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by caro@deploy1003 using scap backport" [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342678 (https://phabricator.wikimedia.org/T434249) (owner: 10Medelius) [13:43:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by caro@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342362 (https://phabricator.wikimedia.org/T436574) (owner: 10Medelius) [13:44:00] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12333798 (10elukey) To keep archives happy - new big ML nodes need adjustments like https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1342679 [13:45:22] (03Merged) 10jenkins-bot: Make VE the default editor on enwiki desktop [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342362 (https://phabricator.wikimedia.org/T436574) (owner: 10Medelius) [13:45:25] 06SRE-OnFire, 06MediaWiki-Engineering, 06ServiceOps, 06SRE Observability, 07Sustainability (Incident Followup): Reduce the amount of messages sent through channel:Memcached during failures - https://phabricator.wikimedia.org/T390529#12333802 (10Krinkle) My understanding is that to a first approximation n... [13:45:36] cmede: go ahead I think [13:45:48] (03Merged) 10jenkins-bot: enwiki desktop VE: add education popup for switching to source editor [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342678 (https://phabricator.wikimedia.org/T434249) (owner: 10Medelius) [13:45:55] ah, it was just waiting for CI ^^ [13:46:00] :) [13:46:05] !log caro@deploy1003 Started scap sync-world: Backport for [[gerrit:1342678|enwiki desktop VE: add education popup for switching to source editor (T434249)]], [[gerrit:1342362|Make VE the default editor on enwiki desktop (T436574)]] [13:46:11] T434249: Design user experience for helping newcomers discover source editing (desktop, en.wiki) - https://phabricator.wikimedia.org/T434249 [13:46:11] T436574: Make VisualEditor the default editor on enwiki's desktop web for especially newbs and TAs - https://phabricator.wikimedia.org/T436574 [13:47:27] cmede: Can you ping me once you're done? [13:47:31] sure thing [13:47:36] (03CR) 10Herron: [C:03+1] kafka-logging: remove kafka-logging1003 [puppet] - 10https://gerrit.wikimedia.org/r/1329299 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [13:47:38] (03PS1) 10Majavah: Add new role for cloudinfra database backups [puppet] - 10https://gerrit.wikimedia.org/r/1342681 (https://phabricator.wikimedia.org/T301640) [13:47:55] (FWIW cscott also wanted to deploy but doesn’t seem to be onine atm) [13:48:02] oh, nevermind, my tab complete lied to me [13:48:35] No you're right, I'd forgotten! [13:48:40] But I am here now [13:49:02] Is there still time in the window for some config patches? [13:49:03] (03CR) 10Herron: [C:03+1] kafka-logging: bring up kafka-logging1006 with node id 1006 [puppet] - 10https://gerrit.wikimedia.org/r/1329302 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [13:49:05] cmede is deploying at the moment [13:49:14] and I think the window will be over then but there’s at least 30 minutes until the next one [13:49:26] Msz2001: how urgent is your deployment? ^^ [13:49:30] My patches aren't super critical, they can slide to the late window no problem [13:49:36] Not urgent [13:49:53] ah, sorry cscott! [13:49:54] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:50:07] (03CR) 10Herron: [C:03+1] kafka-logging: add kafka-logging100[7-8] to eqiad cluster [puppet] - 10https://gerrit.wikimedia.org/r/1327496 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [13:50:13] then my suggestion would be for cscott to deploy first after cmede (just extend the window into the break), and then Msz2001 [13:50:21] if cscott’s config changes can all go together then that should be plenty of time [13:50:30] Yeah they can [13:50:30] (03CR) 10Herron: [C:03+1] kafka-logging1006: disable icinga notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342257 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [13:50:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:50:49] (03CR) 10Herron: [C:03+1] kafka-logging100[78]: disable icinga notifications [puppet] - 10https://gerrit.wikimedia.org/r/1342553 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [13:51:01] (03CR) 10CI reject: [V:04-1] Add new role for cloudinfra database backups [puppet] - 10https://gerrit.wikimedia.org/r/1342681 (https://phabricator.wikimedia.org/T301640) (owner: 10Majavah) [13:52:44] (03CR) 10Herron: "LGTM as a future step, would you be ok adding a self -2 "do not merge" until the time we're ready to proceed?" [puppet] - 10https://gerrit.wikimedia.org/r/1342557 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [13:54:54] (03CR) 10Andrew Bogott: "I like this idea! Will implement in a followup." [alerts] - 10https://gerrit.wikimedia.org/r/1342349 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [13:55:07] (03CR) 10Btullis: [C:03+2] ceph: Add the ceph::admin role and its two hosts [puppet] - 10https://gerrit.wikimedia.org/r/1342644 (https://phabricator.wikimedia.org/T435608) (owner: 10Btullis) [13:58:27] 06SRE, 10SRE-Access-Requests: Requesting access to Superset Dashboard for Hany EL Mokadem - https://phabricator.wikimedia.org/T438034#12333883 (10tappof) @KFrancis The requesting user doesn't have an NDA on file. Could you please get in touch with him to provide one? Thanks! [13:59:14] VE finally the default editor on enwiki. I thought I'd never see the day. [13:59:30] woohooo [13:59:41] given ten years, we can do anything! [14:00:22] just imagine what we'll do in twenty [14:00:32] (03CR) 10Michael Große: [C:03+1] JsonSchemaBuilder: Cache the root schema in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342265 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [14:00:39] (03CR) 10Michael Große: [C:03+1] JsonSchemaBuilder: Cache the root schema in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342264 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [14:00:48] cscott: omg :o [14:00:48] maybe turn on collaborative editing [14:01:03] that code has also been sitting around for 10 years [14:01:06] cscott: Maybe get rid of the mobile site. [14:01:20] If we’re talking about efforts that have been stalled for decades. [14:01:33] oh man, mobilefrontend [14:03:12] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12333924 (10tappof) I'm putting the task on hold, awaiting user input, since we need the user's SSH key. @EAlbizzati-WMF, please update the task once you've generated the key. Cheers [14:04:45] (03PS2) 10Mszwarc: SI: Preserve the username filter when switching queues [extensions/CheckUser] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342684 (https://phabricator.wikimedia.org/T438308) [14:06:20] !log caro@deploy1003 caro: Backport for [[gerrit:1342678|enwiki desktop VE: add education popup for switching to source editor (T434249)]], [[gerrit:1342362|Make VE the default editor on enwiki desktop (T436574)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:06:25] T434249: Design user experience for helping newcomers discover source editing (desktop, en.wiki) - https://phabricator.wikimedia.org/T434249 [14:06:25] T436574: Make VisualEditor the default editor on enwiki's desktop web for especially newbs and TAs - https://phabricator.wikimedia.org/T436574 [14:07:49] !log caro@deploy1003 caro: Continuing with deployment [14:12:13] ohh, the backport touches i18n, that’s why this is taking so long [14:12:23] yeah.. [14:13:40] 06SRE, 10SRE-Access-Requests: Requesting access to Superset Dashboard for Hany EL Mokadem - https://phabricator.wikimedia.org/T438034#12333998 (10tappof) @RStallman-legalteam @NAramayo-WMF [14:15:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.5% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:17:27] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:19:07] (03CR) 10Elukey: [C:03+2] docker_registry: remove the Swift support [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [14:19:27] Just wondering, will anything happen if we step into the Test Kitchen window at :30? From the description, it seems to me that nothing is actually deployed at that time (in a sense like during the backport window) [14:19:57] !log caro@deploy1003 Finished scap sync-world: Backport for [[gerrit:1342678|enwiki desktop VE: add education popup for switching to source editor (T434249)]], [[gerrit:1342362|Make VE the default editor on enwiki desktop (T436574)]] (duration: 33m 52s) [14:20:02] T434249: Design user experience for helping newcomers discover source editing (desktop, en.wiki) - https://phabricator.wikimedia.org/T434249 [14:20:02] T436574: Make VisualEditor the default editor on enwiki's desktop web for especially newbs and TAs - https://phabricator.wikimedia.org/T436574 [14:20:14] i'm all set. thanks for your patience the last 30 min/10 years [14:20:56] ok, i'll throw my patches in there. no i18n = should be quick? [14:21:03] are config patches running <10min? [14:21:38] I guess they usually are singe-digit quick [14:24:26] 06SRE, 10LDAP-Access-Requests: Request for LDAP NDA Access for CentralNotice Metrics for Yahya - https://phabricator.wikimedia.org/T436294#12334060 (10tappof) 05Open→03Resolved a:03tappof [14:24:30] (03PS3) 10C. Scott Ananian: Turn on variant conversion for PageAssessments [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340216 (https://phabricator.wikimedia.org/T328012) [14:25:13] ok, we'll see! [14:25:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cscott@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1331830 (https://phabricator.wikimedia.org/T436398) (owner: 10C. Scott Ananian) [14:25:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cscott@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340216 (https://phabricator.wikimedia.org/T328012) (owner: 10C. Scott Ananian) [14:25:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cscott@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341949 (https://phabricator.wikimedia.org/T437917) (owner: 10Subramanya Sastry) [14:25:35] 06SRE, 10LDAP-Access-Requests: Request for LDAP NDA Access for CentralNotice Metrics for Yahya - https://phabricator.wikimedia.org/T436294#12334069 (10tappof) User @Yahya has been added to NDA ldap group. [14:26:20] (03Merged) 10jenkins-bot: Keep Balinese Palm Leaf variants enabled on wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1331830 (https://phabricator.wikimedia.org/T436398) (owner: 10C. Scott Ananian) [14:26:26] (03Merged) 10jenkins-bot: Turn on variant conversion for PageAssessments [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340216 (https://phabricator.wikimedia.org/T328012) (owner: 10C. Scott Ananian) [14:26:30] (03Merged) 10jenkins-bot: Parsoid Read Views: Enable on 61 wikiquote wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341949 (https://phabricator.wikimedia.org/T437917) (owner: 10Subramanya Sastry) [14:26:49] !log cscott@deploy1003 Started scap sync-world: Backport for [[gerrit:1331830|Keep Balinese Palm Leaf variants enabled on wikisource (T436398)]], [[gerrit:1340216|Turn on variant conversion for PageAssessments (T328012)]], [[gerrit:1341949|Parsoid Read Views: Enable on 61 wikiquote wikis (T437917)]] [14:26:56] T436398: Enable ban-x-dharma, ban-x-palmleaf, and ban-x-pku variants only on Balinese wikisource - https://phabricator.wikimedia.org/T436398 [14:26:56] T328012: Chinese Language Converter does not work in Extension:PageAssessments - https://phabricator.wikimedia.org/T328012 [14:26:57] T437917: Parsoid Read Views: Deploy to wikiquote wikis - https://phabricator.wikimedia.org/T437917 [14:27:11] (03CR) 10JHathaway: [C:03+1] sre.hosts.provision: allow re-provisioning ml-serve1016 [cookbooks] - 10https://gerrit.wikimedia.org/r/1342679 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1430) [14:31:22] FIRING: [3x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:32:01] jouncebot: nowandnext [14:32:02] For the next 0 hour(s) and 27 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1430) [14:32:02] In 1 hour(s) and 27 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1600) [14:32:20] (03CR) 10Urbanecm: [C:03+2] JsonSchemaBuilder: Cache the root schema in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342265 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [14:32:24] (03CR) 10Urbanecm: [C:03+2] JsonSchemaBuilder: Cache the root schema in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342264 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [14:32:29] my config patch deploy is still in progress but should be finishing up shortly [14:32:54] !log cscott@deploy1003 ssastry, cscott: Backport for [[gerrit:1331830|Keep Balinese Palm Leaf variants enabled on wikisource (T436398)]], [[gerrit:1340216|Turn on variant conversion for PageAssessments (T328012)]], [[gerrit:1341949|Parsoid Read Views: Enable on 61 wikiquote wikis (T437917)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:33:01] T436398: Enable ban-x-dharma, ban-x-palmleaf, and ban-x-pku variants only on Balinese wikisource - https://phabricator.wikimedia.org/T436398 [14:33:01] T328012: Chinese Language Converter does not work in Extension:PageAssessments - https://phabricator.wikimedia.org/T328012 [14:33:02] T437917: Parsoid Read Views: Deploy to wikiquote wikis - https://phabricator.wikimedia.org/T437917 [14:33:25] FIRING: ProbeDown: Service registry1004:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:33:41] !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir4003.ulsfo.wmnet with OS trixie [14:33:42] cscott: oh, sorry. i didn't see your scap [14:33:50] fortunately, scap did see it and it says it is waiting for your run to finish [14:33:51] (03CR) 10Michael Große: mw::maintenance::growthexperiment: Do not use topictype=ores (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1342224 (https://phabricator.wikimedia.org/T437889) (owner: 10Urbanecm) [14:34:43] (03CR) 10Urbanecm: mw::maintenance::growthexperiment: Do not use topictype=ores (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1342224 (https://phabricator.wikimedia.org/T437889) (owner: 10Urbanecm) [14:34:59] urbanecm: I was also waiting in a queue :) [14:35:11] (03PS2) 10Majavah: Add new role for cloudinfra database backups [puppet] - 10https://gerrit.wikimedia.org/r/1342681 (https://phabricator.wikimedia.org/T301640) [14:35:12] (03PS1) 10Majavah: dynamicproxy: Do not provision backup timer on trixie [puppet] - 10https://gerrit.wikimedia.org/r/1342694 (https://phabricator.wikimedia.org/T301640) [14:35:14] also the test kitchen window has started now [14:35:26] but idk if that’s needed and/or conflicts with scap to begin with [14:35:38] !log cscott@deploy1003 ssastry, cscott: Continuing with deployment [14:35:45] Msz2001: above too many lines. i stopped my run [14:35:49] let me know if you want me to revoke the +2 [14:36:15] I was going to deploy patch to CheckUser, so it shouldn't conflict if yours get merged [14:36:22] RESOLVED: [3x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:36:47] Or they can be deployed together, if that's quicker [14:37:18] Msz2001: deploying together sounds good. [14:37:36] So, it's this one: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/1342684 [14:37:40] (03CR) 10Mszwarc: [C:03+2] "Ahead of deployment" [extensions/CheckUser] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342684 (https://phabricator.wikimedia.org/T438308) (owner: 10Mszwarc) [14:38:25] FIRING: [4x] ProbeDown: Service registry1004:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:38:38] (03CR) 10CI reject: [V:04-1] Add new role for cloudinfra database backups [puppet] - 10https://gerrit.wikimedia.org/r/1342681 (https://phabricator.wikimedia.org/T301640) (owner: 10Majavah) [14:39:32] (03PS3) 10Majavah: Add new role for cloudinfra database backups [puppet] - 10https://gerrit.wikimedia.org/r/1342681 (https://phabricator.wikimedia.org/T301640) [14:39:32] (03PS2) 10Majavah: dynamicproxy: Do not provision backup timer on trixie [puppet] - 10https://gerrit.wikimedia.org/r/1342694 (https://phabricator.wikimedia.org/T301640) [14:39:44] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host registry1005.eqiad.wmnet with OS trixie [14:40:17] (03PS4) 10Majavah: Add new role for cloudinfra database backups [puppet] - 10https://gerrit.wikimedia.org/r/1342681 (https://phabricator.wikimedia.org/T301640) [14:40:17] (03PS3) 10Majavah: dynamicproxy: Do not provision backup timer on trixie [puppet] - 10https://gerrit.wikimedia.org/r/1342694 (https://phabricator.wikimedia.org/T301640) [14:40:51] Msz2001: sounds good. want to run scap, or should i? [14:41:15] (03CR) 10Michael Große: [C:03+1] mw::maintenance::growthexperiment: Do not use topictype=ores (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1342224 (https://phabricator.wikimedia.org/T437889) (owner: 10Urbanecm) [14:41:19] you can do it [14:41:29] (03Merged) 10jenkins-bot: JsonSchemaBuilder: Cache the root schema in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342265 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [14:42:05] will do [14:42:20] !log cscott@deploy1003 Finished scap sync-world: Backport for [[gerrit:1331830|Keep Balinese Palm Leaf variants enabled on wikisource (T436398)]], [[gerrit:1340216|Turn on variant conversion for PageAssessments (T328012)]], [[gerrit:1341949|Parsoid Read Views: Enable on 61 wikiquote wikis (T437917)]] (duration: 15m 31s) [14:42:26] T436398: Enable ban-x-dharma, ban-x-palmleaf, and ban-x-pku variants only on Balinese wikisource - https://phabricator.wikimedia.org/T436398 [14:42:27] T328012: Chinese Language Converter does not work in Extension:PageAssessments - https://phabricator.wikimedia.org/T328012 [14:42:27] T437917: Parsoid Read Views: Deploy to wikiquote wikis - https://phabricator.wikimedia.org/T437917 [14:42:33] cscott: anything else to deploy? :) [14:42:59] (03Merged) 10jenkins-bot: JsonSchemaBuilder: Cache the root schema in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342264 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [14:43:41] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM" [alerts] - 10https://gerrit.wikimedia.org/r/1342349 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [14:43:50] (03PS1) 10AOkoth: site: phab2003 to insetup for re-image [puppet] - 10https://gerrit.wikimedia.org/r/1342698 (https://phabricator.wikimedia.org/T438365) [14:44:26] (03CR) 10Andrew Bogott: [C:03+2] Add a few alerts about magnum uptime [alerts] - 10https://gerrit.wikimedia.org/r/1342349 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [14:45:38] Afterwards, I'll have also a private code patch to deploy, but I'll do it after those 3 patches that are getting merged right now [14:45:45] sounds good [14:45:52] cscott: are you done? :) [14:46:19] (03Merged) 10jenkins-bot: SI: Preserve the username filter when switching queues [extensions/CheckUser] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1342684 (https://phabricator.wikimedia.org/T438308) (owner: 10Mszwarc) [14:46:31] Lucas_WMDE: to your question, the window is not actually a window. it just marks a time at which experiments starts., as far as i can see. [14:46:51] (03Merged) 10jenkins-bot: Add a few alerts about magnum uptime [alerts] - 10https://gerrit.wikimedia.org/r/1342349 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [14:47:03] urbanecm: ok… [14:47:34] Should it be reduced to zero-length on the [[Deployments]] page, then? [14:47:37] (03CR) 10AOkoth: [C:03+2] site: phab2003 to insetup for re-image [puppet] - 10https://gerrit.wikimedia.org/r/1342698 (https://phabricator.wikimedia.org/T438365) (owner: 10AOkoth) [14:47:49] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1342265|JsonSchemaBuilder: Cache the root schema in the process (T437588)]], [[gerrit:1342264|JsonSchemaBuilder: Cache the root schema in the process (T437588)]], [[gerrit:1342684|SI: Preserve the username filter when switching queues (T438308)]] [14:47:55] T437588: Wikimedia\RequestTimeout\RequestTimeoutException: The maximum execution time of {limit} seconds was exceeded - https://phabricator.wikimedia.org/T437588 [14:47:55] T438308: Switching queues causes account name cleared - https://phabricator.wikimedia.org/T438308 [14:48:07] Msz2001: that might make sense. [14:48:22] no response from c.scott, and i also see he's in a meeting, proceeding [14:50:14] !log aokoth@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on phab2003.codfw.wmnet with reason: Reimage [14:51:35] !log aokoth@cumin1004 START - Cookbook sre.hosts.reimage for host phab2003.codfw.wmnet with OS trixie [14:51:45] !log urbanecm@deploy1003 mszwarc, urbanecm: Backport for [[gerrit:1342265|JsonSchemaBuilder: Cache the root schema in the process (T437588)]], [[gerrit:1342264|JsonSchemaBuilder: Cache the root schema in the process (T437588)]], [[gerrit:1342684|SI: Preserve the username filter when switching queues (T438308)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:51:53] Msz2001: ^^ [14:52:06] looking [14:52:42] Works fine [14:55:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:55:55] !log urbanecm@deploy1003 mszwarc, urbanecm: Continuing with deployment [14:55:58] proceeding, ty [14:58:00] !log elukey@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on registry1005.eqiad.wmnet with reason: host reimage [14:58:47] !log cdobbins@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir4003.ulsfo.wmnet with reason: host reimage [14:59:56] (03PS5) 10Majavah: Add new role for cloudinfra database backups [puppet] - 10https://gerrit.wikimedia.org/r/1342681 (https://phabricator.wikimedia.org/T301640) [14:59:56] (03PS4) 10Majavah: dynamicproxy: Do not provision backup timer on trixie [puppet] - 10https://gerrit.wikimedia.org/r/1342694 (https://phabricator.wikimedia.org/T301640) [15:00:07] !log btullis@cumin1004 START - Cookbook sre.ganeti.makevm for new host ceph-admin1001.eqiad.wmnet [15:00:09] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [15:00:24] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1342265|JsonSchemaBuilder: Cache the root schema in the process (T437588)]], [[gerrit:1342264|JsonSchemaBuilder: Cache the root schema in the process (T437588)]], [[gerrit:1342684|SI: Preserve the username filter when switching queues (T438308)]] (duration: 12m 34s) [15:00:27] and done [15:00:28] T437588: Wikimedia\RequestTimeout\RequestTimeoutException: The maximum execution time of {limit} seconds was exceeded - https://phabricator.wikimedia.org/T437588 [15:00:29] T438308: Switching queues causes account name cleared - https://phabricator.wikimedia.org/T438308 [15:00:29] Msz2001: over to you then [15:00:38] Thanks! I'll do the private change [15:01:09] !log btullis@cumin1004 START - Cookbook sre.ganeti.makevm for new host ceph-admin2001.codfw.wmnet [15:02:12] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry1005.eqiad.wmnet with reason: host reimage [15:02:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.82% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:04:12] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ceph-admin1001.eqiad.wmnet - btullis@cumin1004" [15:04:15] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ceph-admin1001.eqiad.wmnet - btullis@cumin1004" [15:04:16] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:04:16] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache ceph-admin1001.eqiad.wmnet on all recursors [15:04:18] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ceph-admin1001.eqiad.wmnet on all recursors [15:04:49] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ceph-admin1001.eqiad.wmnet - btullis@cumin1004" [15:04:52] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ceph-admin1001.eqiad.wmnet - btullis@cumin1004" [15:04:56] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [15:04:57] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ceph-admin1001.eqiad.wmnet with OS bookworm [15:05:08] 06SRE, 10vm-requests: eqiad: 1 VM requested for ceph-admin - https://phabricator.wikimedia.org/T438037#12334393 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1004 for host ceph-admin1001.eqiad.wmnet with OS bookworm [15:05:14] !log aokoth@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on phab2003.codfw.wmnet with reason: host reimage [15:05:45] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir4003.ulsfo.wmnet with reason: host reimage [15:06:26] !log Deployed private code changes to Suggestedinvestigations [15:06:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:06:29] Done [15:07:14] (03CR) 10Elukey: [C:03+2] sre.hosts.provision: allow re-provisioning ml-serve1016 [cookbooks] - 10https://gerrit.wikimedia.org/r/1342679 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [15:07:37] urbanecm: yeah, sorry about the delay, you're right about the meeting. [15:08:22] cscott: no worries, otto's sitting right next to me today :D [15:08:58] !log aokoth@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab2003.codfw.wmnet with reason: host reimage [15:10:15] (03CR) 10MVernon: "Hi," [puppet] - 10https://gerrit.wikimedia.org/r/1305183 (https://phabricator.wikimedia.org/T429630) (owner: 10MVernon) [15:10:26] btullis@cumin1004 makevm (PID 3242326) is awaiting input [15:10:39] (03PS1) 10Tiziano Fogli: tcpircbot: replace newlines with spaces on privmsg [puppet] - 10https://gerrit.wikimedia.org/r/1342704 (https://phabricator.wikimedia.org/T438269) [15:11:08] (03PS2) 10Tiziano Fogli: tcpircbot: replace newlines with spaces in PRIVMSG [puppet] - 10https://gerrit.wikimedia.org/r/1342704 (https://phabricator.wikimedia.org/T438269) [15:15:15] (03PS1) 10Andrew Bogott: Add wmcs-magnum-kubectl [puppet] - 10https://gerrit.wikimedia.org/r/1342707 (https://phabricator.wikimedia.org/T429557) [15:15:19] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ceph-admin2001.codfw.wmnet - btullis@cumin1004" [15:15:23] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ceph-admin2001.codfw.wmnet - btullis@cumin1004" [15:15:24] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:15:24] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache ceph-admin2001.codfw.wmnet on all recursors [15:15:26] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ceph-admin2001.codfw.wmnet on all recursors [15:15:28] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be110[1-6] - https://phabricator.wikimedia.org/T438368 (10RobH) 03NEW [15:16:07] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ceph-admin2001.codfw.wmnet - btullis@cumin1004" [15:16:10] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ceph-admin2001.codfw.wmnet - btullis@cumin1004" [15:16:14] (03CR) 10CI reject: [V:04-1] Add wmcs-magnum-kubectl [puppet] - 10https://gerrit.wikimedia.org/r/1342707 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [15:16:15] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ceph-admin2001.codfw.wmnet with OS bookworm [15:16:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:16:25] 06SRE, 10vm-requests: codfw: 1 VM requested for ceph-admin - https://phabricator.wikimedia.org/T438038#12334508 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1004 for host ceph-admin2001.codfw.wmnet with OS bookworm [15:16:29] (03PS2) 10Andrew Bogott: Add wmcs-magnum-kubectl [puppet] - 10https://gerrit.wikimedia.org/r/1342707 (https://phabricator.wikimedia.org/T429557) [15:16:52] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be110[1-6] - https://phabricator.wikimedia.org/T438368#12334517 (10RobH) a:03MatthewVernon @MatthewVernon, I've moved the ordering tasks along as they were going to expire if not placed to order, and I've now taken some assumptio... [15:17:03] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be110[1-6] - https://phabricator.wikimedia.org/T438368#12334525 (10RobH) [15:17:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.32% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:17:35] (03CR) 10CI reject: [V:04-1] Add wmcs-magnum-kubectl [puppet] - 10https://gerrit.wikimedia.org/r/1342707 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [15:17:51] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ceph-admin1001.eqiad.wmnet with reason: host reimage [15:18:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:19:02] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry1005.eqiad.wmnet with OS trixie [15:19:02] (03PS3) 10Andrew Bogott: Add wmcs-magnum-kubectl [puppet] - 10https://gerrit.wikimedia.org/r/1342707 (https://phabricator.wikimedia.org/T429557) [15:19:39] 10ops-codfw, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be2[099-102] - https://phabricator.wikimedia.org/T438370 (10RobH) 03NEW [15:20:17] 10ops-codfw, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be2[099-102] - https://phabricator.wikimedia.org/T438370#12334561 (10RobH) a:03MatthewVernon @MatthewVernon, I've moved the ordering tasks along as they were going to expire if not placed to order, and I've now taken some assumpt... [15:20:47] 10ops-codfw, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be2[099-102] - https://phabricator.wikimedia.org/T438370#12334573 (10RobH) [15:21:29] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be110[1-6] - https://phabricator.wikimedia.org/T438368#12334577 (10RobH) [15:21:51] 10ops-codfw, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be2[099-102] - https://phabricator.wikimedia.org/T438370#12334581 (10RobH) [15:22:28] 10ops-codfw, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be2[099-102] - https://phabricator.wikimedia.org/T438370#12334585 (10RobH) [15:22:44] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be110[1-6] - https://phabricator.wikimedia.org/T438368#12334587 (10RobH) [15:23:22] 10ops-codfw, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be2[099-104] - https://phabricator.wikimedia.org/T438370#12334590 (10RobH) [15:23:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ceph-admin1001.eqiad.wmnet with reason: host reimage [15:23:44] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be110[1-4] - https://phabricator.wikimedia.org/T438368#12334592 (10RobH) [15:25:27] (03PS3) 10CWilliams: mediabackups: Perform backups via systemd units [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) [15:25:49] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be2[099-104] - https://phabricator.wikimedia.org/T438370#12334601 (10MatthewVernon) [15:25:49] !log aokoth@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host phab2003.codfw.wmnet with OS trixie [15:26:46] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir4003.ulsfo.wmnet with OS trixie [15:26:48] (03PS6) 10Dduvall: buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1329694 (https://phabricator.wikimedia.org/T434958) [15:28:39] (03PS1) 10Muehlenhoff: Revert "Stop building a Bullseye base image" [puppet] - 10https://gerrit.wikimedia.org/r/1342715 [15:28:44] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be110[1-4] - https://phabricator.wikimedia.org/T438368#12334622 (10MatthewVernon) [15:28:53] (03PS2) 10Muehlenhoff: Revert "Stop building a Bullseye base image" [puppet] - 10https://gerrit.wikimedia.org/r/1342715 [15:28:58] (03PS1) 10AOkoth: site: phab2003 apply migration role [puppet] - 10https://gerrit.wikimedia.org/r/1342716 [15:29:27] (03PS11) 10Federico Ceratto: mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) [15:30:05] (03PS3) 10Tiziano Fogli: tcpircbot: replace newlines with spaces in PRIVMSG [puppet] - 10https://gerrit.wikimedia.org/r/1342704 (https://phabricator.wikimedia.org/T438269) [15:30:43] (03CR) 10AOkoth: [C:03+2] site: phab2003 apply migration role [puppet] - 10https://gerrit.wikimedia.org/r/1342716 (owner: 10AOkoth) [15:30:54] (03PS3) 10Muehlenhoff: Revert "Stop building a Bullseye base image" [puppet] - 10https://gerrit.wikimedia.org/r/1342715 [15:31:03] (03CR) 10CI reject: [V:04-1] tcpircbot: replace newlines with spaces in PRIVMSG [puppet] - 10https://gerrit.wikimedia.org/r/1342704 (https://phabricator.wikimedia.org/T438269) (owner: 10Tiziano Fogli) [15:31:32] (03CR) 10Federico Ceratto: mariadb: Deploy backup_metrics.py (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [15:31:55] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342656 (https://phabricator.wikimedia.org/T435314) (owner: 10Muehlenhoff) [15:32:33] (03PS4) 10Tiziano Fogli: tcpircbot: replace newlines with spaces in PRIVMSG [puppet] - 10https://gerrit.wikimedia.org/r/1342704 (https://phabricator.wikimedia.org/T438269) [15:33:22] 06SRE-OnFire, 06MediaWiki-Engineering, 06ServiceOps, 06SRE Observability, 07Sustainability (Incident Followup): Reduce the amount of messages sent through channel:Memcached during failures - https://phabricator.wikimedia.org/T390529#12334667 (10herron) >>! In T390529#12333802, @Krinkle wrote: >My underst... [15:33:42] (03CR) 10CI reject: [V:04-1] tcpircbot: replace newlines with spaces in PRIVMSG [puppet] - 10https://gerrit.wikimedia.org/r/1342704 (https://phabricator.wikimedia.org/T438269) (owner: 10Tiziano Fogli) [15:33:43] (03CR) 10Marostegui: [C:03+1] "ok from my side - @cwilliams@wikimedia.org all good from your side too?" [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [15:33:48] (03CR) 10CWilliams: "Just to note, the dashboard hits a replica with these queries and not the primary." [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [15:34:58] (03CR) 10RLazarus: [C:03+2] editcheck-headless: Add deployment for the technical pilot [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [15:35:26] 06SRE, 10SRE-Access-Requests: Requesting access to Superset Dashboard for Hany EL Mokadem - https://phabricator.wikimedia.org/T438034#12334678 (10WMDE-leszek) I approve this request on WMDE's end. I would also approve the NDA request once it is submitted (I suppose it might be a separate ticket) [15:35:28] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be110[1-4] - https://phabricator.wikimedia.org/T438368#12334679 (10MatthewVernon) a:05MatthewVernon→03None https://gerrit.wikimedia.org/r/c/operations/puppet/+/1280373 already made the necessary changes fo... [15:36:27] (03PS5) 10Tiziano Fogli: tcpircbot: replace newlines with spaces in PRIVMSG [puppet] - 10https://gerrit.wikimedia.org/r/1342704 (https://phabricator.wikimedia.org/T438269) [15:37:55] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ceph-admin2001.codfw.wmnet with reason: host reimage [15:39:44] !log cdobbins@cumin1004 conftool action : set/pooled=yes; selector: name=ncredir4003.* [15:40:02] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ceph-admin1001.eqiad.wmnet with OS bookworm [15:40:02] !log btullis@cumin1004 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ceph-admin1001.eqiad.wmnet [15:40:12] 06SRE, 10vm-requests: eqiad: 1 VM requested for ceph-admin - https://phabricator.wikimedia.org/T438037#12334689 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1004 for host ceph-admin1001.eqiad.wmnet with OS bookworm completed: - ceph-admin1001 (**PASS**) - Removed from P... [15:40:53] (03PS1) 10MVernon: swift: prep for ms-be21* [puppet] - 10https://gerrit.wikimedia.org/r/1342718 (https://phabricator.wikimedia.org/T438370) [15:43:29] (03PS1) 10Andrew Bogott: magnum alerts: split into two files [alerts] - 10https://gerrit.wikimedia.org/r/1342719 (https://phabricator.wikimedia.org/T429557) [15:44:02] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ceph-admin2001.codfw.wmnet with reason: host reimage [15:45:42] (03Merged) 10jenkins-bot: editcheck-headless: Add deployment for the technical pilot [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [15:46:21] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1342718 (https://phabricator.wikimedia.org/T438370) (owner: 10MVernon) [15:46:56] (03CR) 10MVernon: [C:03+2] swift: prep for ms-be21* [puppet] - 10https://gerrit.wikimedia.org/r/1342718 (https://phabricator.wikimedia.org/T438370) (owner: 10MVernon) [15:47:15] (03CR) 10Andrew Bogott: [C:03+2] magnum alerts: split into two files [alerts] - 10https://gerrit.wikimedia.org/r/1342719 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [15:48:42] !log installing libde265 security updates [15:48:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:48:46] (03CR) 10Andrew Bogott: [C:03+2] Add wmcs-magnum-kubectl [puppet] - 10https://gerrit.wikimedia.org/r/1342707 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [15:49:42] (03PS1) 10AOkoth: migration: attempt type error fix [puppet] - 10https://gerrit.wikimedia.org/r/1342721 (https://phabricator.wikimedia.org/T438365) [15:50:04] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, and 2 others: Q1:rack/setup/install ms-be2[099-104] - https://phabricator.wikimedia.org/T438370#12334743 (10MatthewVernon) a:05MatthewVernon→03None [15:50:11] (03Merged) 10jenkins-bot: magnum alerts: split into two files [alerts] - 10https://gerrit.wikimedia.org/r/1342719 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [15:51:20] !log vriley@cumin1004 START - Cookbook sre.hosts.reimage for host ms-be1099.eqiad.wmnet with OS trixie [15:51:35] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12334747 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1004 for host ms-be1099.eqiad.wmnet with... [15:52:03] (03CR) 10AOkoth: [C:03+2] migration: attempt type error fix [puppet] - 10https://gerrit.wikimedia.org/r/1342721 (https://phabricator.wikimedia.org/T438365) (owner: 10AOkoth) [15:52:24] !log rzl@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [15:53:17] !log rzl@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [15:53:47] !log rzl@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [15:54:05] !log rzl@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [15:54:19] (03CR) 10Dzahn: [C:03+2] deployment_server: run scap clean-images daily [puppet] - 10https://gerrit.wikimedia.org/r/1342337 (https://phabricator.wikimedia.org/T438148) (owner: 10Ahmon Dancy) [15:54:31] !log rzl@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [15:54:49] !log rzl@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [15:55:08] (03PS1) 10Dbrant: wikifeeds: promote to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342724 [15:55:14] !log rzl@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [15:55:30] !log rzl@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [15:57:16] (03CR) 10Muehlenhoff: [C:03+2] Switch firmware management host to cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1342284 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [15:58:45] (03CR) 10Dbrant: [C:03+2] wikifeeds: promote to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342724 (owner: 10Dbrant) [15:59:28] (03CR) 10Marostegui: mediabackups: Perform backups via systemd units (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) (owner: 10CWilliams) [16:00:04] jhathaway and rzl: I, the Bot under the Fountain, call upon thee, The Deployer, to do Puppet request window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1600). [16:00:04] Urbanecm: A patch you scheduled for Puppet request window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [16:00:12] * urbanecm around [16:00:18] (03PS12) 10Federico Ceratto: mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) [16:00:34] 06SRE, 10vm-requests: eqiad: 1 VM requested for ceph-admin - https://phabricator.wikimedia.org/T438037#12334776 (10BTullis) 05Open→03Resolved [16:00:36] (03CR) 10Federico Ceratto: mariadb: Deploy backup_metrics.py (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [16:00:38] (03CR) 10Dzahn: "this isn't correct. it needs to be an array of host names." [puppet] - 10https://gerrit.wikimedia.org/r/1342721 (https://phabricator.wikimedia.org/T438365) (owner: 10AOkoth) [16:01:19] 06SRE-OnFire, 06MediaWiki-Engineering, 06ServiceOps, 06SRE Observability, 07Sustainability (Incident Followup): Reduce the amount of messages sent through channel:Memcached during failures - https://phabricator.wikimedia.org/T390529#12334795 (10herron) [16:01:21] (03Merged) 10jenkins-bot: wikifeeds: promote to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342724 (owner: 10Dbrant) [16:01:23] urbanecm: this lgtm but I can't merge it right this sec -- I'll be back in half an hour and I'm happy to merge it then, sorry about that! no need to be around unless you'd like to coordinate a test run [16:01:24] (03PS13) 10Federico Ceratto: mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) [16:01:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ceph-admin2001.codfw.wmnet with OS bookworm [16:01:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ceph-admin2001.codfw.wmnet [16:01:41] 06SRE, 10vm-requests: codfw: 1 VM requested for ceph-admin - https://phabricator.wikimedia.org/T438038#12334796 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1004 for host ceph-admin2001.codfw.wmnet with OS bookworm completed: - ceph-admin2001 (**PASS**) - Removed from P... [16:01:45] !log installing aom security updates [16:01:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:02:03] rzl: sounds good, test run shouldn't be needed [16:02:50] (03PS1) 10AOkoth: Revert "migration: attempt type error fix" [puppet] - 10https://gerrit.wikimedia.org/r/1342730 [16:04:47] (03CR) 10CWilliams: mediabackups: Perform backups via systemd units (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) (owner: 10CWilliams) [16:06:02] urbanecm and rzl, happy to merge it as well [16:06:07] (03PS4) 10CWilliams: mediabackups: Perform backups via systemd units [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) [16:06:18] or that :) up2you [16:06:18] oh great! thanks [16:06:23] !log vriley@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1099.eqiad.wmnet with reason: host reimage [16:07:13] (03CR) 10JHathaway: [C:03+2] mw::maintenance::growthexperiment: Do not use topictype=ores [puppet] - 10https://gerrit.wikimedia.org/r/1342224 (https://phabricator.wikimedia.org/T437889) (owner: 10Urbanecm) [16:07:58] !log rzl@deploy1003 helmfile [staging] START helmfile.d/services/editcheck-headless: apply [16:08:07] urbanecm: merged [16:08:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.41% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:08:49] perfect. it seems to have resolved an alert! [16:08:56] thanks for the help jhathaway [16:08:57] !log rzl@deploy1003 helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply [16:09:20] (03PS1) 10Eevans: cassandra: add grants for new LAC tables [puppet] - 10https://gerrit.wikimedia.org/r/1342736 (https://phabricator.wikimedia.org/T431499) [16:09:31] urbanecm: can you do the same for the remaining alerts please [16:09:41] :P [16:09:48] unfortunately, my powers are limited! [16:10:36] (03CR) 10Federico Ceratto: [C:03+2] mariadb: Deploy backup_metrics.py [puppet] - 10https://gerrit.wikimedia.org/r/1342302 (https://phabricator.wikimedia.org/T438003) (owner: 10Federico Ceratto) [16:11:03] !log dbrant@deploy1003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [16:11:24] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, and 2 others: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12334834 (10VRiley-WMF) [16:11:24] (03CR) 10CWilliams: mediabackups: Perform backups via systemd units (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) (owner: 10CWilliams) [16:11:27] !log dbrant@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [16:11:59] (03CR) 10Dzahn: "try this instead:" [puppet] - 10https://gerrit.wikimedia.org/r/1342721 (https://phabricator.wikimedia.org/T438365) (owner: 10AOkoth) [16:12:06] !log vriley@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1099.eqiad.wmnet with reason: host reimage [16:12:11] !log installing libapache-mod-auth-oidc security updates [16:12:12] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:12:39] (03CR) 10CWilliams: mediabackups: Perform backups via systemd units (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) (owner: 10CWilliams) [16:13:02] (03CR) 10AOkoth: [C:03+2] Revert "migration: attempt type error fix" [puppet] - 10https://gerrit.wikimedia.org/r/1342730 (owner: 10AOkoth) [16:14:27] !log dbrant@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [16:14:53] !log dbrant@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [16:15:08] !log dbrant@deploy1003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [16:15:27] !log vriley@cumin1004 START - Cookbook sre.dns.netbox [16:15:36] !log dbrant@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [16:16:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:17:12] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:34] (03PS1) 10AOkoth: phabricator: fix migration profile type error [puppet] - 10https://gerrit.wikimedia.org/r/1342740 (https://phabricator.wikimedia.org/T438365) [16:19:26] !log vriley@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [ms-be1100] - vriley@cumin1004" [16:19:44] !log vriley@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [ms-be1100] - vriley@cumin1004" [16:19:44] !log vriley@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:20:07] (03PS2) 10JHathaway: WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 [16:20:15] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342382 (owner: 10JHathaway) [16:20:27] !log vriley@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host ms-be1100 [16:20:53] !log vriley@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1100 [16:21:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:21:26] !log vriley@cumin1004 START - Cookbook sre.hosts.provision for host ms-be1100.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [16:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [16:21:54] (03CR) 10AOkoth: [C:03+2] phabricator: fix migration profile type error [puppet] - 10https://gerrit.wikimedia.org/r/1342740 (https://phabricator.wikimedia.org/T438365) (owner: 10AOkoth) [16:23:53] (03PS3) 10JHathaway: WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 [16:24:09] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342382 (owner: 10JHathaway) [16:31:01] !log aokoth@deploy1003 Started deploy [phabricator/deployment@c386249]: Deploy Phab [16:31:20] !log aokoth@deploy1003 Finished deploy [phabricator/deployment@c386249]: Deploy Phab (duration: 00m 19s) [16:33:52] (03CR) 10Dzahn: [C:03+2] backup/zuul: add backup set for zuul [puppet] - 10https://gerrit.wikimedia.org/r/1339983 (https://phabricator.wikimedia.org/T436131) (owner: 10Dzahn) [16:35:45] !log vriley@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1100.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [16:35:55] (03PS1) 10AOkoth: site: apply production role phab2003 [puppet] - 10https://gerrit.wikimedia.org/r/1342752 (https://phabricator.wikimedia.org/T438365) [16:39:34] (03CR) 10AOkoth: [C:03+2] site: apply production role phab2003 [puppet] - 10https://gerrit.wikimedia.org/r/1342752 (https://phabricator.wikimedia.org/T438365) (owner: 10AOkoth) [16:42:18] !log vriley@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1004" [16:42:50] !log vriley@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1004" [16:42:51] !log vriley@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1099.eqiad.wmnet with OS trixie [16:43:08] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12334998 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1004 for host ms-be1099.eqiad.wmnet with OS... [16:48:23] PROBLEM - Host ms-be1098 is DOWN: PING CRITICAL - Packet loss = 100% [16:52:26] !log aokoth@deploy1003 Started deploy [phabricator/deployment@c386249]: Deploy Phab [16:52:38] !log aokoth@deploy1003 Finished deploy [phabricator/deployment@c386249]: Deploy Phab (duration: 00m 12s) [16:53:01] (03PS2) 10RLazarus: editcheck-headless: Add service catalog entry [puppet] - 10https://gerrit.wikimedia.org/r/1338081 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [16:55:11] (03CR) 10RLazarus: [C:03+2] editcheck-headless: Add service catalog entry [puppet] - 10https://gerrit.wikimedia.org/r/1338081 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [16:55:36] RECOVERY - Host ms-be1098 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [17:00:05] bd808: It is that lovely time of the day again! You are hereby commanded to deploy Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1700). [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1700) [17:02:02] (03Abandoned) 10Urbanecm: [DNM] Test CI [extensions/GrowthExperiments] (wmf/1.43.0-wmf.28) - 10https://gerrit.wikimedia.org/r/1085358 (owner: 10Sergio Gimeno) [17:04:40] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12335039 (10Dzahn) @LPetty-WMF We would need the full scp command you are running and also your ssh config to be able to really debug this. To simplify things; let's start with t... [17:05:10] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12335040 (10Dzahn) 05Resolved→03Open [17:05:32] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12335041 (10Dzahn) a:05hnowlan→03Dzahn [17:07:50] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12335047 (10Dzahn) a:03EAlbizzati-WMF [17:11:05] (03CR) 10RLazarus: [C:03+1] "Thank you for jumping on this!" [puppet] - 10https://gerrit.wikimedia.org/r/1342704 (https://phabricator.wikimedia.org/T438269) (owner: 10Tiziano Fogli) [17:20:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:23:04] !log vriley@cumin1004 START - Cookbook sre.hosts.reimage for host ms-be1100.eqiad.wmnet with OS trixie [17:23:18] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, and 2 others: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12335108 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1004 for host ms-be1100.eqiad.wmnet wi... [17:25:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:26:39] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:29:23] (03CR) 10Eevans: [C:03+2] cassandra: add grants for new LAC tables [puppet] - 10https://gerrit.wikimedia.org/r/1342736 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [17:37:43] !log vriley@cumin1004 START - Cookbook sre.hosts.provision for host ms-be1100.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:38:07] !log vriley@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1100.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:38:31] !log vriley@cumin1004 START - Cookbook sre.hosts.provision for host ms-be1100.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:47:08] !log vriley@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1100.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:51:27] jhathaway: i think something's wrong with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1342224. the job still seems to have the removed parameter, despite it's well after 30 minutes? [17:51:31] https://www.irccloud.com/pastebin/D0YreHEu/ [17:51:42] do i need to do something specific to see the effect in k8s? [17:54:30] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, and 2 others: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12335328 (10VRiley-WMF) [17:57:14] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Automate PDU Deployment Process - https://phabricator.wikimedia.org/T403173#12335335 (10VRiley-WMF) @LSobanski We have some that are ready to be setup. Is there anything I can help with on this activity? [17:58:06] (03PS1) 10Andrew Bogott: magnum-capi-worker-stats-exporter.py: handle a few more exceptions [puppet] - 10https://gerrit.wikimedia.org/r/1342771 (https://phabricator.wikimedia.org/T429557) [18:00:05] jnuche and dduvall: I, the Bot under the Fountain, call upon thee, The Deployer, to do MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T1800). [18:01:20] (03PS1) 10Majavah: Remove bullseye based images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1342772 (https://phabricator.wikimedia.org/T400258) [18:03:52] jnuche: dduvall: train is all on wmf.20, so i assume this window's unused? [18:04:26] FIRING: OutboundMXQueueHigh: MX host mx-out1001:9154 has many queued messages: 15778 #page - https://wikitech.wikimedia.org/wiki/Postfix - https://grafana.wikimedia.org/d/h36Havfik/mail-postfix-servers - https://alerts.wikimedia.org/?q=alertname%3DOutboundMXQueueHigh [18:04:35] urbanecm: correct! [18:04:40] (03PS1) 10Cmelo: Worklist Promotion test kitchen - Enable flag in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342774 (https://phabricator.wikimedia.org/T434513) [18:04:55] ok, then i'm going to steal it :D [18:05:23] sup? [18:06:30] <_joe_> federico3: here [18:06:54] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342774 (https://phabricator.wikimedia.org/T434513) (owner: 10Cmelo) [18:08:49] (03PS1) 10Dduvall: Fix missing dependencies [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1342775 (https://phabricator.wikimedia.org/T434958) [18:14:50] <_joe_> !ack [18:14:51] 8349 (ACKED) OutboundMXQueueHigh sre (mx-out1001:9154 eqiad) [18:20:49] !log Deploy a security fix for T438389 [18:20:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:25:56] RESOLVED: OutboundMXQueueHigh: MX host mx-out1001:9154 has many queued messages: 1218 #page - https://wikitech.wikimedia.org/wiki/Postfix - https://grafana.wikimedia.org/d/h36Havfik/mail-postfix-servers - https://alerts.wikimedia.org/?q=alertname%3DOutboundMXQueueHigh [18:28:19] (03CR) 10JHathaway: [C:03+1] Add explicit hostnames to LDAP cert SNIs [puppet] - 10https://gerrit.wikimedia.org/r/1342660 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [18:32:08] urbanecm, jhathaway: belatedly -- after the puppet merge and running puppet on the deploy host, mw-cron changes also need a "helmfile apply", https://wikitech.wikimedia.org/wiki/Mw-cron_jobs#Deploying_periodic_jobs -- it's an extra step but I typically do it when deploying in the puppet window, still newish so not top-of-mind for everybody [18:32:35] ah, thanks, sorry for missing that [18:32:39] but, in this case it also got picked up along with your security patch (which deploys mw-cron among other things) so you should be good to go [18:32:57] !log vriley@cumin1004 START - Cookbook sre.hosts.reimage for host ms-be1100.eqiad.wmnet with OS trixie [18:33:09] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12335479 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1004 for host ms-be1100.eqiad.wmnet with... [18:33:45] yep yep, it now seems fully applied. thanks for checking rzl! [18:34:38] (fwiw any deployer can do that helmfile apply too -- without root you just have to wait for the puppet run instead of being able to force one) [18:38:25] FIRING: [4x] ProbeDown: Service registry1004:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:44:43] (03PS1) 10Tsevener: Exclude returntoapp query from app interception on iOS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342781 (https://phabricator.wikimedia.org/T438395) [18:46:07] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342781 (https://phabricator.wikimedia.org/T438395) (owner: 10Tsevener) [18:48:10] !log vriley@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1100.eqiad.wmnet with reason: host reimage [18:52:33] !log vriley@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1100.eqiad.wmnet with reason: host reimage [18:52:49] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12335544 (10Dzahn) @VRiley-WMF I think you got the wrong ticket in the cookbook run. Cheers [19:07:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:07:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 16.02% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:10:05] !log vriley@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1004" [19:13:08] vriley@cumin1004 reimage (PID 3286854) is awaiting input [19:15:15] !log vriley@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1004" [19:15:16] !log vriley@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1100.eqiad.wmnet with OS trixie [19:15:27] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12335621 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1004 for host ms-be1100.eqiad.wmnet with OS... [19:16:37] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:16:53] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for VolkerE - https://phabricator.wikimedia.org/T437634#12335625 (10Jdrewniak) I approve @Volker_E 's request to access `analytics-privatedata-users` [19:20:14] PROBLEM - Host ms-be1100 is DOWN: PING CRITICAL - Packet loss = 100% [19:27:00] RECOVERY - Host ms-be1100 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [19:30:12] (03PS2) 10Dzahn: openstack: enable zookeeper logging in codfw1dev [puppet] - 10https://gerrit.wikimedia.org/r/1342354 [19:30:29] (03PS3) 10Dzahn: openstack: enable zookeeper logging in codfw1dev [puppet] - 10https://gerrit.wikimedia.org/r/1342354 (https://phabricator.wikimedia.org/T435503) [19:30:37] (03CR) 10Andrew Bogott: [C:03+2] magnum-capi-worker-stats-exporter.py: handle a few more exceptions [puppet] - 10https://gerrit.wikimedia.org/r/1342771 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [19:30:42] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, and 2 others: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12335645 (10VRiley-WMF) [19:30:59] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, and 2 others: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12335650 (10VRiley-WMF) 05In progress→03Resolved These have been reimaged and ready. [19:31:10] (03PS1) 10Xcollazo: dumps: re-add mirror PDApps [puppet] - 10https://gerrit.wikimedia.org/r/1342785 [19:31:44] (03PS2) 10Xcollazo: dumps: re-add mirror PDApps [puppet] - 10https://gerrit.wikimedia.org/r/1342785 (https://phabricator.wikimedia.org/T437896) [19:31:47] (03CR) 10Dzahn: "what it actually does: https://puppet-compiler.wmflabs.org/output/1342354/9449/cloudcontrol2005-dev.codfw.wmnet/index.html" [puppet] - 10https://gerrit.wikimedia.org/r/1342354 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:32:22] (03CR) 10Dzahn: "@abogott@wikimedia.org Only if you are interested. Letting you know this is now an option if you want logs." [puppet] - 10https://gerrit.wikimedia.org/r/1342354 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:32:31] FIRING: [2x] Traffic bill over quota: Alert for device cr1-eqiad.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [19:34:28] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12335678 (10VRiley-WMF) 05Resolved→03Open @Dzahn Yes, I was aware. I was trying to copy and paste commands and forgot to change it. Than... [19:34:40] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12335682 (10VRiley-WMF) 05Open→03Resolved [19:36:35] (03CR) 10Dzahn: "well, if you want it and everybody else wants it, we can simplify code again or make it an opt-out instead of an opt-in, which would be sl" [puppet] - 10https://gerrit.wikimedia.org/r/1342354 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:37:27] (03CR) 10Dzahn: "So I guess it's more like asking if you mind turning it on. :)" [puppet] - 10https://gerrit.wikimedia.org/r/1342354 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:37:31] FIRING: [3x] Traffic bill over quota: Alert for device cr1-eqiad.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [19:45:34] (03PS2) 10Dzahn: traffserver/cache: add attribution.wikimedia.org map and default caching [puppet] - 10https://gerrit.wikimedia.org/r/1341338 (https://phabricator.wikimedia.org/T437635) [19:47:56] (03PS1) 10Dzahn: microsites: add monitoring for attribution.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1342791 (https://phabricator.wikimedia.org/T437635) [19:48:51] (03CR) 10Dzahn: [C:03+1] "I feel like would be best if this is confirmed/deployed by observability." [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [19:50:26] 06SRE, 10Internet-Archive, 06Traffic, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12335776 (10BCornwall) 05Open→03Stalled [19:51:41] 06SRE, 10Internet-Archive, 06Traffic, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12335792 (10Cyberpower678) Sorry. I missed the last message. [19:52:31] FIRING: [3x] Traffic bill over quota: Alert for device cr1-eqiad.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [19:52:45] (03PS1) 10Robertsky: Revert "Update wikimania wordmark for 2026" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342798 (https://phabricator.wikimedia.org/T437625) [19:55:49] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 17 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342798 (https://phabricator.wikimedia.org/T437625) (owner: 10Robertsky) [19:56:25] (03PS1) 10Dduvall: buildx: Alternative implementation using bake [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) [19:57:24] (03PS2) 10Dduvall: buildx: Alternative implementation using bake [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) [19:57:31] RESOLVED: Traffic bill over quota: Alert for device cr2-codfw.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [19:58:20] (03PS1) 10Dzahn: site/codesearch: test codesearch puppet role [puppet] - 10https://gerrit.wikimedia.org/r/1342800 (https://phabricator.wikimedia.org/T268199) [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: That opportune time for a UTC late backport window deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T2000). [20:00:05] cmelo, toni_, and Robertsky: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:10] \o/ [20:00:16] o/ [20:00:19] o/ [20:02:59] Do you all need a deployer? [20:03:08] I do, thanks [20:03:15] yes [20:03:31] yes [20:03:37] Since these are all config changes, is it already if I deploy them simultaneously? [20:03:41] alright** [20:03:55] fine with me [20:03:57] no issue from me. [20:04:01] fine for me [20:04:13] 👍 [20:05:47] (03PS1) 10Dzahn: codesearch: copy Hiera data from cloud realm [puppet] - 10https://gerrit.wikimedia.org/r/1342802 (https://phabricator.wikimedia.org/T268199) [20:07:04] jeena: you're deploying, i assume? [20:07:20] yeah was just taking a look at the patches first [20:07:28] (03CR) 10Dzahn: [C:03+2] codesearch: copy Hiera data from cloud realm [puppet] - 10https://gerrit.wikimedia.org/r/1342802 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [20:07:41] sounds good, i'll let you to it [20:10:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jhuneidi@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342774 (https://phabricator.wikimedia.org/T434513) (owner: 10Cmelo) [20:10:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jhuneidi@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342781 (https://phabricator.wikimedia.org/T438395) (owner: 10Tsevener) [20:10:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jhuneidi@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342798 (https://phabricator.wikimedia.org/T437625) (owner: 10Robertsky) [20:11:13] (03Merged) 10jenkins-bot: Worklist Promotion test kitchen - Enable flag in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342774 (https://phabricator.wikimedia.org/T434513) (owner: 10Cmelo) [20:11:16] (03Merged) 10jenkins-bot: Exclude returntoapp query from app interception on iOS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342781 (https://phabricator.wikimedia.org/T438395) (owner: 10Tsevener) [20:11:26] (03Merged) 10jenkins-bot: Revert "Update wikimania wordmark for 2026" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342798 (https://phabricator.wikimedia.org/T437625) (owner: 10Robertsky) [20:11:43] !log jhuneidi@deploy1003 Started scap sync-world: Backport for [[gerrit:1342774|Worklist Promotion test kitchen - Enable flag in production (T434513)]], [[gerrit:1342781|Exclude returntoapp query from app interception on iOS (T438395)]], [[gerrit:1342798|Revert "Update wikimania wordmark for 2026"]] [20:11:48] T434513: Worklist Promotion test kitchen instrumentation - Enable flag in production - https://phabricator.wikimedia.org/T434513 [20:11:49] T438395: Visual Editor: Exclude returntoapp query item from app site association file - https://phabricator.wikimedia.org/T438395 [20:16:08] brb, quick bio break [20:17:12] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:17:37] back [20:18:02] (03PS1) 10Dzahn: codesearch: add role description and owner [puppet] - 10https://gerrit.wikimedia.org/r/1342803 (https://phabricator.wikimedia.org/T268199) [20:20:39] still building [20:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [20:22:41] (03PS1) 10Dzahn: trafficserver: add maps for codesearch (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/1342804 (https://phabricator.wikimedia.org/T268199) [20:22:44] (03CR) 10Dzahn: [C:03+2] codesearch: add role description and owner [puppet] - 10https://gerrit.wikimedia.org/r/1342803 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [20:23:43] (03PS4) 10JHathaway: WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 [20:24:01] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342382 (owner: 10JHathaway) [20:24:52] (03PS1) 10Dzahn: add discovery record for codesearch [dns] - 10https://gerrit.wikimedia.org/r/1342805 (https://phabricator.wikimedia.org/T268199) [20:25:01] (03CR) 10CI reject: [V:04-1] WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 (owner: 10JHathaway) [20:31:42] !log jhuneidi@deploy1003 robertsky, jhuneidi, cmelo, tsev: Backport for [[gerrit:1342774|Worklist Promotion test kitchen - Enable flag in production (T434513)]], [[gerrit:1342781|Exclude returntoapp query from app interception on iOS (T438395)]], [[gerrit:1342798|Revert "Update wikimania wordmark for 2026"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:31:47] T434513: Worklist Promotion test kitchen instrumentation - Enable flag in production - https://phabricator.wikimedia.org/T434513 [20:31:48] T438395: Visual Editor: Exclude returntoapp query item from app site association file - https://phabricator.wikimedia.org/T438395 [20:32:37] toni_: cmelo robertsky please do any checks you need on debug [20:32:38] my changes are verified. [20:32:40] thanks [20:32:44] (03PS2) 10Dzahn: add discovery record for codesearch [dns] - 10https://gerrit.wikimedia.org/r/1342805 (https://phabricator.wikimedia.org/T268199) [20:32:45] thanks [20:34:55] mine is fine too [20:34:59] mine look good [20:35:01] (03PS1) 10Dzahn: admin/codesearch: create a group for shell access by role [puppet] - 10https://gerrit.wikimedia.org/r/1342807 (https://phabricator.wikimedia.org/T268199) [20:35:10] okay, thanks all [20:35:29] !log jhuneidi@deploy1003 robertsky, jhuneidi, cmelo, tsev: Continuing with deployment [20:47:42] !log jhuneidi@deploy1003 Finished scap sync-world: Backport for [[gerrit:1342774|Worklist Promotion test kitchen - Enable flag in production (T434513)]], [[gerrit:1342781|Exclude returntoapp query from app interception on iOS (T438395)]], [[gerrit:1342798|Revert "Update wikimania wordmark for 2026"]] (duration: 35m 59s) [20:47:47] T434513: Worklist Promotion test kitchen instrumentation - Enable flag in production - https://phabricator.wikimedia.org/T434513 [20:47:47] T438395: Visual Editor: Exclude returntoapp query item from app site association file - https://phabricator.wikimedia.org/T438395 [20:52:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 18.53% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:53:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.56% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:55:46] !log tsev@deploy1003 mwscript-k8s job started: purgeList.php # T438395 [20:55:49] T438395: Visual Editor: Exclude returntoapp query item from app site association file - https://phabricator.wikimedia.org/T438395 [21:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260917T2100) [21:03:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.56% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:03:56] !log tsev@deploy1003 mwscript-k8s job started: purgeList.php # T438395 [21:04:02] T438395: Visual Editor: Exclude returntoapp query item from app site association file - https://phabricator.wikimedia.org/T438395 [21:06:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.14% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:16:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.18% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:20:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:25:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:26:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:28:53] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342812 [22:11:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [22:16:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [22:38:40] FIRING: [4x] ProbeDown: Service registry1004:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:16:37] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:41:04] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1342820 [23:41:04] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1342820 (owner: 10TrainBranchBot) [23:50:36] (03CR) 10CI reject: [V:04-1] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1342820 (owner: 10TrainBranchBot)