[00:05:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [00:09:51] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1025.eqiad.wmnet with OS bookworm [00:20:07] FIRING: [2x] ProbeDown: Service aqs1025-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:22:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:25:07] FIRING: [4x] ProbeDown: Service aqs1025-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:27:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.04% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:28:02] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 62225456 and 32 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [00:29:02] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 69472 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [00:29:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:35:07] FIRING: [4x] ProbeDown: Service aqs1025-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:40:07] RESOLVED: [4x] ProbeDown: Service aqs1025-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:43:02] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 44394568 and 7 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [00:57:02] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3598256 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [01:04:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-esams and Deutsche Telekom (2001:7f8:1::a500:3320:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [01:11:02] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 72665608 and 10 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [01:11:46] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1338075 [01:11:46] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1338075 (owner: 10TrainBranchBot) [01:13:02] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2545784 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [01:15:10] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:18:02] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1338075 (owner: 10TrainBranchBot) [01:24:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:28:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.89% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:43:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:53:36] (03PS3) 10Samwilson: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (owner: 10Ladsgroup-claude) [01:55:38] FIRING: GnmiInterfaceCountersDrop: ... [01:55:38] asw1-bw27-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=asw1-bw27-esams:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [01:59:13] (03CR) 10Subramanya Sastry: [C:03+1] Enable Produnto on pilot wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324966 (https://phabricator.wikimedia.org/T421436) (owner: 10Tim Starling) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T0200) [02:00:42] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:02:12] (03CR) 10CI reject: [V:04-1] Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (owner: 10Ladsgroup-claude) [02:04:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr1-esams and Deutsche Telekom (2001:7f8:1::a500:3320:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [02:08:27] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 45s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:39] (03PS4) 10Samwilson: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [02:29:11] (03CR) 10CI reject: [V:04-1] Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [02:49:48] (03PS5) 10Samwilson: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [02:58:19] (03CR) 10CI reject: [V:04-1] Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [03:01:39] (03PS6) 10DLynch: editcheck-headless: Add deployment for the technical pilot [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) [03:02:23] (03CR) 10DLynch: editcheck-headless: Add deployment for the technical pilot (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [03:05:51] (03PS6) 10Samwilson: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [03:08:34] (03CR) 10CI reject: [V:04-1] Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [03:20:29] (03PS1) 10Andrea Denisse: kubernetes: Switch to GitLab origin for LibreNMS [puppet] - 10https://gerrit.wikimedia.org/r/1338078 (https://phabricator.wikimedia.org/T436578) [03:22:09] (03PS1) 10DLynch: editcheck-headless: Add deployment server users [puppet] - 10https://gerrit.wikimedia.org/r/1338079 (https://phabricator.wikimedia.org/T434109) [03:25:28] (03PS1) 10DLynch: editcheck-headless: Add ingress discovery records [dns] - 10https://gerrit.wikimedia.org/r/1338080 (https://phabricator.wikimedia.org/T434109) [03:33:09] (03PS1) 10DLynch: editcheck-headless: Add service catalog entry [puppet] - 10https://gerrit.wikimedia.org/r/1338081 (https://phabricator.wikimedia.org/T434109) [03:35:51] FIRING: Transit, peering or transport OUT traffic above 90% capacity - cr2-eqiad:xe-3/3/3 (Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, ... [03:35:56] MAC filter)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=eqiad+prometheus%2Fops&var-device=cr2-eqiad:9804&var-interface=xe-3%2F3%2F3 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [03:36:51] FIRING: [2x] CoreRouterInterfaceDropPercent: Excess priority packet drops on cr1-eqiad:ae6 (External: Equinix IXP 111916-DC6-IX-02) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#CoreRouterInterfaceDropPercent - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDropPercent [03:37:10] !incidents [03:37:11] 8331 (UNACKED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [03:37:11] 8332 (UNACKED) [2x] CoreRouterInterfaceDropPercent network sre (cr1-eqiad:9804 eqiad) [03:37:11] 8328 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [03:37:18] !ack [03:37:20] 8331 (ACKED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [03:37:20] 8332 (ACKED) [2x] CoreRouterInterfaceDropPercent network sre (cr1-eqiad:9804 eqiad) [03:41:51] FIRING: [4x] CoreRouterInterfaceDropPercent: Excess priority packet drops on cr1-eqiad:ae6 (External: Equinix IXP 111916-DC6-IX-02) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#CoreRouterInterfaceDropPercent - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDropPercent [03:42:01] !ack [03:42:02] All incidents are already acked. [03:45:51] FIRING: [2x] Transit, peering or transport OUT traffic above 90% capacity - cr1-eqiad:xe-3/0/6 (Peering: Equinix (21958836-A 111916-DC6-IX-02, MAC filter)) #page - https://w.wiki/Gbyf - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [03:46:22] !incidents [03:46:23] 8331 (ACKED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [03:46:24] 8332 (ACKED) [2x] CoreRouterInterfaceDropPercent network sre (cr1-eqiad:9804 eqiad) [03:46:24] 8328 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [03:55:39] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:56:15] (03PS7) 10Samwilson: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [04:01:51] FIRING: [4x] CoreRouterInterfaceDropPercent: Excess priority packet drops on cr1-eqiad:ae6 (External: Equinix IXP 111916-DC6-IX-02) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#CoreRouterInterfaceDropPercent - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDropPercent [04:02:11] !incidents [04:02:11] 8331 (ACKED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [04:02:11] 8332 (ACKED) [2x] CoreRouterInterfaceDropPercent network sre (cr1-eqiad:9804 eqiad) [04:02:12] 8328 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [04:03:32] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 09 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335706 (https://phabricator.wikimedia.org/T436426) (owner: 10Hamish) [04:05:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [04:05:51] FIRING: [2x] Transit, peering or transport OUT traffic above 90% capacity - cr1-eqiad:xe-3/0/6 (Peering: Equinix (21958836-A 111916-DC6-IX-02, MAC filter)) #page - https://w.wiki/Gbyf - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [04:05:57] !incidents [04:05:58] 8331 (ACKED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [04:05:58] 8332 (ACKED) [2x] CoreRouterInterfaceDropPercent network sre (cr1-eqiad:9804 eqiad) [04:05:58] 8328 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [04:06:12] (03CR) 10Samwilson: "Right, I think it's all sorted now. :)" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [04:06:51] RESOLVED: [4x] CoreRouterInterfaceDropPercent: Excess priority packet drops on cr1-eqiad:ae6 (External: Equinix IXP 111916-DC6-IX-02) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#CoreRouterInterfaceDropPercent - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDropPercent [04:07:21] (03CR) 10Samwilson: Run tests on CI (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [04:20:13] !incidents [04:20:14] 8331 (ACKED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [04:20:14] 8332 (RESOLVED) [2x] CoreRouterInterfaceDropPercent network sre (cr1-eqiad:9804 eqiad) [04:20:14] 8328 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [04:34:52] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:xe-1/0/1:0 (Transport: cr2-eqord:xe-0/1/5 (Arelion, IC-314533 24ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [04:45:51] RESOLVED: Transit, peering or transport OUT traffic above 90% capacity - cr2-eqiad:xe-3/3/3 (Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, ... [04:45:51] MAC filter)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=eqiad+prometheus%2Fops&var-device=cr2-eqiad:9804&var-interface=xe-3%2F3%2F3 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [04:46:14] !incidents [04:46:14] 8331 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [04:46:15] 8332 (RESOLVED) [2x] CoreRouterInterfaceDropPercent network sre (cr1-eqiad:9804 eqiad) [04:46:15] 8328 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [05:15:10] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:55:38] FIRING: GnmiInterfaceCountersDrop: ... [05:55:38] asw1-bw27-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=asw1-bw27-esams:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T0600) [06:27:54] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12301524 (10MoritzMuehlenhoff) [06:31:48] (03PS1) 10Samwilson: Switch from datetime.utcfromtimestamp() to .fromtimestamp() [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1338090 (https://phabricator.wikimedia.org/T437404) [06:31:55] (03CR) 10CI reject: [V:04-1] Switch from datetime.utcfromtimestamp() to .fromtimestamp() [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1338090 (https://phabricator.wikimedia.org/T437404) (owner: 10Samwilson) [06:35:16] (03CR) 10Slyngshede: "That is in fact the only thing this patch does. My understanding is that we do this, to indicate that we've switched DC to those who use t" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319818 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [06:38:55] (03PS2) 10Samwilson: Switch from datetime.utcfromtimestamp() to .fromtimestamp() [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1338090 (https://phabricator.wikimedia.org/T437404) [06:38:55] (03PS1) 10Jcrespo: Revert "admin: temp SSH key removal for jynus" [puppet] - 10https://gerrit.wikimedia.org/r/1338091 [06:51:27] (03CR) 10Jcrespo: [C:03+2] "+2 but I cannot deploy or test with my grants right now." [puppet] - 10https://gerrit.wikimedia.org/r/1331494 (owner: 10Muehlenhoff) [06:54:22] (03CR) 10Jcrespo: "What's the context? The body doesn't tell and the ticket is unclear. Was cumin1004 upgraded, removed, etc.?" [puppet] - 10https://gerrit.wikimedia.org/r/1332752 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [06:56:00] (03CR) 10Jcrespo: [C:03+2] "Same, I cannot deploy right now." [puppet] - 10https://gerrit.wikimedia.org/r/1332746 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [06:57:19] (03CR) 10Jcrespo: [C:03+1] Puppet 8: Replace unscoped legacy facts in module bacula [puppet] - 10https://gerrit.wikimedia.org/r/1332516 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [07:00:05] Amir1, urbanecm, and awight: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for UTC morning backport window . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T0700). [07:00:05] hamishcz: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:15] i can deploy [07:00:33] (03CR) 10Jcrespo: [C:03+2] "Ideally this is unused, but we keep it because there could be some processes using it on cloud or non core backups. This is all ok from my" [puppet] - 10https://gerrit.wikimedia.org/r/1335952 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [07:05:07] you around hamishcz? [07:07:08] (03PS1) 10Andrea Denisse: Revert "klaxon: Switch to GitLab origin" [puppet] - 10https://gerrit.wikimedia.org/r/1338092 [07:08:06] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] "LGTM lol" [puppet] - 10https://gerrit.wikimedia.org/r/1338092 (owner: 10Andrea Denisse) [07:09:23] (03CR) 10Marostegui: [C:03+1] "cumin1004 is a new trixie cumin and 1003 will eventually be decommissioned." [puppet] - 10https://gerrit.wikimedia.org/r/1332752 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [07:11:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:12:08] err hi im here [07:12:14] for the deployment... [07:12:21] awesome [07:12:32] you ready to start? [07:12:48] yes sry for waiting [07:12:54] all good :) [07:13:08] laptop got stuck just now... [07:13:22] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2196 crashed - https://phabricator.wikimedia.org/T437015#12301654 (10Marostegui) 05Open→03Resolved a:03Marostegui Thank you - I've started mariadb. I will close this for now as the follow up is being done via email with dell. We can reopen if needed. [07:13:41] (03CR) 10TrainBranchBot: [C:03+2] "Approved by chlod@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335706 (https://phabricator.wikimedia.org/T436426) (owner: 10Hamish) [07:13:54] (03PS1) 10Andrea Denisse: Revert "matomo: Allow dse-k8s pods to access database" [puppet] - 10https://gerrit.wikimedia.org/r/1338094 [07:14:02] that's okay [07:14:24] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] "Sorry for the revert but it wasn't merged on Puppetmaster and I needed to merge a change." [puppet] - 10https://gerrit.wikimedia.org/r/1338094 (owner: 10Andrea Denisse) [07:14:39] (03Merged) 10jenkins-bot: thwikibooks: update tagline and wordmark [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335706 (https://phabricator.wikimedia.org/T436426) (owner: 10Hamish) [07:14:49] (03PS3) 10Daniel Kertesz: admin: move dkertesz from ldap_only_users to users; add to ops-limited [puppet] - 10https://gerrit.wikimedia.org/r/1337863 (https://phabricator.wikimedia.org/T437271) [07:14:56] feel free to ping me for future deploys btw (here or on telegram), i know UTC morning is usually unattended and i see you've tried to get this patch out already before [07:15:43] !log chlod@deploy1003 Started scap sync-world: Backport for [[gerrit:1335706|thwikibooks: update tagline and wordmark (T436426)]] [07:15:47] T436426: Requesting logo change for th.wikibooks.org - https://phabricator.wikimedia.org/T436426 [07:15:56] ah okay so nice of you my frd [07:16:01] :h [07:18:10] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to production for dkertesz - https://phabricator.wikimedia.org/T437271#12301662 (10dkertesz) I have amended the CR to put myself into the `ops-limited` group [07:20:02] !log chlod@deploy1003 chlod, hamishz: Backport for [[gerrit:1335706|thwikibooks: update tagline and wordmark (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:20:14] Hamishcz_: check if the logo looks good? [07:20:18] verifying [07:20:19] a sec [07:23:34] weird [07:23:42] can load out the logo [07:23:47] lemme try another device [07:24:08] gotcha [07:26:01] (03PS1) 10Andrea Denisse: Revert^2 "klaxon: Switch to GitLab origin" [puppet] - 10https://gerrit.wikimedia.org/r/1338095 [07:26:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:26:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:27:39] (03PS1) 10Muehlenhoff: Blocklist MPLS [puppet] - 10https://gerrit.wikimedia.org/r/1338096 [07:27:56] fwiw, for logo/tagline/wordmark changes you may need to hard refresh to wipe your browser's local cache [07:28:08] the server should be set to k8s-server right? [07:28:18] k8s-mwdebug yes [07:28:29] and it should be on, of course [07:29:01] off [07:29:30] ofc [07:29:54] I’ll go to boot my Windows [07:30:35] allow me another sec [07:31:03] sure thing [07:31:09] (03CR) 10Ayounsi: [C:03+1] Blocklist MPLS [puppet] - 10https://gerrit.wikimedia.org/r/1338096 (owner: 10Muehlenhoff) [07:31:23] and yes I did hard refresh but it’s still empty on my laptop [07:31:49] Okay confirmed as expected [07:31:52] good to go [07:31:57] sry for waiting [07:32:30] alrighty, proceeding [07:32:35] !log chlod@deploy1003 chlod, hamishz: Continuing with deployment [07:32:43] (03PS1) 10Santiago Faci: Test Kitchen UI: Deploy v1.5.4 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338098 (https://phabricator.wikimedia.org/T421813) [07:32:46] (03PS2) 10Andrea Denisse: klaxon: Switch to GitLab origin [puppet] - 10https://gerrit.wikimedia.org/r/1338095 (https://phabricator.wikimedia.org/T436694) [07:33:22] (03CR) 10CI reject: [V:04-1] klaxon: Switch to GitLab origin [puppet] - 10https://gerrit.wikimedia.org/r/1338095 (https://phabricator.wikimedia.org/T436694) (owner: 10Andrea Denisse) [07:33:34] (03PS3) 10Andrea Denisse: klaxon: Switch to GitLab origin [puppet] - 10https://gerrit.wikimedia.org/r/1338095 (https://phabricator.wikimedia.org/T436694) [07:33:52] (03CR) 10Andrea Denisse: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1338095 (https://phabricator.wikimedia.org/T436694) (owner: 10Andrea Denisse) [07:34:14] (03CR) 10CI reject: [V:04-1] klaxon: Switch to GitLab origin [puppet] - 10https://gerrit.wikimedia.org/r/1338095 (https://phabricator.wikimedia.org/T436694) (owner: 10Andrea Denisse) [07:34:55] btw do we have a channel on Telegram for this chat? [07:35:03] nope, all on IRC [07:35:15] (03PS4) 10Andrea Denisse: klaxon: Switch to GitLab origin [puppet] - 10https://gerrit.wikimedia.org/r/1338095 (https://phabricator.wikimedia.org/T436694) [07:35:30] (03CR) 10Andrea Denisse: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1338095 (https://phabricator.wikimedia.org/T436694) (owner: 10Andrea Denisse) [07:36:00] But I thought you said I can find you there? or you meant WMCS channel [07:36:32] (03CR) 10Andrea Denisse: [C:03+2] klaxon: Switch to GitLab origin [puppet] - 10https://gerrit.wikimedia.org/r/1338095 (https://phabricator.wikimedia.org/T436694) (owner: 10Andrea Denisse) [07:36:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:36:52] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338096 (owner: 10Muehlenhoff) [07:37:20] !log chlod@deploy1003 Finished scap sync-world: Backport for [[gerrit:1335706|thwikibooks: update tagline and wordmark (T436426)]] (duration: 21m 36s) [07:37:23] T436426: Requesting logo change for th.wikibooks.org - https://phabricator.wikimedia.org/T436426 [07:37:31] and let me just quickly run the cache purges... [07:38:03] new wordmark/tagline should be live now :) [07:38:11] yes it works [07:38:18] finally [07:39:51] Thank you! [07:40:24] (03CR) 10Cathal Mooney: [C:03+1] Blocklist MPLS [puppet] - 10https://gerrit.wikimedia.org/r/1338096 (owner: 10Muehlenhoff) [07:40:29] no prob :D [07:40:52] nothing else in the window so closing it here [07:40:53] !log UTC morning backport window done [07:40:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:44:14] (03CR) 10Cathal Mooney: [C:03+1] "Yes it's all good, the patch is definitely a good idea. Like I say we should probably do a test to verify at some stage, and observe the " [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 (owner: 10Ssingh) [07:47:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:55:29] (03CR) 10Elukey: "Terribly sorry for the lag, I added some comments, let me know!" [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [07:55:39] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:00:21] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12301743 (10BTullis) p:05Triage→03Low [08:00:24] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:00:54] !log btullis@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1198.eqiad.wmnet [08:01:03] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1198.eqiad.wmnet [08:01:24] !log btullis@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1198.eqiad.wmnet [08:01:28] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1198.eqiad.wmnet [08:01:39] (03PS5) 10Giuseppe Lavagetto: aptrepo: add thirdparty/gvisor to newer distros as well [puppet] - 10https://gerrit.wikimedia.org/r/1328540 (https://phabricator.wikimedia.org/T435758) [08:02:53] (03CR) 10Giuseppe Lavagetto: [C:03+2] aptrepo: add thirdparty/gvisor to newer distros as well [puppet] - 10https://gerrit.wikimedia.org/r/1328540 (https://phabricator.wikimedia.org/T435758) (owner: 10Giuseppe Lavagetto) [08:03:47] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1260.eqiad.wmnet with reason: Cloning sanitarium [08:04:51] (03PS1) 10Marostegui: db1260: Add note [puppet] - 10https://gerrit.wikimedia.org/r/1338101 (https://phabricator.wikimedia.org/T431309) [08:05:11] (03PS1) 10Marostegui: Revert "db2196: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1338102 [08:05:18] (03CR) 10Marostegui: [C:04-2] "Not ready yet." [puppet] - 10https://gerrit.wikimedia.org/r/1338102 (owner: 10Marostegui) [08:05:34] (03CR) 10Marostegui: "This is a noop" [puppet] - 10https://gerrit.wikimedia.org/r/1338101 (https://phabricator.wikimedia.org/T431309) (owner: 10Marostegui) [08:05:46] (03CR) 10Marostegui: [C:03+2] db1260: Add note [puppet] - 10https://gerrit.wikimedia.org/r/1338101 (https://phabricator.wikimedia.org/T431309) (owner: 10Marostegui) [08:06:20] 10ops-eqiad, 06SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12301769 (10Gehel) a:05klausman→03RobH [08:06:22] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1260: Needs to clone another host from this one [08:06:46] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1260: Needs to clone another host from this one [08:10:23] RESOLVED: GnmiInterfaceCountersDrop: ... [08:10:23] asw1-bw27-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=asw1-bw27-esams:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [08:10:36] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip for mobileapps, proton, push-notifications codfw [puppet] - 10https://gerrit.wikimedia.org/r/1337923 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:10:53] (03PS1) 10Marostegui: db1155: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1338103 (https://phabricator.wikimedia.org/T431309) [08:10:54] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip for mobileapps, proton, push-notifications eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1337924 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:12:04] (03CR) 10Marostegui: [C:03+2] db1155: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1338103 (https://phabricator.wikimedia.org/T431309) (owner: 10Marostegui) [08:13:09] (03CR) 10Giuseppe Lavagetto: [C:03+2] profile::containerd: format according to our style guide [puppet] - 10https://gerrit.wikimedia.org/r/1330128 (owner: 10Giuseppe Lavagetto) [08:17:02] PROBLEM - mysqld processes on db1155 is CRITICAL: PROCS CRITICAL: 4 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [08:19:02] 06SRE, 10Data-Persistence-Backup, 10database-backups: Put db2201 back into backup production as a backup source - https://phabricator.wikimedia.org/T437411 (10jcrespo) 03NEW [08:19:08] 06SRE, 10Data-Persistence-Backup, 10database-backups: Put db2201 back into backup production as a backup source - https://phabricator.wikimedia.org/T437411#12301808 (10jcrespo) p:05Triage→03Medium [08:20:34] (03CR) 10Fabfur: [C:03+1] admin: move dkertesz from ldap_only_users to users; add to ops-limited [puppet] - 10https://gerrit.wikimedia.org/r/1337863 (https://phabricator.wikimedia.org/T437271) (owner: 10Daniel Kertesz) [08:21:19] (03CR) 10Phuedx: [C:03+1] Test Kitchen UI: Deploy v1.5.4 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337862 (https://phabricator.wikimedia.org/T421813) (owner: 10Santiago Faci) [08:21:24] (03CR) 10Phuedx: [C:03+1] Test Kitchen UI: Deploy v1.5.4 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338098 (https://phabricator.wikimedia.org/T421813) (owner: 10Santiago Faci) [08:22:14] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove golang1.5 production image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337931 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [08:23:57] (03PS1) 10Brouberol: setup a RR DNS for +druid-analytics.svc.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/1338123 (https://phabricator.wikimedia.org/T437310) [08:26:54] PROBLEM - MariaDB Replica SQL: x4 on db1155 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [08:27:06] (03CR) 10Btullis: [C:03+1] "LGTM." [dns] - 10https://gerrit.wikimedia.org/r/1338123 (https://phabricator.wikimedia.org/T437310) (owner: 10Brouberol) [08:27:38] (03CR) 10Brouberol: [C:03+2] setup a RR DNS for +druid-analytics.svc.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/1338123 (https://phabricator.wikimedia.org/T437310) (owner: 10Brouberol) [08:28:23] !log brouberol@dns1004 START - running authdns-update [08:28:35] !log pruned obsolete Bullseye image golang1.15 from the docker registry T416452 [08:28:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:28:38] T416452: Migrate Docker images running in Production away from Bullseye - https://phabricator.wikimedia.org/T416452 [08:30:33] !log brouberol@dns1004 END - running authdns-update [08:31:54] PROBLEM - MariaDB read only x4 on db1155 is CRITICAL: Could not connect to localhost:3364 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [08:35:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:xe-1/0/1:0 (Transport: cr2-eqord:xe-0/1/5 (Arelion, IC-314533 24ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [08:36:05] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1155.eqiad.wmnet with reason: Cloning sanitarium [08:36:23] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] "The images(s) were pruned from the registry" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337931 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [08:36:56] (03CR) 10Muehlenhoff: [C:03+2] Blocklist MPLS [puppet] - 10https://gerrit.wikimedia.org/r/1338096 (owner: 10Muehlenhoff) [08:37:31] (03PS4) 10Daniel Kertesz: admin: move dkertesz from ldap_only_users to users; add to ops-limited [puppet] - 10https://gerrit.wikimedia.org/r/1337863 (https://phabricator.wikimedia.org/T437271) [08:38:32] (03PS1) 10Cathal Mooney: Remove vrrp_peer var for cr3-eqsin [puppet] - 10https://gerrit.wikimedia.org/r/1338125 (https://phabricator.wikimedia.org/T435406) [08:39:16] (03CR) 10Brouberol: [C:03+2] global_config: add the LVS VIPs to the public druid external service endpoints [puppet] - 10https://gerrit.wikimedia.org/r/1338021 (https://phabricator.wikimedia.org/T437310) (owner: 10Brouberol) [08:39:52] (03CR) 10Fabfur: [C:03+2] admin: move dkertesz from ldap_only_users to users; add to ops-limited [puppet] - 10https://gerrit.wikimedia.org/r/1337863 (https://phabricator.wikimedia.org/T437271) (owner: 10Daniel Kertesz) [08:40:59] (03CR) 10Ladsgroup: [C:03+2] Switch from datetime.utcfromtimestamp() to .fromtimestamp() [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1338090 (https://phabricator.wikimedia.org/T437404) (owner: 10Samwilson) [08:41:35] (03PS1) 10Muehlenhoff: Remove Python 2 exception from Hadoop nodes [puppet] - 10https://gerrit.wikimedia.org/r/1338126 [08:46:17] (03Merged) 10jenkins-bot: Switch from datetime.utcfromtimestamp() to .fromtimestamp() [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1338090 (https://phabricator.wikimedia.org/T437404) (owner: 10Samwilson) [08:46:58] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06DC-Ops: Disk (sdf) failed in ms-be2089 - https://phabricator.wikimedia.org/T437293#12301870 (10MatthewVernon) @Jhancock.wm looks good. Thanks for the rapid fix :-) [08:48:27] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device ssw1-d1-codfw [08:48:44] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d1-codfw [08:49:01] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-d3-codfw [08:49:13] (03CR) 10Ladsgroup: [C:03+2] "Thank you!" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [08:49:19] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d3-codfw [08:49:29] 10SRE-swift-storage, 10Cloud-VPS (Debian Bullseye Deprecation): Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12301877 (10MatthewVernon) @komla you'll want to ask @Eevans as I think the remaining bullseye systems are his; I note, though, that we're still... [08:50:10] (03CR) 10CI reject: [V:04-1] Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [08:55:06] (03CR) 10Giuseppe Lavagetto: "I've moved the shim options in the template, but I think the location of the runsc_config where it is makes sense." [puppet] - 10https://gerrit.wikimedia.org/r/1330129 (https://phabricator.wikimedia.org/T435796) (owner: 10Giuseppe Lavagetto) [08:55:54] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [08:55:58] (03CR) 10Ladsgroup: [C:03+2] "let me fix the breakage. It's simple." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [08:56:29] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [08:58:39] (03PS8) 10Ladsgroup: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [08:58:59] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 22 hosts with reason: upgrade ssw1-a1-eqiad [08:59:06] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12301906 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=bb00a99c-5cd5-4943-9771-6bfb08321711) set by cmooney@cumin1004 for 2:00:00 o... [09:01:17] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: upgrade ssw1-a1-eqiad [09:01:24] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12301920 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=bf370135-d73c-4a25-91fa-c29757024f16) set by cmooney@cumin1004 for 2:00:00 o... [09:02:49] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [09:03:28] (03PS1) 10FNegri: wmcs:db:wikireplicas: remove unused template [puppet] - 10https://gerrit.wikimedia.org/r/1338127 [09:03:47] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [09:04:00] (03CR) 10FNegri: [V:03+1] "Checked on clouddb1025 after merge, the puppet run was a no-op as expected." [puppet] - 10https://gerrit.wikimedia.org/r/1335918 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [09:05:39] (03CR) 10CI reject: [V:04-1] Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [09:06:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.59% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:11:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:15:10] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:17:40] (03PS9) 10Ladsgroup: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [09:17:54] RECOVERY - MariaDB Replica SQL: x4 on db1155 is OK: OK slave_sql_state not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [09:17:56] RECOVERY - MariaDB read only x4 on db1155 is OK: Version 10.11.16-MariaDB-log, Uptime 22s, read_only: True, event_scheduler: True, 12.34 QPS, connection latency: 0.030084s, query latency: 0.000395s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [09:18:02] RECOVERY - mysqld processes on db1155 is OK: PROCS OK: 5 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [09:19:23] (03CR) 10CI reject: [V:04-1] Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [09:20:38] (03PS10) 10Ladsgroup: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [09:21:00] (03CR) 10JMeybohm: [C:03+2] vap: Makefile improvements [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337982 (owner: 10JMeybohm) [09:21:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.63% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:24:02] PROBLEM - BFD status on ssw1-a1-codfw.mgmt is CRITICAL: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [09:25:10] FIRING: BFDdown: BFD session down between ssw1-a1-codfw and 10.192.252.2 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-a1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:25:39] FIRING: CoreBGPDown: Core BGP session down between ssw1-a1-codfw and ssw1-a8-codfw (10.192.252.2) - group EVPN_IBGP - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=ssw1-a1-codfw:9804&var-bgp_group=EVPN_IBGP&var-bgp_neighbor=ssw1-a8-codfw - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:27:58] !log btullis@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1198.eqiad.wmnet [09:27:59] (03CR) 10Santiago Faci: [C:03+2] Test Kitchen UI: Deploy v1.5.4 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337862 (https://phabricator.wikimedia.org/T421813) (owner: 10Santiago Faci) [09:28:01] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1198.eqiad.wmnet [09:28:24] (03CR) 10Clément Goubert: [C:03+1] editcheck-headless: Add deployment server users [puppet] - 10https://gerrit.wikimedia.org/r/1338079 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [09:30:20] (03PS5) 10Btullis: ceph: Make the CephX import guard compare the key [puppet] - 10https://gerrit.wikimedia.org/r/1337876 (https://phabricator.wikimedia.org/T437233) [09:30:21] (03PS1) 10Btullis: ceph: Install the CephX verify script only where it works [puppet] - 10https://gerrit.wikimedia.org/r/1338130 (https://phabricator.wikimedia.org/T437233) [09:30:23] (03PS1) 10Btullis: ceph: Correct four faults in verify-cephx-keys [puppet] - 10https://gerrit.wikimedia.org/r/1338131 (https://phabricator.wikimedia.org/T437233) [09:31:04] (03PS1) 10Filippo Giunchedi: hieradata: organize wmcs cluster into separate cloud clusters [puppet] - 10https://gerrit.wikimedia.org/r/1338132 (https://phabricator.wikimedia.org/T437272) [09:33:07] (03Merged) 10jenkins-bot: vap: Makefile improvements [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337982 (owner: 10JMeybohm) [09:33:52] (03CR) 10Clément Goubert: [C:03+1] "LGTM if `editcheck-headless` is effectively supposed to be active/active (writes directly to its local datacenter's datastore, or doesn't " [dns] - 10https://gerrit.wikimedia.org/r/1338080 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [09:34:05] 06SRE-OnFire, 06MediaWiki-Engineering, 06ServiceOps, 07Sustainability (Incident Followup): Reduce the amount of messages sent through channel:Memcached during failures - https://phabricator.wikimedia.org/T390529#12302019 (10MLechvien-WMF) @OWresch-WMF @MSantos would you be able to triage this to MW Eng tea... [09:34:59] (03CR) 10Filippo Giunchedi: "A proposal, let me know what you think !" [puppet] - 10https://gerrit.wikimedia.org/r/1338132 (https://phabricator.wikimedia.org/T437272) (owner: 10Filippo Giunchedi) [09:35:02] RECOVERY - BFD status on ssw1-a1-codfw.mgmt is OK: UP: 17 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [09:35:03] (03PS1) 10Brouberol: mediawiki-dumps-legacy: upgrade the image used by the dump-toolbox [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338133 (https://phabricator.wikimedia.org/T416455) [09:35:59] (03CR) 10Btullis: [C:03+1] mediawiki-dumps-legacy: upgrade the image used by the dump-toolbox [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338133 (https://phabricator.wikimedia.org/T416455) (owner: 10Brouberol) [09:36:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:36:26] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove Java 8/Bullseye and Java 11 production images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337929 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [09:37:57] (03CR) 10Brouberol: [C:03+2] mediawiki-dumps-legacy: upgrade the image used by the dump-toolbox [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338133 (https://phabricator.wikimedia.org/T416455) (owner: 10Brouberol) [09:39:39] (03Merged) 10jenkins-bot: Test Kitchen UI: Deploy v1.5.4 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337862 (https://phabricator.wikimedia.org/T421813) (owner: 10Santiago Faci) [09:40:10] RESOLVED: BFDdown: BFD session down between ssw1-a1-codfw and 10.192.252.2 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-a1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:40:39] RESOLVED: CoreBGPDown: Core BGP session down between ssw1-a1-codfw and ssw1-a8-codfw (10.192.252.2) - group EVPN_IBGP - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=ssw1-a1-codfw:9804&var-bgp_group=EVPN_IBGP&var-bgp_neighbor=ssw1-a8-codfw - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDow [09:43:20] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [09:43:21] (03PS1) 10Arnaudb: cache: pipe websocket upgrades for the gitlab hostnames [puppet] - 10https://gerrit.wikimedia.org/r/1338134 (https://phabricator.wikimedia.org/T425441) [09:43:50] (03PS2) 10Arnaudb: cache: pipe websocket upgrades for the gitlab hostnames [puppet] - 10https://gerrit.wikimedia.org/r/1338134 (https://phabricator.wikimedia.org/T425441) [09:44:43] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM, when the time comes" [puppet] - 10https://gerrit.wikimedia.org/r/1337876 (https://phabricator.wikimedia.org/T437233) (owner: 10Btullis) [09:46:09] (03CR) 10Clément Goubert: [C:03+1] editcheck-headless: Add service catalog entry [puppet] - 10https://gerrit.wikimedia.org/r/1338081 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [09:47:33] (03PS1) 10Marostegui: check_private_data_report: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1338135 (https://phabricator.wikimedia.org/T431309) [09:48:38] (03CR) 10Marostegui: "Host green in icinga." [puppet] - 10https://gerrit.wikimedia.org/r/1338102 (owner: 10Marostegui) [09:48:42] (03CR) 10Marostegui: [C:03+2] Revert "db2196: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1338102 (owner: 10Marostegui) [09:48:44] (03PS1) 10Elukey: docker_registry: remove the Swift support [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) [09:49:31] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1338135 (https://phabricator.wikimedia.org/T431309) (owner: 10Marostegui) [09:49:56] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12302079 (10cmooney) [09:50:19] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12302083 (10cmooney) 05Open→03Resolved a:03cmooney Closing this one out, all devices have been upgraded to 23.4R2 now. [09:50:34] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [09:51:56] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [09:52:35] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-d3-codfw [09:52:53] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d3-codfw [09:53:35] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1260: Repooling after cloning [09:55:53] !log pruned obsolete Bullseye images openjdk-8-jdk/openjdk-8-jre/openjdk-11-jre/openjdk-11-jdk from the docker registry T416452 [09:55:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:55:57] T416452: Migrate Docker images running in Production away from Bullseye - https://phabricator.wikimedia.org/T416452 [09:56:23] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] "openjdk-8-jdk, openjdk-8-jre, openjdk-11-jdk and openjdk-11-jre have been pruned from the registry" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337929 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [09:57:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:58:40] 06SRE, 06Commons, 10MediaWiki-File-management, 06Traffic, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12302139 (10OttoG) Today, it took about an hour for the original SVG content of this file to be updated t... [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1000) [10:01:42] (03CR) 10Ladsgroup: [C:03+2] "Getting the tests running again" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [10:02:58] (03CR) 10Samwilson: [C:03+2] "Looks good!" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [10:05:44] (03CR) 10Samwilson: [C:03+2] Run tests on CI (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [10:06:44] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [10:07:08] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [10:07:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:09:56] (03CR) 10Sergio Gimeno: [C:03+1] [Growth] Enable iterative Add Link task pool population on all wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338031 (https://phabricator.wikimedia.org/T392944) (owner: 10Urbanecm) [10:09:59] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db2196: Repooling db2196 [10:10:09] (03Merged) 10jenkins-bot: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup-claude) [10:10:24] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-d3-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:10:27] jouncebot: nowandnext [10:10:27] For the next 0 hour(s) and 49 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1000) [10:10:27] In 0 hour(s) and 49 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1100) [10:10:54] (03CR) 10Samwilson: "Oops, that was me, sorry!" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337608 (owner: 10Ladsgroup) [10:14:51] (03CR) 10Clément Goubert: [C:03+1] editcheck-headless: Add deployment for the technical pilot [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [10:21:35] (03PS2) 10Santiago Faci: Test Kitchen UI: Deploy v1.5.4 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338098 (https://phabricator.wikimedia.org/T421813) [10:23:34] (03PS3) 10Santiago Faci: Test Kitchen UI: Deploy v1.5.4 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338098 (https://phabricator.wikimedia.org/T421813) [10:26:16] (03CR) 10Clément Goubert: "I think we should make a first CR that only enables it to staging, test it thoroughly there, then move on to production. That way even if " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335285 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [10:28:48] (03PS4) 10Santiago Faci: Test Kitchen UI: Deploy v1.5.4 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338098 (https://phabricator.wikimedia.org/T421813) [10:30:48] 10SRE-swift-storage: Unknown error while undeleting a file on Wikimedia Commons - https://phabricator.wikimedia.org/T437114#12302242 (10MatthewVernon) So this is https://commons.wikimedia.org/wiki/File:%D9%88%DB%8E%D9%86%D9%87%E2%80%8C%DB%8C_%DA%A9%DB%8E%D8%B4%D8%B1%D8%A7%D9%88%DB%8C_%DA%A9%DA%86%DB%8C_%DA%A9%D8... [10:33:02] (03CR) 10Muehlenhoff: [C:03+1] "Sounds good" [puppet] - 10https://gerrit.wikimedia.org/r/1329227 (owner: 10Hashar) [10:33:23] (03PS1) 10Ladsgroup: Move testcommonswiki links tables to x4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338142 (https://phabricator.wikimedia.org/T398709) [10:33:47] (03PS1) 10Santiago Faci: test-kitchen: Add `validate_deployment_windows` config property [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338143 (https://phabricator.wikimedia.org/T433733) [10:34:25] (03PS5) 10Santiago Faci: Test Kitchen UI: Deploy v1.5.4 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338098 (https://phabricator.wikimedia.org/T421813) [10:36:32] (03CR) 10Clément Goubert: [C:03+1] Stop building Node 12/14/16 images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337934 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [10:38:41] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1260: Repooling after cloning [10:41:54] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12302321 (10MoritzMuehlenhoff) [10:42:21] (03PS3) 10Samwilson: Set development log level to 'debug' [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337196 [10:43:09] !log installing Bird security updates [10:43:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:44:19] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Stop building Node 12/14/16 images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337934 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [10:44:52] I'm gonna take the last few minutes of the infra window to remove a bunch of graphite puppet config [10:46:52] (03PS2) 10Hnowlan: graphite: remove module, references, config [puppet] - 10https://gerrit.wikimedia.org/r/1332767 (https://phabricator.wikimedia.org/T435340) [10:47:00] !log pruned obsolete Bullseye images nodejs12-slim/nodejs12-devel/nodejs14-slim/nodejs16-slim from the docker registry T416452 [10:47:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:47:04] T416452: Migrate Docker images running in Production away from Bullseye - https://phabricator.wikimedia.org/T416452 [10:47:30] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] "nodejs12-slim/nodejs12-devel/nodejs14-slim/nodejs16-slim have been pruned from the Docker registry" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337934 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [10:49:40] (03CR) 10CI reject: [V:04-1] graphite: remove module, references, config [puppet] - 10https://gerrit.wikimedia.org/r/1332767 (https://phabricator.wikimedia.org/T435340) (owner: 10Hnowlan) [10:50:39] (03CR) 10Phuedx: [C:03+1] test-kitchen: Add `validate_deployment_windows` config property [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338143 (https://phabricator.wikimedia.org/T433733) (owner: 10Santiago Faci) [10:51:06] (03PS1) 10Muehlenhoff: Remove dispatch images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1338145 (https://phabricator.wikimedia.org/T416452) [10:51:50] (03CR) 10Hnowlan: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1333749 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [10:55:06] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2196: Repooling db2196 [10:57:32] (03CR) 10Santiago Faci: [C:03+2] test-kitchen: Add `validate_deployment_windows` config property [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338143 (https://phabricator.wikimedia.org/T433733) (owner: 10Santiago Faci) [11:00:03] (03Merged) 10jenkins-bot: test-kitchen: Add `validate_deployment_windows` config property [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338143 (https://phabricator.wikimedia.org/T433733) (owner: 10Santiago Faci) [11:00:04] mvolz: OwO what's this, a deployment window?? Services – Citoid / Zotero. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1100). nyaa~ [11:06:50] (03PS1) 10Ladsgroup: Thumbor: Bump to the most recent merged commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338149 (https://phabricator.wikimedia.org/T437398) [11:07:42] (03PS1) 10Santiago Faci: test-kitchen: Bump chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338150 (https://phabricator.wikimedia.org/T433733) [11:08:13] (03PS1) 10Tryvix1509: core-Namespaces.php: Disallow indexing on talk namespaces on ukwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338151 (https://phabricator.wikimedia.org/T437409) [11:08:50] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 09 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#de" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338151 (https://phabricator.wikimedia.org/T437409) (owner: 10Tryvix1509) [11:12:01] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Migrate remaining container build/report steps from build2001 to build2004 - https://phabricator.wikimedia.org/T417389#12302398 (10MoritzMuehlenhoff) The pytorch errors are unrelated and I've opened https://phabricator.wikimedia.org/T437427 for the ML... [11:14:14] (03CR) 10Brouberol: [C:03+1] ceph: Install the CephX verify script only where it works [puppet] - 10https://gerrit.wikimedia.org/r/1338130 (https://phabricator.wikimedia.org/T437233) (owner: 10Btullis) [11:15:28] (03CR) 10Brouberol: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1338131 (https://phabricator.wikimedia.org/T437233) (owner: 10Btullis) [11:19:56] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337861 (owner: 10Muehlenhoff) [11:20:06] (03CR) 10Clément Goubert: [C:03+2] editcheck-headless: Add deployment server users [puppet] - 10https://gerrit.wikimedia.org/r/1338079 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [11:24:04] (03CR) 10Ladsgroup: "Based on output of I2ef44ae6f2970e5 (https://integration.wikimedia.org/ci/job/thumbor-plugins-pipeline-publish/133/console comparing libd" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337861 (owner: 10Muehlenhoff) [11:25:44] (03PS1) 10Muehlenhoff: Record LDAP access for dpislaru [puppet] - 10https://gerrit.wikimedia.org/r/1338155 [11:26:57] (03CR) 10Ayounsi: [C:03+1] Remove vrrp_peer var for cr3-eqsin [puppet] - 10https://gerrit.wikimedia.org/r/1338125 (https://phabricator.wikimedia.org/T435406) (owner: 10Cathal Mooney) [11:28:02] (03CR) 10Cathal Mooney: [C:03+2] Remove vrrp_peer var for cr3-eqsin [puppet] - 10https://gerrit.wikimedia.org/r/1338125 (https://phabricator.wikimedia.org/T435406) (owner: 10Cathal Mooney) [11:28:49] (03CR) 10Muehlenhoff: [C:03+2] Record LDAP access for dpislaru [puppet] - 10https://gerrit.wikimedia.org/r/1338155 (owner: 10Muehlenhoff) [11:29:03] jouncebot: nowandnext [11:29:03] For the next 0 hour(s) and 30 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1100) [11:29:03] In 1 hour(s) and 30 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1300) [11:29:14] (03PS1) 10Hnowlan: icinga: remove dead ElasticSearch checks breaking CI [puppet] - 10https://gerrit.wikimedia.org/r/1338157 [11:29:31] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338031 (https://phabricator.wikimedia.org/T392944) (owner: 10Urbanecm) [11:30:41] (03Merged) 10jenkins-bot: [Growth] Enable iterative Add Link task pool population on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338031 (https://phabricator.wikimedia.org/T392944) (owner: 10Urbanecm) [11:31:03] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1338031|[Growth] Enable iterative Add Link task pool population on all wikis (T392944)]] [11:31:07] T392944: Enable the iterative way of refreshing LinkRecommendations for all wikis - https://phabricator.wikimedia.org/T392944 [11:32:51] (03CR) 10Hnowlan: [C:03+1] Thumbor: Bump to the most recent merged commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338149 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup) [11:35:26] !log urbanecm@deploy1003 urbanecm: Backport for [[gerrit:1338031|[Growth] Enable iterative Add Link task pool population on all wikis (T392944)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:36:10] (03CR) 10Muehlenhoff: "Ack, looks good! I'll abandon the patch." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337861 (owner: 10Muehlenhoff) [11:36:26] (03Abandoned) 10Muehlenhoff: thumbor-plugins: Rebuild against latest package versions in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337861 (owner: 10Muehlenhoff) [11:45:22] FIRING: [2x] GnmiInterfaceCountersDrop: lsw1-a7-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [11:48:16] !log urbanecm@deploy1003 urbanecm: Continuing with deployment [11:53:01] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338031|[Growth] Enable iterative Add Link task pool population on all wikis (T392944)]] (duration: 21m 58s) [11:53:04] T392944: Enable the iterative way of refreshing LinkRecommendations for all wikis - https://phabricator.wikimedia.org/T392944 [12:00:37] (03CR) 10Ladsgroup: [C:03+2] Thumbor: Bump to the most recent merged commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338149 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup) [12:03:15] (03Merged) 10jenkins-bot: Thumbor: Bump to the most recent merged commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338149 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup) [12:06:56] Any deployers available? [12:11:57] FIRING: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:12:07] (03CR) 10Ozge: [C:03+1] "resolving the comments as it is in the your turn list for awhile" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1305382 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [12:12:31] (03PS2) 10Elukey: docker_registry: remove the Swift support [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) [12:12:39] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [12:13:52] (03CR) 10Ozge: [V:03+2 C:03+2] "I think it's good to keep this in mind as we may need it in the future e.g. working with the references." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306915 (https://phabricator.wikimedia.org/T430812) (owner: 10Ozge) [12:16:20] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [12:16:37] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [12:16:57] RESOLVED: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:20:55] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.14 point update - https://phabricator.wikimedia.org/T426759#12302792 (10MoritzMuehlenhoff) [12:21:45] thumbor on staging [12:21:46] > HTTP/1.1 504 Gateway Time-out [12:21:56] (03CR) 10Phuedx: [C:03+1] test-kitchen: Bump chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338150 (https://phabricator.wikimedia.org/T433733) (owner: 10Santiago Faci) [12:22:56] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [12:25:51] k8s logs look okay though. weird [12:27:36] Amir1: Wondering if you are available for wmf-config patch deployment [12:27:50] what is the patch? [12:28:19] if it's straightforward, I can push it while debugging thumbor [12:28:19] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1338151 [12:29:24] (03CR) 10Btullis: [C:03+2] admin_ng: Add tool-server namespace to LiftWing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337869 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [12:29:59] sure [12:29:59] (03CR) 10Btullis: [C:03+2] profile::k8s::deployment_server: add config for tool-server [puppet] - 10https://gerrit.wikimedia.org/r/1337888 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [12:30:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338151 (https://phabricator.wikimedia.org/T437409) (owner: 10Tryvix1509) [12:31:04] perfect, thanks [12:31:13] (03CR) 10Elukey: "I am going to wait some other days before removing everything, since we may need to copy images etc.." [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [12:31:20] (03Merged) 10jenkins-bot: core-Namespaces.php: Disallow indexing on talk namespaces on ukwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338151 (https://phabricator.wikimedia.org/T437409) (owner: 10Tryvix1509) [12:31:21] Note: WikimediaDebug already installed. [12:31:39] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1338151|core-Namespaces.php: Disallow indexing on talk namespaces on ukwiki (T437409)]] [12:31:42] T437409: Disallow indexing on talk namespaces on ukwiki - https://phabricator.wikimedia.org/T437409 [12:33:57] (03PS1) 10Majavah: P:wmcs::novaproxy: Set maximum connection pool size for backends [puppet] - 10https://gerrit.wikimedia.org/r/1338185 (https://phabricator.wikimedia.org/T437350) [12:34:17] (03CR) 10Urbanecm: [Growth] Enable iterative Add Link task pool population on all wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338031 (https://phabricator.wikimedia.org/T392944) (owner: 10Urbanecm) [12:34:57] FIRING: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:35:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:xe-1/0/1:0 (Transport: cr2-eqord:xe-0/1/5 (Arelion, IC-314533 24ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [12:35:42] (03CR) 10Filippo Giunchedi: [C:03+1] Remove dispatch images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1338145 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [12:35:45] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9390/co" [puppet] - 10https://gerrit.wikimedia.org/r/1338185 (https://phabricator.wikimedia.org/T437350) (owner: 10Majavah) [12:35:56] !log ladsgroup@deploy1003 tryvix1509, ladsgroup: Backport for [[gerrit:1338151|core-Namespaces.php: Disallow indexing on talk namespaces on ukwiki (T437409)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:35:57] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12302855 (10Geertivp) There is no e-mail forward for my account. Originally I was registered as an admin with my e-mail address geert@ and I was on the `wikimediabe-l-owner@lists.wikimedia.org`... [12:36:02] (03CR) 10Muehlenhoff: [C:03+2] Setup a syncrepl cluster on trixie/MDB [puppet] - 10https://gerrit.wikimedia.org/r/1335825 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [12:36:25] (03PS4) 10Muehlenhoff: Create a separate role openldap::replica_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1337910 (https://phabricator.wikimedia.org/T331699) [12:36:39] Hide_on_rosie: it's live on mwdebug [12:37:02] (03PS2) 10Majavah: P:wmcs::novaproxy: Set maximum connection pool size for backends [puppet] - 10https://gerrit.wikimedia.org/r/1338185 (https://phabricator.wikimedia.org/T437350) [12:37:47] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9391/co" [puppet] - 10https://gerrit.wikimedia.org/r/1338185 (https://phabricator.wikimedia.org/T437350) (owner: 10Majavah) [12:39:57] RESOLVED: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:39:58] (03Merged) 10jenkins-bot: admin_ng: Add tool-server namespace to LiftWing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337869 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [12:40:47] Amir1: Check OK. [12:41:41] !log ladsgroup@deploy1003 tryvix1509, ladsgroup: Continuing with deployment [12:44:04] (03CR) 10Zabe: [C:03+1] "will you copy over the tables from x1 to x4?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338142 (https://phabricator.wikimedia.org/T398709) (owner: 10Ladsgroup) [12:44:48] 06SRE, 10SRE-Access-Requests: Requesting access to production for dkertesz - https://phabricator.wikimedia.org/T437271#12302883 (10dkertesz) 05Open→03Resolved this is all done, thanks @Fabfur ! [12:44:57] (03CR) 10Ladsgroup: "yeah, we have reconcileTables." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338142 (https://phabricator.wikimedia.org/T398709) (owner: 10Ladsgroup) [12:46:19] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338151|core-Namespaces.php: Disallow indexing on talk namespaces on ukwiki (T437409)]] (duration: 14m 39s) [12:46:22] T437409: Disallow indexing on talk namespaces on ukwiki - https://phabricator.wikimedia.org/T437409 [12:46:53] !log btullis@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [12:48:21] !log btullis@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [12:49:38] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS trixie [12:52:11] (03PS4) 10Giuseppe Lavagetto: profile::containerd: add gVisor support [puppet] - 10https://gerrit.wikimedia.org/r/1330129 (https://phabricator.wikimedia.org/T435796) [12:52:12] (03PS4) 10Giuseppe Lavagetto: containerd: use hiera parameters to detect if dragonfly is enabled [puppet] - 10https://gerrit.wikimedia.org/r/1330130 [12:52:12] (03PS4) 10Giuseppe Lavagetto: kubernetes::staging: make containerd support gVisor runtime in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1330131 (https://phabricator.wikimedia.org/T435796) [12:52:12] (03PS4) 10Giuseppe Lavagetto: profile::kubernetes::node: Add gvisor labels [puppet] - 10https://gerrit.wikimedia.org/r/1330187 (https://phabricator.wikimedia.org/T436212) [12:52:13] (03PS1) 10Giuseppe Lavagetto: kubernetes::staging: make containerd support gVisor runtime [puppet] - 10https://gerrit.wikimedia.org/r/1338189 [12:55:53] 06SRE, 06Traffic: Migrated busted container image to Trixie - https://phabricator.wikimedia.org/T437442 (10MoritzMuehlenhoff) 03NEW [12:56:36] PROBLEM - LDAP -writable server- on ldap-rw1001 is CRITICAL: Could not search/find objectclasses in dc=wikimedia,dc=org https://wikitech.wikimedia.org/wiki/LDAP%23Troubleshooting [12:59:20] ^ ldap-rw1001 is expected, WIP,I'll down time it some more [13:00:05] urbanecm and TheresNoTime: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for UTC afternoon backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1300). [13:00:05] stephanebisson and Hide_on_rosie: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:10] o/ [13:00:34] I'm ready to start with my patches if there's no objections [13:01:23] stephanebisson: are you deploying yourself? If so, go ahead :) [13:01:41] TheresNoTime yea [13:01:49] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbisson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334945 (https://phabricator.wikimedia.org/T434487) (owner: 10Sbisson) [13:01:49] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbisson@deploy1003 using scap backport" [extensions/ArticleGuidance] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1337983 (https://phabricator.wikimedia.org/T434487) (owner: 10Sbisson) [13:02:16] !log jmm@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on ldap-rw[1001,2001].wikimedia.org with reason: work in progress [13:02:48] (03Merged) 10jenkins-bot: ArticleGuidance: Add the redirect configuration keys [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334945 (https://phabricator.wikimedia.org/T434487) (owner: 10Sbisson) [13:04:34] (03Merged) 10jenkins-bot: Replace experiment with instrument and config-driven redirect [extensions/ArticleGuidance] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1337983 (https://phabricator.wikimedia.org/T434487) (owner: 10Sbisson) [13:04:46] (03CR) 10Volans: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1338185 (https://phabricator.wikimedia.org/T437350) (owner: 10Majavah) [13:04:55] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:57] !log sbisson@deploy1003 Started scap sync-world: Backport for [[gerrit:1334945|ArticleGuidance: Add the redirect configuration keys (T434487)]], [[gerrit:1337983|Replace experiment with instrument and config-driven redirect (T434487)]] [13:05:00] T434487: Stop AG experiment [tr, en.simple, fr] and switch to enabled by default for junior editors in tr and en.simple - https://phabricator.wikimedia.org/T434487 [13:05:02] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [13:05:30] (03CR) 10Majavah: [V:03+1 C:03+2] P:wmcs::novaproxy: Set maximum connection pool size for backends [puppet] - 10https://gerrit.wikimedia.org/r/1338185 (https://phabricator.wikimedia.org/T437350) (owner: 10Majavah) [13:05:44] (03PS1) 10CDanis: NodeTextfileStale: exclude geodns-conftool-state [alerts] - 10https://gerrit.wikimedia.org/r/1338192 [13:06:31] (03PS1) 10Michael Große: testwiki: allow testing new AccountSetup experiment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) [13:09:13] !log sbisson@deploy1003 sbisson: Backport for [[gerrit:1334945|ArticleGuidance: Add the redirect configuration keys (T434487)]], [[gerrit:1337983|Replace experiment with instrument and config-driven redirect (T434487)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:10:13] !log sbisson@deploy1003 sbisson: Continuing with deployment [13:10:26] (03CR) 10Ssingh: [C:03+1] "Thanks!" [alerts] - 10https://gerrit.wikimedia.org/r/1338192 (owner: 10CDanis) [13:10:42] (03CR) 10CDanis: [C:03+2] NodeTextfileStale: exclude geodns-conftool-state [alerts] - 10https://gerrit.wikimedia.org/r/1338192 (owner: 10CDanis) [13:11:23] !log sukhe@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie [13:11:30] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12302967 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host sretest2013.codfw.wmnet with OS trixie [13:13:27] (03Merged) 10jenkins-bot: NodeTextfileStale: exclude geodns-conftool-state [alerts] - 10https://gerrit.wikimedia.org/r/1338192 (owner: 10CDanis) [13:14:35] !log elukey@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS trixie [13:15:12] !log sbisson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1334945|ArticleGuidance: Add the redirect configuration keys (T434487)]], [[gerrit:1337983|Replace experiment with instrument and config-driven redirect (T434487)]] (duration: 10m 15s) [13:15:15] T434487: Stop AG experiment [tr, en.simple, fr] and switch to enabled by default for junior editors in tr and en.simple - https://phabricator.wikimedia.org/T434487 [13:17:11] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a4-eqiad [13:17:45] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a4-eqiad [13:18:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.14% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:18:26] (03CR) 10Herron: [C:03+1] Remove dispatch images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1338145 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [13:19:12] (03CR) 10Herron: [C:03+1] icinga: remove dead ElasticSearch checks breaking CI [puppet] - 10https://gerrit.wikimedia.org/r/1338157 (owner: 10Hnowlan) [13:19:54] (03CR) 10Blake: [C:03+2] memcached: Clean up pki migration switch. [puppet] - 10https://gerrit.wikimedia.org/r/1337942 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:20:13] (03CR) 10Herron: [C:03+1] icinga: migrate cert check to prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1333749 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [13:20:19] (03PS1) 10Bking: Revert "matomo: Allow dse-k8s pods to access database" [puppet] - 10https://gerrit.wikimedia.org/r/1338196 [13:20:25] (03CR) 10Bking: [V:03+2 C:03+2] Revert "matomo: Allow dse-k8s pods to access database" [puppet] - 10https://gerrit.wikimedia.org/r/1338196 (owner: 10Bking) [13:20:45] (03CR) 10Herron: [C:03+1] kubernetes: Switch to GitLab origin for LibreNMS [puppet] - 10https://gerrit.wikimedia.org/r/1338078 (https://phabricator.wikimedia.org/T436578) (owner: 10Andrea Denisse) [13:20:57] (03CR) 10CI reject: [V:04-1] Revert "matomo: Allow dse-k8s pods to access database" [puppet] - 10https://gerrit.wikimedia.org/r/1338196 (owner: 10Bking) [13:21:08] (03Abandoned) 10Bking: Revert "matomo: Allow dse-k8s pods to access database" [puppet] - 10https://gerrit.wikimedia.org/r/1338196 (owner: 10Bking) [13:21:35] (03PS1) 10Bking: Revert^2 "matomo: Allow dse-k8s pods to access database" [puppet] - 10https://gerrit.wikimedia.org/r/1338197 [13:21:46] (03CR) 10Bking: [V:03+2 C:03+2] Revert^2 "matomo: Allow dse-k8s pods to access database" [puppet] - 10https://gerrit.wikimedia.org/r/1338197 (owner: 10Bking) [13:21:58] (03CR) 10Btullis: [C:03+2] ceph: Install the CephX verify script only where it works [puppet] - 10https://gerrit.wikimedia.org/r/1338130 (https://phabricator.wikimedia.org/T437233) (owner: 10Btullis) [13:22:09] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [13:22:11] (03PS1) 10Majavah: P:lvs::realserver::ipip: Fix clsact config on networkd hosts [puppet] - 10https://gerrit.wikimedia.org/r/1338198 [13:22:19] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [13:22:32] (03CR) 10Herron: [C:03+1] aptrepo: Add grafana-plugins to main apt repo [puppet] - 10https://gerrit.wikimedia.org/r/1338064 (https://phabricator.wikimedia.org/T436056) (owner: 10Andrea Denisse) [13:23:52] (03CR) 10JMeybohm: [V:03+1] "PCC SUCCESS (CORE_DIFF 13): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9392/c" [puppet] - 10https://gerrit.wikimedia.org/r/1330130 (owner: 10Giuseppe Lavagetto) [13:24:50] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1338064 (https://phabricator.wikimedia.org/T436056) (owner: 10Andrea Denisse) [13:24:52] !log sukhe@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage [13:25:43] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b7-eqiad [13:26:08] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b7-eqiad [13:26:48] (03CR) 10JMeybohm: kubernetes::staging: make containerd support gVisor runtime (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1338189 (owner: 10Giuseppe Lavagetto) [13:27:21] (03CR) 10JMeybohm: [V:03+1] "PCC SUCCESS (CORE_DIFF 13): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9393/c" [puppet] - 10https://gerrit.wikimedia.org/r/1330129 (https://phabricator.wikimedia.org/T435796) (owner: 10Giuseppe Lavagetto) [13:28:00] PROBLEM - Backup freshness on backup1014 is CRITICAL: Stale: 2 (ldap-rw1001, ...), Fresh: 141 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [13:28:11] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove dispatch images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1338145 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [13:28:29] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage [13:29:18] !log pruned obsolete Bullseye image dispatch from the docker registry T416452 [13:29:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:29:21] T416452: Migrate Docker images running in Production away from Bullseye - https://phabricator.wikimedia.org/T416452 [13:29:44] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] "dispatch has been pruned from the Docker registry" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1338145 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [13:33:11] (03CR) 10Herron: [C:03+1] "Patch lgtm, could you please add a bit of context in the patch and/or task about the plugin? Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1338056 (https://phabricator.wikimedia.org/T436075) (owner: 10Andrea Denisse) [13:35:02] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [13:36:20] (03CR) 10JMeybohm: [V:03+1] "PCC SUCCESS (): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9396/console" [puppet] - 10https://gerrit.wikimedia.org/r/1330131 (https://phabricator.wikimedia.org/T435796) (owner: 10Giuseppe Lavagetto) [13:38:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.87% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:40:28] (03CR) 10JMeybohm: [V:03+1 C:04-1] profile::containerd: add gVisor support (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1330129 (https://phabricator.wikimedia.org/T435796) (owner: 10Giuseppe Lavagetto) [13:42:37] !log btullis@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [13:43:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.7% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:43:23] !log btullis@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [13:44:57] (03CR) 10JMeybohm: [V:03+1 C:03+1] "You could pull this out of the relation chain or as first patch in chain so it can be merged independently of the gVisor changes." [puppet] - 10https://gerrit.wikimedia.org/r/1330130 (owner: 10Giuseppe Lavagetto) [13:46:18] (03PS1) 10Ladsgroup: memcached: Set connect_timeout and timeout [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1338201 (https://phabricator.wikimedia.org/T437398) [13:47:40] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 100% [13:48:22] RECOVERY - Host titan1002 is UP: PING WARNING - Packet loss = 0%, RTA = 944.40 ms [13:48:57] FIRING: [3x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:50:16] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - thanos-query_443: Servers titan1001.eqiad.wmnet are marked down but pooled: thanos-web_443: Servers titan1001.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [13:50:24] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-08-26-163757 to 2026-09-08-175644 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338203 (https://phabricator.wikimedia.org/T420747) [13:50:27] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-09-03-172537 to 2026-09-08-173612 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338204 (https://phabricator.wikimedia.org/T300170) [13:50:34] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 100% [13:50:46] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [13:50:57] FIRING: ProbeDown: Service thanos-query:443 has failed probes (http_thanos-query_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#thanos-query:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:51:13] !ack [13:51:14] 8333 (ACKED) ProbeDown sre (10.2.2.53 ip4 thanos-query:443 probes/service http_thanos-query_ip4 eqiad) [13:51:16] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [13:51:29] tappof: is that expected? [13:51:43] no herron, I'm checking [13:51:48] kk [13:52:11] !incidents [13:52:11] 8333 (ACKED) ProbeDown sre (10.2.2.53 ip4 thanos-query:443 probes/service http_thanos-query_ip4 eqiad) [13:52:11] 8331 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr2-eqiad:9804 Peering: Equinix (3198427-A Wikimedia-DC5-IX-01, MAC filter) xe-3/3/3 gnmi eqiad) [13:52:12] 8332 (RESOLVED) [2x] CoreRouterInterfaceDropPercent network sre (cr1-eqiad:9804 eqiad) [13:52:12] 8328 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [13:52:35] (03PS1) 10Ssingh: hiera: acme_chief: add sretest2013 to wikipedia25 [puppet] - 10https://gerrit.wikimedia.org/r/1338205 (https://phabricator.wikimedia.org/T436691) [13:53:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:53:33] !log btullis@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [13:53:43] (03CR) 10Ssingh: [C:03+2] hiera: acme_chief: add sretest2013 to wikipedia25 [puppet] - 10https://gerrit.wikimedia.org/r/1338205 (https://phabricator.wikimedia.org/T436691) (owner: 10Ssingh) [13:53:57] RESOLVED: [3x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:55:04] !log btullis@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [13:55:23] (03CR) 10Btullis: [C:03+2] ml-services: Add tool-server deployment configs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337879 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [13:55:57] RESOLVED: ProbeDown: Service thanos-query:443 has failed probes (http_thanos-query_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#thanos-query:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:56:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:57:42] (03Merged) 10jenkins-bot: ml-services: Add tool-server deployment configs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337879 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [13:57:58] (03CR) 10Clément Goubert: [C:03+1] mediawiki::tools: mediawiki-cache-warmup: Remove mobileServer support [puppet] - 10https://gerrit.wikimedia.org/r/1337933 (owner: 10Majavah) [13:58:41] (03CR) 10Hnowlan: [C:03+1] memcached: Set connect_timeout and timeout [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1338201 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup) [14:00:02] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [14:00:04] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1400) [14:00:41] (03CR) 10Ladsgroup: [C:03+2] memcached: Set connect_timeout and timeout [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1338201 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup) [14:03:34] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade evaluators from 2026-08-26-163757 to 2026-09-08-175644 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338203 (https://phabricator.wikimedia.org/T420747) (owner: 10Jforrester) [14:03:49] (03Merged) 10jenkins-bot: memcached: Set connect_timeout and timeout [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1338201 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup) [14:06:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:06:24] !log sukhe@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1003" [14:06:24] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-08-26-163757 to 2026-09-08-175644 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338203 (https://phabricator.wikimedia.org/T420747) (owner: 10Jforrester) [14:07:13] !log sukhe@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1003" [14:07:14] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie [14:07:16] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:07:30] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12303353 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host sretest2013.codfw.wmnet with OS trixie completed: - sr... [14:08:04] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:08:21] !log jforrester@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:10:02] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [14:10:20] !log jforrester@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:10:30] !log jforrester@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:10:42] (03CR) 10Jforrester: [C:03+2] wikifunctions: Add 2 callback tests to check-wf-services.py [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334186 (https://phabricator.wikimedia.org/T421848) (owner: 10David Martin) [14:12:17] !log jforrester@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:13:16] (03Merged) 10jenkins-bot: wikifunctions: Add 2 callback tests to check-wf-services.py [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334186 (https://phabricator.wikimedia.org/T421848) (owner: 10David Martin) [14:14:29] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-09-03-172537 to 2026-09-08-173612 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338204 (https://phabricator.wikimedia.org/T300170) (owner: 10Jforrester) [14:15:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 0% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:15:44] (03PS1) 10Ladsgroup: thumbor: Add memcached timeout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338208 (https://phabricator.wikimedia.org/T437398) [14:15:57] FIRING: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:16:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 0% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:16:32] !ack [14:16:32] 8334 (ACKED) PHPFPMTooBusy sre (mw-web main eqiad) [14:16:35] Oh great. [14:16:54] (03CR) 10Andrew Bogott: [C:03+2] Add cloudvps-project-usage.py [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [14:17:00] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-09-03-172537 to 2026-09-08-173612 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338204 (https://phabricator.wikimedia.org/T300170) (owner: 10Jforrester) [14:17:30] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:17:46] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:18:15] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 2.5s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [14:19:00] !log jforrester@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:19:40] !log jforrester@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:19:47] !log jforrester@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:20:23] !log jforrester@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:20:57] RESOLVED: [3x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:21:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 4.247% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:21:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 3.375% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:21:51] !ack [14:21:52] 8335 (ACKED) PHPFPMTooBusy sre (mw-web main eqiad) [14:23:15] FIRING: [2x] MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-int releases routed via main (k8s) 803.7ms - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [14:23:47] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1337942 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [14:24:07] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12303470 (10elukey) I tried to reimage cloudcephosd1055 and it failed in d-i executing `/tmp/partman_early_command`. Tried to exec it... [14:25:15] FIRING: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 24.53% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:26:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 3.375% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:28:15] RESOLVED: [2x] MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-int releases routed via main (k8s) 1.053s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [14:29:45] FIRING: [2x] MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-int releases routed via main (k8s) 804.9ms - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [14:30:04] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1400) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1430) [14:31:06] RESOLVED: [2x] GnmiInterfaceCountersDrop: lsw1-a7-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:32:35] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tool-server' for release 'main' . [14:34:06] (03PS1) 10Andrew Bogott: profile::memcached::instance: Remove use of a bunch of defunct ssh vars [puppet] - 10https://gerrit.wikimedia.org/r/1338212 (https://phabricator.wikimedia.org/T353511) [14:34:19] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply [14:34:45] RESOLVED: [2x] MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-int releases routed via main (k8s) 1.026s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [14:35:02] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply [14:37:12] (03Abandoned) 10Andrew Bogott: profile::memcached::instance: Remove use of a bunch of defunct ssh vars [puppet] - 10https://gerrit.wikimedia.org/r/1338212 (https://phabricator.wikimedia.org/T353511) (owner: 10Andrew Bogott) [14:38:36] (03PS1) 10Ladsgroup: Disable parsoidCachePrewarm jobs everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338213 (https://phabricator.wikimedia.org/T436001) [14:40:15] FIRING: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-jobrunner releases routed via main at eqiad: 0% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:40:52] FIRING: [5x] GnmiInterfaceCountersDrop: lsw1-a7-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:41:48] (03PS2) 10Blake: memcached: set an undef default if we're not using tls. [puppet] - 10https://gerrit.wikimedia.org/r/1338215 (https://phabricator.wikimedia.org/T353511) [14:42:21] !log half concurrency for parsoidCachePrewarm RecordLintJob and refreshLinks in jobqueue, eqiad & codfw [14:42:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:43:20] PROBLEM - Host an-worker1198 is DOWN: PING CRITICAL - Packet loss = 100% [14:44:52] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:xe-1/0/1:0 (Transport: cr2-eqord:xe-0/1/5 (Arelion, IC-314533 24ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:45:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:xe-1/0/1:0 (Transport: cr2-eqord:xe-0/1/5 (Arelion, IC-314533 24ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:45:32] (03CR) 10Clément Goubert: [C:03+1] memcached: set an undef default if we're not using tls. [puppet] - 10https://gerrit.wikimedia.org/r/1338215 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [14:45:40] (03CR) 10Majavah: [C:03+1] memcached: set an undef default if we're not using tls. [puppet] - 10https://gerrit.wikimedia.org/r/1338215 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [14:45:52] RESOLVED: [5x] GnmiInterfaceCountersDrop: lsw1-a7-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:46:33] (03CR) 10Blake: [C:03+2] memcached: set an undef default if we're not using tls. [puppet] - 10https://gerrit.wikimedia.org/r/1338215 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [14:48:33] (03CR) 10Ssingh: "@mmuhlenhoff@wikimedia.org: given https://wikitech.wikimedia.org/wiki/Url-downloader#Troubleshooting exists, are we OK merging this patch " [puppet] - 10https://gerrit.wikimedia.org/r/1328218 (https://phabricator.wikimedia.org/T435648) (owner: 10Ssingh) [14:49:52] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:xe-1/0/1:0 (Transport: cr2-eqord:xe-0/1/5 (Arelion, IC-314533 24ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:51:20] (03PS3) 10Elukey: docker_registry: remove the Swift support [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) [14:51:20] (03PS1) 10Elukey: Move docker::registry to toolforge::registry [puppet] - 10https://gerrit.wikimedia.org/r/1338216 [14:52:36] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12303703 (10Andrew) a:05Jclark-ctr→03fgiunchedi That's new to me -- looks like the most recent changes to that script are from @B... [14:52:54] (03PS2) 10Elukey: Move docker::registry to toolforge::registry [puppet] - 10https://gerrit.wikimedia.org/r/1338216 [14:52:55] (03PS4) 10Elukey: docker_registry: remove the Swift support [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) [14:55:46] (03CR) 10Muehlenhoff: [C:03+1] "Sounds good to me" [puppet] - 10https://gerrit.wikimedia.org/r/1328218 (https://phabricator.wikimedia.org/T435648) (owner: 10Ssingh) [14:56:23] (03CR) 10Muehlenhoff: [C:03+1] "Sounds good to me. I'm not sure if there's a wider process to make something paging, but from my PoV this is fine to page." [puppet] - 10https://gerrit.wikimedia.org/r/1328218 (https://phabricator.wikimedia.org/T435648) (owner: 10Ssingh) [14:57:09] (03PS3) 10Elukey: Move docker::registry to toolforge::registry [puppet] - 10https://gerrit.wikimedia.org/r/1338216 [14:57:10] (03PS5) 10Elukey: docker_registry: remove the Swift support [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) [14:57:35] (03CR) 10Ssingh: "Thanks. I am not sure about that either but I plan to at least let sre-at-large@ know. I will check with Hugh as well. @hnowlan@wikimedia." [puppet] - 10https://gerrit.wikimedia.org/r/1328218 (https://phabricator.wikimedia.org/T435648) (owner: 10Ssingh) [14:58:42] (03CR) 10Subramanya Sastry: [C:03+1] "This only affects wikis where Parsoid is not used for read views ... and on those wikis, it just means, that the first Parsoid-enabled per" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338213 (https://phabricator.wikimedia.org/T436001) (owner: 10Ladsgroup) [15:00:53] (03PS6) 10Elukey: docker_registry: remove the Swift support [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) [15:00:54] (03PS1) 10Elukey: Rename docker::registry hiera key to docker_registry_endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1338224 [15:01:07] (03CR) 10Elukey: "Thanks to Riccardo: https://puppet-compiler.wmflabs.org/output/1338216/9398/docker-registry-01.cloudinfra.eqiad1.wikimedia.cloud/index.htm" [puppet] - 10https://gerrit.wikimedia.org/r/1338216 (owner: 10Elukey) [15:01:45] !log hnowlan@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply [15:04:43] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=cp6008.drmrs.wmnet [15:05:15] FIRING: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-jobrunner releases routed via main at eqiad: 0% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:06:35] !log installing grub2 bugfix updates from Bookworm point release [15:06:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:07:34] !log hnowlan@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply [15:07:54] !log hnowlan@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply [15:08:03] !log hnowlan@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply [15:10:44] FIRING: KubernetesDeploymentUnavailableReplicas: ... [15:10:44] Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ... [15:10:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [15:15:20] (03CR) 10Elukey: [C:03+1] Create a separate role openldap::replica_mdb [puppet] - 10https://gerrit.wikimedia.org/r/1337910 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [15:15:52] RECOVERY - Host an-worker1198 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [15:17:44] !log Delete all running periodic jobs starting with `growthexperiments-refreshlinkrecommendations-*` (to pick up new configuration; T392944) [15:17:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:17:48] T392944: Enable the iterative way of refreshing LinkRecommendations for all wikis - https://phabricator.wikimedia.org/T392944 [15:19:21] (03CR) 10BCornwall: [C:03+2] P:lvs::realserver::ipip: Fix clsact config on networkd hosts [puppet] - 10https://gerrit.wikimedia.org/r/1338198 (owner: 10Majavah) [15:20:44] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [15:20:44] Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ... [15:20:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [15:23:22] FIRING: GnmiInterfaceCountersDrop: ... [15:23:22] fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=fasw1-f5a-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:28:22] FIRING: [2x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:30:03] (03PS4) 10Gmodena: rest-gateway: front wdqs v2 endpoints in shadow mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333815 (https://phabricator.wikimedia.org/T436778) [15:33:44] (03PS2) 10Ladsgroup: Disable parsoidCachePrewarm jobs everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338213 (https://phabricator.wikimedia.org/T436001) [15:34:04] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338213 (https://phabricator.wikimedia.org/T436001) (owner: 10Ladsgroup) [15:34:24] (03PS5) 10Gmodena: rest-gateway: front wdqs v2 endpoints in shadow mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333815 (https://phabricator.wikimedia.org/T436778) [15:34:40] (03CR) 10Gmodena: rest-gateway: front wdqs v2 endpoints in shadow mode (035 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333815 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [15:35:55] (03Merged) 10jenkins-bot: Disable parsoidCachePrewarm jobs everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338213 (https://phabricator.wikimedia.org/T436001) (owner: 10Ladsgroup) [15:36:19] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1338213|Disable parsoidCachePrewarm jobs everywhere (T436001 T436206)]] [15:36:24] T436001: Dolly Parton High load - https://phabricator.wikimedia.org/T436001 [15:36:24] T436206: ParsoidCachePrewarmJob runs without PoolCounter and even on Parsoid parses - https://phabricator.wikimedia.org/T436206 [15:37:26] (03PS1) 10Elukey: Add pki svc IP A and PTR records [dns] - 10https://gerrit.wikimedia.org/r/1338234 (https://phabricator.wikimedia.org/T436809) [15:37:44] (03PS1) 10Elukey: Assign a role to pki[12]003 [puppet] - 10https://gerrit.wikimedia.org/r/1338235 (https://phabricator.wikimedia.org/T436809) [15:37:46] (03PS1) 10Elukey: Add conftool-data for the pki[12]003 VMs [puppet] - 10https://gerrit.wikimedia.org/r/1338236 (https://phabricator.wikimedia.org/T436809) [15:39:23] (03PS2) 10Elukey: Add conftool-data for the pki[12]003 VMs [puppet] - 10https://gerrit.wikimedia.org/r/1338236 (https://phabricator.wikimedia.org/T436809) [15:40:37] (03PS1) 10Majavah: P:memcached::instance: Cleanup deleted options [puppet] - 10https://gerrit.wikimedia.org/r/1338238 [15:40:52] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1338213|Disable parsoidCachePrewarm jobs everywhere (T436001 T436206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:41:03] (03CR) 10Elukey: "Records already allocated in netbox!" [dns] - 10https://gerrit.wikimedia.org/r/1338234 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:41:27] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [15:46:02] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338213|Disable parsoidCachePrewarm jobs everywhere (T436001 T436206)]] (duration: 09m 43s) [15:46:07] T436001: Dolly Parton High load - https://phabricator.wikimedia.org/T436001 [15:46:07] T436206: ParsoidCachePrewarmJob runs without PoolCounter and even on Parsoid parses - https://phabricator.wikimedia.org/T436206 [15:54:35] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [15:55:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.04% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:55:46] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [16:10:46] (03CR) 10JMeybohm: profile::kubernetes::node: Add gvisor labels (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1330187 (https://phabricator.wikimedia.org/T436212) (owner: 10Giuseppe Lavagetto) [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:57] (03CR) 10JMeybohm: [C:04-1] "You'll need to bump the chart version" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333139 (https://phabricator.wikimedia.org/T436649) (owner: 10Giuseppe Lavagetto) [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:19:23] (03PS2) 10Michael Große: testwiki: allow testing new AccountSetup experiment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) [16:22:22] (03PS1) 10Pppery: Update translations [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1338245 [16:25:16] !log cgoubert@deploy1003 helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [16:26:28] (03PS1) 10Elukey: profile::docker_registry: improve the blackbox exporter's config [puppet] - 10https://gerrit.wikimedia.org/r/1338246 [16:26:35] !log cgoubert@deploy1003 helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [16:26:59] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [16:27:06] (03CR) 10CI reject: [V:04-1] profile::docker_registry: improve the blackbox exporter's config [puppet] - 10https://gerrit.wikimedia.org/r/1338246 (owner: 10Elukey) [16:27:39] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [16:27:59] !log cgoubert@deploy1003 helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [16:28:05] (03PS2) 10Elukey: profile::docker_registry: improve the blackbox exporter's config [puppet] - 10https://gerrit.wikimedia.org/r/1338246 [16:28:17] (03CR) 10FNegri: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1338216 (owner: 10Elukey) [16:28:40] !log cgoubert@deploy1003 helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [16:29:39] jouncebot: nowandnext [16:29:39] No deployments scheduled for the next 0 hour(s) and 30 minute(s) [16:29:39] In 0 hour(s) and 30 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1700) [16:29:53] Amir1: i noticed the prewarm drama, ok to deploy a patch? :)) [16:30:09] (03CR) 10Elukey: [C:03+1] http boot: don't log error message [software/spicerack] - 10https://gerrit.wikimedia.org/r/1336116 (owner: 10JHathaway) [16:30:13] go for it [16:30:43] (03CR) 10Urbanecm: testwiki: allow testing new AccountSetup experiment (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [16:31:12] (03Abandoned) 10FNegri: icinga_exporter: don't route wikitech-static to wmcs [puppet] - 10https://gerrit.wikimedia.org/r/1117575 (https://phabricator.wikimedia.org/T376400) (owner: 10FNegri) [16:31:17] (03CR) 10JHathaway: [C:03+2] http boot: don't log error message [software/spicerack] - 10https://gerrit.wikimedia.org/r/1336116 (owner: 10JHathaway) [16:31:24] (03CR) 10Michael Große: testwiki: allow testing new AccountSetup experiment (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [16:31:43] (03PS3) 10Urbanecm: testwiki: allow testing new AccountSetup experiment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [16:32:21] (03CR) 10Michael Große: testwiki: allow testing new AccountSetup experiment (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [16:33:02] (03CR) 10Urbanecm: testwiki: allow testing new AccountSetup experiment (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [16:34:52] (03PS4) 10Urbanecm: testwiki: allow testing new AccountSetup experiment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [16:35:10] (03CR) 10Urbanecm: testwiki: allow testing new AccountSetup experiment (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [16:35:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [16:36:29] (03Merged) 10jenkins-bot: testwiki: allow testing new AccountSetup experiment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338173 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [16:36:47] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1338173|testwiki: allow testing new AccountSetup experiment (T436872)]] [16:36:50] T436872: Account Setup experiment: test end-to-end flow and verify translations on pilot wikis (DE1.3.1) - https://phabricator.wikimedia.org/T436872 [16:39:04] (03CR) 10SD0001: nlwiki: enable SecurePoll local elections (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) (owner: 10Novem Linguae) [16:41:07] !log urbanecm@deploy1003 migr, urbanecm: Backport for [[gerrit:1338173|testwiki: allow testing new AccountSetup experiment (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:41:44] !log urbanecm@deploy1003 migr, urbanecm: Continuing with deployment [16:46:15] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338173|testwiki: allow testing new AccountSetup experiment (T436872)]] (duration: 09m 28s) [16:46:19] T436872: Account Setup experiment: test end-to-end flow and verify translations on pilot wikis (DE1.3.1) - https://phabricator.wikimedia.org/T436872 [16:51:22] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, 06tools-infrastructure-team: cloudvirt1067 missing hwraid BBU - https://phabricator.wikimedia.org/T437221#12304345 (10VRiley-WMF) 05Open→03In progress Since these were moved, I'm going to check the cable. If it's not that, I may have to see if I can search f... [16:53:35] 10ops-drmrs, 06DC-Ops, 06Traffic: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12304366 (10BCornwall) 05Open→03Resolved Thanks for hanging in there, everyone, and thanks @RobH for taking care of the arduous task of getting the RAM over! [16:56:15] (03PS2) 10Blake: mediawiki: stop using lamp.deployment in cronjob. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338254 (https://phabricator.wikimedia.org/T417800) [16:57:11] (03PS8) 10Blake: mediawiki: stop using lamp.deployment in job. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338227 (https://phabricator.wikimedia.org/T417800) [17:00:04] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1700) [17:03:57] (03PS1) 10Andrew Bogott: cloudceph: enable osd_memory_target_autotune [puppet] - 10https://gerrit.wikimedia.org/r/1338257 (https://phabricator.wikimedia.org/T429387) [17:04:30] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338257 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [17:04:35] (03CR) 10CI reject: [V:04-1] cloudceph: enable osd_memory_target_autotune [puppet] - 10https://gerrit.wikimedia.org/r/1338257 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [17:04:50] !log eevans@cumin1003 START - Cookbook sre.hosts.provision for host aqs1026.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [17:07:04] PROBLEM - Host aqs1026 is DOWN: PING CRITICAL - Packet loss = 100% [17:08:04] (03CR) 10BCornwall: P:tofurkey enable Tofurkey for MAGRU (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [17:08:07] FIRING: [2x] ProbeDown: Service aqs1026-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:10:58] 06SRE, 10LDAP-Access-Requests: Grant Access to <500> for - https://phabricator.wikimedia.org/T437488 (10MMilenkovic-WMF) 03NEW [17:12:19] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs1026.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [17:12:56] RECOVERY - Host aqs1026 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [17:13:07] FIRING: [4x] ProbeDown: Service aqs1026-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:18:35] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12304465 (10Ladsgroup) We haven't done any migration in the past six years. Is the correct way older than six years or it was more recent? [17:19:15] (03CR) 10Ladsgroup: [C:03+2] thumbor: Add memcached timeout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338208 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup) [17:20:06] (03PS2) 10Andrew Bogott: cloudceph: enable osd_memory_target_autotune [puppet] - 10https://gerrit.wikimedia.org/r/1338257 (https://phabricator.wikimedia.org/T429387) [17:21:41] (03Merged) 10jenkins-bot: thumbor: Add memcached timeout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338208 (https://phabricator.wikimedia.org/T437398) (owner: 10Ladsgroup) [17:22:21] (03CR) 10CI reject: [V:04-1] cloudceph: enable osd_memory_target_autotune [puppet] - 10https://gerrit.wikimedia.org/r/1338257 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [17:23:05] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [17:23:07] RESOLVED: [4x] ProbeDown: Service aqs1026-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:23:14] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [17:26:01] (03PS3) 10Andrew Bogott: cloudceph: enable osd_memory_target_autotune [puppet] - 10https://gerrit.wikimedia.org/r/1338257 (https://phabricator.wikimedia.org/T429387) [17:27:39] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [17:29:06] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, 06tools-infrastructure-team: cloudvirt1067 missing hwraid BBU - https://phabricator.wikimedia.org/T437221#12304497 (10VRiley-WMF) 05In progress→03Resolved I unplugged the server, and removed the battery and did a flea power drain on the unit. Plugged the... [17:29:12] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338257 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [17:29:44] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [17:29:47] (03PS1) 10Tiziano Fogli: mw-jobrunner/values: tune replica count [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338261 (https://phabricator.wikimedia.org/T437466) [17:29:52] (03PS1) 10Tiziano Fogli: changeprop-jobqueue: exclude parsoidCachePrewarm job [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338260 (https://phabricator.wikimedia.org/T437466) [17:31:24] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host aqs1026.eqiad.wmnet with OS bookworm [17:32:21] (03CR) 10Clément Goubert: [C:03+1] changeprop-jobqueue: exclude parsoidCachePrewarm job [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338260 (https://phabricator.wikimedia.org/T437466) (owner: 10Tiziano Fogli) [17:33:43] (03CR) 10Clément Goubert: [C:03+1] changeprop-jobqueue: exclude parsoidCachePrewarm job (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338260 (https://phabricator.wikimedia.org/T437466) (owner: 10Tiziano Fogli) [17:33:48] (03CR) 10Clément Goubert: [C:03+1] mw-jobrunner/values: tune replica count [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338261 (https://phabricator.wikimedia.org/T437466) (owner: 10Tiziano Fogli) [17:34:10] (03CR) 10Hnowlan: mw-jobrunner/values: tune replica count (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338261 (https://phabricator.wikimedia.org/T437466) (owner: 10Tiziano Fogli) [17:35:25] FIRING: [3x] ProbeDown: Service aqs1026-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:35:54] (03CR) 10Hnowlan: [C:03+1] changeprop-jobqueue: exclude parsoidCachePrewarm job (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338260 (https://phabricator.wikimedia.org/T437466) (owner: 10Tiziano Fogli) [17:35:55] !log eevans@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host aqs1026.eqiad.wmnet with OS bookworm [17:36:09] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host aqs1026.eqiad.wmnet with OS bookworm [17:37:10] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338264 [17:37:26] (03CR) 10Gergő Tisza: "Yeah, MWP owns ObjectCache which includes the main stash (or at least the MediaWiki part of it) so sending us alerts makes sense. Thanks f" [puppet] - 10https://gerrit.wikimedia.org/r/1334836 (https://phabricator.wikimedia.org/T430940) (owner: 10Jforrester) [17:37:55] (03PS2) 10Tiziano Fogli: changeprop-jobqueue: exclude parsoidCachePrewarm job [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338260 (https://phabricator.wikimedia.org/T437466) [17:37:55] (03PS2) 10Tiziano Fogli: mw-jobrunner/values: tune replica count [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338261 (https://phabricator.wikimedia.org/T437466) [17:38:09] (03CR) 10Tiziano Fogli: mw-jobrunner/values: tune replica count (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338261 (https://phabricator.wikimedia.org/T437466) (owner: 10Tiziano Fogli) [17:39:02] (03CR) 10Tiziano Fogli: changeprop-jobqueue: exclude parsoidCachePrewarm job (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338260 (https://phabricator.wikimedia.org/T437466) (owner: 10Tiziano Fogli) [17:40:25] FIRING: [4x] ProbeDown: Service aqs1026-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:40:28] (03CR) 10RLazarus: [C:03+2] Periodic jobs: Add mainstash_metrics (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1334836 (https://phabricator.wikimedia.org/T430940) (owner: 10Jforrester) [17:45:37] !log eevans@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1026.eqiad.wmnet with reason: host reimage [17:48:40] (03CR) 10Tiziano Fogli: [C:03+2] mw-jobrunner/values: tune replica count [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338261 (https://phabricator.wikimedia.org/T437466) (owner: 10Tiziano Fogli) [17:49:12] (03Merged) 10jenkins-bot: changeprop-jobqueue: exclude parsoidCachePrewarm job [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338260 (https://phabricator.wikimedia.org/T437466) (owner: 10Tiziano Fogli) [17:49:29] !log rzl@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-cron: apply [17:49:36] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply [17:49:40] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1026.eqiad.wmnet with reason: host reimage [17:49:56] !log rzl@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [17:49:58] (03Merged) 10jenkins-bot: mw-jobrunner/values: tune replica count [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338261 (https://phabricator.wikimedia.org/T437466) (owner: 10Tiziano Fogli) [17:50:10] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply [17:50:17] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [17:50:25] FIRING: [4x] ProbeDown: Service aqs1026-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:50:28] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12304554 (10VRiley-WMF) Rebooted the device, commenced a flea power drain and it seems to have come up healthy before oving forward with the provisioning. It seems to have come back up and sh... [17:50:54] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [17:51:11] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [17:51:29] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12304557 (10VRiley-WMF) 05Open→03Resolved [17:51:49] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [17:52:05] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [17:52:17] (03CR) 10RLazarus: [C:03+2] "Deployed:" [puppet] - 10https://gerrit.wikimedia.org/r/1334836 (https://phabricator.wikimedia.org/T430940) (owner: 10Jforrester) [17:52:23] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12304559 (10VRiley-WMF) Starting this now... [17:52:28] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [17:52:31] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12304560 (10VRiley-WMF) 05Open→03In progress [17:53:24] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply [17:53:48] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply [17:54:01] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [17:54:09] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [17:54:34] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [17:55:25] FIRING: [4x] ProbeDown: Service aqs1026-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:55:30] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply [17:56:04] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply [17:56:15] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [17:56:27] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply [17:56:45] (03CR) 10BBlack: P:tofurkey enable Tofurkey for MAGRU (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [17:56:47] (03PS1) 10Dzahn: create wikipedia-ar-arbcom.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) [17:56:55] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply [17:57:04] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply [17:57:43] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply [17:58:07] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply [17:58:41] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply [17:59:25] (03CR) 10Dzahn: "Aware that these get auto-created for standard wikis but this type is not. is that right?" [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn) [18:00:05] dduvall and dancy: I, the Bot under the Fountain, call upon thee, The Deployer, to do MediaWiki train - Utc-7 Version deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T1800). [18:00:25] FIRING: [4x] ProbeDown: Service aqs1026-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:00:33] mutante: Welcome back! [18:00:43] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [18:00:58] dancy: thank you ) [18:01:24] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [18:01:50] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [18:01:55] (03PS1) 10Brouberol: dse-k8s/wdqs-next: increase resource quotas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338268 [18:02:35] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [18:02:43] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply [18:03:07] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply [18:03:12] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [18:03:13] (03CR) 10Dbrant: [C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338264 (owner: 10PipelineBot) [18:03:47] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [18:05:25] FIRING: [4x] ProbeDown: Service aqs1026-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:05:34] (03Merged) 10jenkins-bot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338264 (owner: 10PipelineBot) [18:06:56] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1026.eqiad.wmnet with OS bookworm [18:13:39] (03PS1) 10TrainBranchBot: group1 to 1.47.0-wmf.19 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338270 (https://phabricator.wikimedia.org/T430838) [18:13:42] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dduvall@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338270 (https://phabricator.wikimedia.org/T430838) (owner: 10TrainBranchBot) [18:14:44] (03Merged) 10jenkins-bot: group1 to 1.47.0-wmf.19 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338270 (https://phabricator.wikimedia.org/T430838) (owner: 10TrainBranchBot) [18:15:06] (03CR) 10Brouberol: [C:03+2] dse-k8s/wdqs-next: increase resource quotas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338268 (owner: 10Brouberol) [18:23:50] !log dduvall@deploy1003 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.19 refs T430838 [18:23:54] T430838: 1.47.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T430838 [18:35:34] (03CR) 10Andrea Denisse: [C:03+1] "LGTM!!" [puppet] - 10https://gerrit.wikimedia.org/r/1338157 (owner: 10Hnowlan) [18:36:09] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thanks!!" [puppet] - 10https://gerrit.wikimedia.org/r/1333749 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [18:40:26] (03PS1) 10Jforrester: wikifunctions: Move client fragments to mainstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338273 (https://phabricator.wikimedia.org/T432849) [18:40:28] (03PS1) 10Jforrester: wikifunctions: Move abstract fragments to mainstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338274 (https://phabricator.wikimedia.org/T432849) [18:50:09] (03CR) 10Bking: [C:03+1] dse-k8s/wdqs-next: increase resource quotas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338268 (owner: 10Brouberol) [18:52:08] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12304905 (10VRiley-WMF) E4 u38 cableID: 20220046 port 38 [18:58:55] (03CR) 10Ssingh: P:tofurkey enable Tofurkey for MAGRU (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [18:59:58] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [19:01:12] (03CR) 10Ssingh: P:tofurkey enable Tofurkey for MAGRU (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [19:01:33] (03PS4) 10Atsuko: airflow3: updating airflow-client [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335780 (https://phabricator.wikimedia.org/T423248) [19:02:55] (03CR) 10Bking: [C:03+2] Kerberos: Present krb1004 as first choice to clients [puppet] - 10https://gerrit.wikimedia.org/r/1335159 (https://phabricator.wikimedia.org/T435873) (owner: 10Bking) [19:03:37] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1075] - vriley@cumin1003" [19:04:16] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1075] - vriley@cumin1003" [19:04:16] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [19:05:06] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1075 [19:05:15] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1075 [19:05:45] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1075.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:05:57] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1075.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:06:07] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1075.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:11:21] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team (Radar): apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12304969 (10Scott_French) Checking in, it looks like this is still broken in the same way as yesterday T437069#12297979 (InRel... [19:18:14] (03PS2) 10Novem Linguae: nlwiki: enable SecurePoll local elections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) [19:18:31] (03PS3) 10Scott French: O:deployment_server::kubernetes: Bootstrap 'scap deploy-service' config [puppet] - 10https://gerrit.wikimedia.org/r/1338068 (https://phabricator.wikimedia.org/T412941) [19:19:18] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1075.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:19:48] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1075.eqiad.wmnet with OS trixie [19:20:05] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12304990 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1075.eqiad.wmnet with OS trixie [19:20:18] (03CR) 10Novem Linguae: nlwiki: enable SecurePoll local elections (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) (owner: 10Novem Linguae) [19:24:41] (03CR) 10Ahmon Dancy: [C:03+1] O:deployment_server::kubernetes: Bootstrap 'scap deploy-service' config [puppet] - 10https://gerrit.wikimedia.org/r/1338068 (https://phabricator.wikimedia.org/T412941) (owner: 10Scott French) [19:26:39] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1075.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:28:37] FIRING: [2x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [19:28:51] !log eevans@cumin1003 START - Cookbook sre.hosts.provision for host aqs1027.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [19:30:25] RESOLVED: [4x] ProbeDown: Service aqs1026-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:30:40] (03PS1) 10Aranyap: Revert^2 "Filter out non-http(s) license urls" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338278 [19:30:48] PROBLEM - Host aqs1027 is DOWN: PING CRITICAL - Packet loss = 100% [19:32:19] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team (Radar): apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12305008 (10Scott_French) If I'm understanding the very end of T437069#12290846 correctly, it sounds like the one potential wo... [19:33:40] (03PS1) 10Aranyap: Revert^2 "Filter out non-http(s) license urls" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338279 [19:35:17] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1075.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:35:20] (03PS1) 10Aranyap: Revert^2 "Filter out non-http(s) license urls" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338280 [19:36:07] (03PS1) 10Aranyap: Revert^2 "Filter out non-http(s) license urls" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338281 [19:36:10] FIRING: [8x] ProbeDown: Service aqs1026-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:36:16] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs1027.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [19:36:53] (03PS1) 10Aranyap: Revert^2 "Filter out non-http(s) license urls" [extensions/MediaSearch] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338282 [19:36:56] RECOVERY - Host aqs1027 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms [19:37:25] (03PS1) 10Aranyap: Revert^2 "Filter out non-http(s) license urls" [extensions/MediaSearch] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338283 [19:41:17] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 09 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338278 (owner: 10Aranyap) [19:41:34] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host aqs1027.eqiad.wmnet with OS bookworm [19:42:01] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 09 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338279 (owner: 10Aranyap) [19:42:34] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 09 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338280 (owner: 10Aranyap) [19:42:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 09 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338281 (owner: 10Aranyap) [19:43:01] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 09 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/MediaSearch] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338282 (owner: 10Aranyap) [19:43:12] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 09 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/MediaSearch] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338283 (owner: 10Aranyap) [19:44:41] (03CR) 10Scott French: [C:03+2] O:deployment_server::kubernetes: Bootstrap 'scap deploy-service' config [puppet] - 10https://gerrit.wikimedia.org/r/1338068 (https://phabricator.wikimedia.org/T412941) (owner: 10Scott French) [19:51:10] RESOLVED: [4x] ProbeDown: Service aqs1027-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:51:44] !log eevans@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1027.eqiad.wmnet with reason: host reimage [19:57:50] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1027.eqiad.wmnet with reason: host reimage [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: Your horoscope predicts another UTC late backport window deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T2000). [20:00:05] sbassett, aranyap: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:06:15] 06SRE, 06Infrastructure-Foundations: Create nodejs 26 production images - https://phabricator.wikimedia.org/T437509 (10Jdforrester-WMF) 03NEW [20:07:02] 06SRE, 06Infrastructure-Foundations: Add Nodejs 26 from Nodesource to Wikimedia apt - https://phabricator.wikimedia.org/T437510 (10Jdforrester-WMF) 03NEW [20:09:17] 06SRE, 06Infrastructure-Foundations: Create nodejs 26 production images - https://phabricator.wikimedia.org/T437509#12305150 (10Jdforrester-WMF) [20:10:49] (03PS4) 10CDobbins: prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) [20:11:21] 06SRE, 06Infrastructure-Foundations: Create nodejs 26 production images - https://phabricator.wikimedia.org/T437509#12305170 (10Jdforrester-WMF) [20:11:31] (03PS3) 10SBassett: Fix invalid input into strtolower [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338272 (https://phabricator.wikimedia.org/T435999) (owner: 10Aranyap) [20:12:46] (03PS4) 10SBassett: Fix invalid input into strtolower [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338272 (https://phabricator.wikimedia.org/T435999) (owner: 10Aranyap) [20:13:07] (03CR) 10CI reject: [V:04-1] Fix invalid input into strtolower [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338272 (https://phabricator.wikimedia.org/T435999) (owner: 10Aranyap) [20:13:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:13:35] (03CR) 10CI reject: [V:04-1] prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins) [20:14:49] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T437170#12305192 (10Jclark-ctr) Parts have not arrived yet will follow up tomorrow with dell [20:15:31] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1027.eqiad.wmnet with OS bookworm [20:16:50] (03PS5) 10CDobbins: prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) [20:19:10] (03CR) 10CI reject: [V:04-1] prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins) [20:19:24] (03PS2) 10SBassett: Revert^2 "Filter out non-http(s) license urls" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338278 (owner: 10Aranyap) [20:19:24] (03PS5) 10SBassett: Fix invalid input into strtolower [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338272 (https://phabricator.wikimedia.org/T435999) (owner: 10Aranyap) [20:19:56] (03PS2) 10SBassett: Fix invalid input into strtolower [extensions/CommonsMetadata] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338277 (https://phabricator.wikimedia.org/T435999) (owner: 10Aranyap) [20:20:25] FIRING: [3x] ProbeDown: Service aqs1027-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:21:38] (03PS1) 10Jdlrobson: Allow donor consent workflow to work on non-Minerva skins [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338287 (https://phabricator.wikimedia.org/T436572) [20:21:57] (03PS2) 10Jdlrobson: Enable ReadingLists on mediawiki and wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337984 (https://phabricator.wikimedia.org/T437113) [20:22:12] (03PS6) 10CDobbins: prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) [20:22:17] (03PS2) 10SBassett: Revert^2 "Filter out non-http(s) license urls" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338279 (owner: 10Aranyap) [20:22:17] (03PS3) 10SBassett: Fix invalid input into strtolower [extensions/CommonsMetadata] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338277 (https://phabricator.wikimedia.org/T435999) (owner: 10Aranyap) [20:22:57] !log rzl@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1342.eqiad.wmnet [20:23:01] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1342.eqiad.wmnet [20:23:37] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1342.eqiad.wmnet [20:23:56] !log rzl@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1342.eqiad.wmnet with OS trixie [20:24:25] !log rzl@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1342 [20:24:58] !log rzl@cumin2003 START - Cookbook sre.dns.netbox [20:25:25] RESOLVED: [4x] ProbeDown: Service aqs1027-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:26:01] Hey all - Aranya and I have a few patches to try to get out here again (the ones that failed yesterday) [20:27:03] (03CR) 10CDobbins: "I updated it to use python3-iptables. I haven't yet tested it, except to play around on my laptop. Would it be a problem if, assuming this" [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins) [20:27:22] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337196 (owner: 10Samwilson) [20:28:58] !log rzl@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1342 - rzl@cumin2003" [20:29:02] !log rzl@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1342 - rzl@cumin2003" [20:29:02] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [20:29:03] !log rzl@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1342.eqiad.wmnet 160.32.64.10.in-addr.arpa 0.6.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [20:29:05] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1342.eqiad.wmnet 160.32.64.10.in-addr.arpa 0.6.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [20:29:06] !log rzl@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1342 [20:29:43] !log rzl@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1342 [20:29:44] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1342 [20:32:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338280 (owner: 10Aranyap) [20:32:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338281 (owner: 10Aranyap) [20:32:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy1003 using scap backport" [extensions/MediaSearch] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338283 (owner: 10Aranyap) [20:32:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy1003 using scap backport" [extensions/MediaSearch] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338282 (owner: 10Aranyap) [20:32:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy1003 using scap backport" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338278 (owner: 10Aranyap) [20:32:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy1003 using scap backport" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338279 (owner: 10Aranyap) [20:32:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy1003 using scap backport" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338272 (https://phabricator.wikimedia.org/T435999) (owner: 10Aranyap) [20:32:26] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy1003 using scap backport" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338277 (https://phabricator.wikimedia.org/T435999) (owner: 10Aranyap) [20:33:37] 06SRE, 10LDAP-Access-Requests: Request for LDAP NDA Access for CentralNotice Metrics for Yahya - https://phabricator.wikimedia.org/T436294#12305254 (10KFrancis) Hi all, I am currently on sabbatical. Please contact rstallman@wikimedia.org or naramayo@wikimedia.org to help process this request. Thank you! [20:33:56] (03Merged) 10jenkins-bot: Revert^2 "Filter out non-http(s) license urls" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338280 (owner: 10Aranyap) [20:33:57] (03Merged) 10jenkins-bot: Revert^2 "Filter out non-http(s) license urls" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338281 (owner: 10Aranyap) [20:33:58] (03Merged) 10jenkins-bot: Revert^2 "Filter out non-http(s) license urls" [extensions/MediaSearch] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338283 (owner: 10Aranyap) [20:33:59] (03Merged) 10jenkins-bot: Revert^2 "Filter out non-http(s) license urls" [extensions/MediaSearch] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338282 (owner: 10Aranyap) [20:34:01] (03Merged) 10jenkins-bot: Revert^2 "Filter out non-http(s) license urls" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338278 (owner: 10Aranyap) [20:34:03] (03Merged) 10jenkins-bot: Revert^2 "Filter out non-http(s) license urls" [extensions/CommonsMetadata] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338279 (owner: 10Aranyap) [20:34:09] (03Merged) 10jenkins-bot: Fix invalid input into strtolower [extensions/CommonsMetadata] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338272 (https://phabricator.wikimedia.org/T435999) (owner: 10Aranyap) [20:34:15] (03Merged) 10jenkins-bot: Fix invalid input into strtolower [extensions/CommonsMetadata] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338277 (https://phabricator.wikimedia.org/T435999) (owner: 10Aranyap) [20:34:40] !log sbassett@deploy1003 Started scap sync-world: Backport for [[gerrit:1338280|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338281|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338283|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338282|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338278|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338279|Revert^2 " [20:34:40] Filter out non-http(s) license urls"]], [[gerrit:1338272|Fix invalid input into strtolower (T435999)]], [[gerrit:1338277|Fix invalid input into strtolower (T435999)]] [20:35:22] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1075.eqiad.wmnet with OS trixie [20:35:38] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12305258 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1075.eqiad.wmnet with OS trixie [20:39:06] !log sbassett@deploy1003 aranyap, sbassett: Backport for [[gerrit:1338280|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338281|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338283|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338282|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338278|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338279|Revert^2 "Filter [20:39:06] out non-http(s) license urls"]], [[gerrit:1338272|Fix invalid input into strtolower (T435999)]], [[gerrit:1338277|Fix invalid input into strtolower (T435999)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:39:59] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12305260 (10Geertivp) I don't remember when I last used my admin rights... could be more than 8 years... anyway my user account geert@ got deleted... but my e-mail adress is still on the `wikim... [20:40:49] !log sbassett@deploy1003 aranyap, sbassett: Continuing with deployment [20:41:46] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Attempt to use a VM as Kerberos replica - https://phabricator.wikimedia.org/T435873#12305274 (10bking) The maintenance is complete, but the results were a little unexpected. We hoped to exercise `krb1004` by putting it first in t... [20:41:50] !log eevans@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sessionstore2004.codfw.wmnet [20:42:03] !log rzl@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1342.eqiad.wmnet with reason: host reimage [20:45:26] !log sbassett@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338280|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338281|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338283|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338282|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338278|Revert^2 "Filter out non-http(s) license urls"]], [[gerrit:1338279|Revert^2 [20:45:26] "Filter out non-http(s) license urls"]], [[gerrit:1338272|Fix invalid input into strtolower (T435999)]], [[gerrit:1338277|Fix invalid input into strtolower (T435999)]] (duration: 10m 46s) [20:48:33] (03PS7) 10CDobbins: prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) [20:49:27] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1342.eqiad.wmnet with reason: host reimage [20:50:09] !log vriley@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1075.eqiad.wmnet with reason: host reimage [20:52:05] (03PS8) 10CDobbins: prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) [20:53:12] !log bking@cumin2003 START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. [20:53:29] !log bking@cumin2003 END (ERROR) - Cookbook sre.presto.roll-restart-workers (exit_code=97) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. [20:53:48] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1075.eqiad.wmnet with reason: host reimage [20:54:12] !log eevans@cumin1003 START - Cookbook sre.hosts.reboot-single for host sessionstore2004.codfw.wmnet [20:54:50] !log bking@cumin2003 START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. [20:56:59] !log bking@cumin2003 END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. [20:57:11] !log bking@cumin2003 START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-canary cluster: Roll restart of all Presto's jvm daemons. [20:57:40] FIRING: [2x] ProbeDown: Service sessionstore2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:59:32] !log bking@cumin2003 END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-canary cluster: Roll restart of all Presto's jvm daemons. [21:00:05] Deploy window Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T2100) [21:01:33] (03CR) 10Volans: "[this is not a review] I've just noticed a question/comment about dry-run and had a quick look." [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (https://phabricator.wikimedia.org/T435265) (owner: 10Herron) [21:05:00] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore2004.codfw.wmnet [21:05:01] !log eevans@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts sessionstore2004.codfw.wmnet [21:06:52] !log eevans@cumin1003 START - Cookbook sre.hosts.provision for host sessionstore2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [21:07:30] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [21:07:31] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1342.eqiad.wmnet with OS trixie [21:07:40] RESOLVED: [2x] ProbeDown: Service sessionstore2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:08:00] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host sessionstore2004.codfw.wmnet with OS bookworm [21:08:25] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338273 (https://phabricator.wikimedia.org/T432849) (owner: 10Jforrester) [21:09:20] (03Merged) 10jenkins-bot: wikifunctions: Move client fragments to mainstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338273 (https://phabricator.wikimedia.org/T432849) (owner: 10Jforrester) [21:09:40] !log jforrester@deploy1003 Started scap sync-world: Backport for [[gerrit:1338273|wikifunctions: Move client fragments to mainstash (T432849)]] [21:09:43] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [21:09:44] T432849: [Milestone 1] Observability - Build tools, document and observe behavior and evolution of our multi-layer caching system - https://phabricator.wikimedia.org/T432849 [21:12:47] vriley@cumin1003 reimage (PID 4017867) is awaiting input [21:12:55] FIRING: [2x] ProbeDown: Service sessionstore2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:14:00] !log jforrester@deploy1003 jforrester: Backport for [[gerrit:1338273|wikifunctions: Move client fragments to mainstash (T432849)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:15:33] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [21:15:34] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1075.eqiad.wmnet with OS trixie [21:15:48] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12305397 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1075.eqiad.wmnet with OS trixie completed: -... [21:16:57] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12305407 (10VRiley-WMF) 05In progress→03Resolved @fgiunchedi this one is completed [21:17:33] !log jforrester@deploy1003 jforrester: Continuing with deployment [21:17:55] RESOLVED: [2x] ProbeDown: Service sessionstore2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:19:08] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1342.eqiad.wmnet [21:19:10] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1342.eqiad.wmnet [21:19:11] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1342.eqiad.wmnet [21:20:49] !log rzl@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1343.eqiad.wmnet [21:20:53] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1343.eqiad.wmnet [21:21:28] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1343.eqiad.wmnet [21:21:55] !log rzl@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1343.eqiad.wmnet with OS trixie [21:22:09] !log jforrester@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338273|wikifunctions: Move client fragments to mainstash (T432849)]] (duration: 12m 29s) [21:22:13] T432849: [Milestone 1] Observability - Build tools, document and observe behavior and evolution of our multi-layer caching system - https://phabricator.wikimedia.org/T432849 [21:22:25] !log rzl@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1343 [21:23:02] !log rzl@cumin2003 START - Cookbook sre.dns.netbox [21:27:27] !log rzl@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1343 - rzl@cumin2003" [21:27:31] !log rzl@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1343 - rzl@cumin2003" [21:27:31] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [21:27:32] !log rzl@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1343.eqiad.wmnet 193.48.64.10.in-addr.arpa 3.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [21:27:35] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1343.eqiad.wmnet 193.48.64.10.in-addr.arpa 3.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [21:27:36] !log rzl@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1343 [21:28:31] !log rzl@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1343 [21:28:31] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1343 [21:36:36] !log eevans@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore2004.codfw.wmnet with OS bookworm [21:36:51] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host sessionstore2004.codfw.wmnet with OS bookworm [21:40:32] !log rzl@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1343.eqiad.wmnet with reason: host reimage [21:44:06] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1343.eqiad.wmnet with reason: host reimage [21:59:17] !log eevans@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore2004.codfw.wmnet with OS bookworm [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260909T2200) [22:00:07] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host sessionstore2004.codfw.wmnet with OS bookworm [22:00:19] (03PS3) 10Scott French: mediawiki: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338000 (https://phabricator.wikimedia.org/T427666) [22:00:48] (03PS2) 10Scott French: Update mesh.{configuration,networkpolicy} and dependents in all charts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338030 (https://phabricator.wikimedia.org/T427666) [22:00:54] Hi all. @derenrich and I will be deploying https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1337999 in the readers backport window shortly [22:01:03] 👋 [22:03:07] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1343.eqiad.wmnet with OS trixie [22:04:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by derenrich@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337999 (https://phabricator.wikimedia.org/T437344) (owner: 10DErenrich) [22:04:39] (03CR) 10Scott French: [C:04-1] Update mesh.{configuration,networkpolicy} and dependents in all charts (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338030 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [22:05:25] (03Merged) 10jenkins-bot: Enable discord preview extension code on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337999 (https://phabricator.wikimedia.org/T437344) (owner: 10DErenrich) [22:05:50] !log derenrich@deploy1003 Started scap sync-world: Backport for [[gerrit:1337999|Enable discord preview extension code on testwiki (T437344)]] [22:05:54] T437344: Enable WMC Discord REST Endpoint - https://phabricator.wikimedia.org/T437344 [22:10:10] !log derenrich@deploy1003 derenrich: Backport for [[gerrit:1337999|Enable discord preview extension code on testwiki (T437344)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:13:13] (03Abandoned) 10Scott French: P:kubernetes::deployment_server::global_config: Extend services_proxy [puppet] - 10https://gerrit.wikimedia.org/r/1325992 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [22:13:13] (03Abandoned) 10Scott French: P:services_proxy::envoy: Add support for split host_regex [puppet] - 10https://gerrit.wikimedia.org/r/1325989 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [22:13:14] (03Abandoned) 10Scott French: P:services_proxy::envoy: Fix split set_sni behavior [puppet] - 10https://gerrit.wikimedia.org/r/1325988 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [22:13:15] (03Abandoned) 10Scott French: Profile::Service_listener: Add sni_rewrites_host_header to 'split' [puppet] - 10https://gerrit.wikimedia.org/r/1325987 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [22:16:53] !log derenrich@deploy1003 derenrich: Rolling back deployment [22:19:31] !log derenrich@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337999|Enable discord preview extension code on testwiki (T437344)]] (duration: 13m 40s) [22:19:34] T437344: Enable WMC Discord REST Endpoint - https://phabricator.wikimedia.org/T437344 [22:19:55] (03PS1) 10Eric Gardner: Revert "Enable discord preview extension code on testwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338303 [22:20:22] (03CR) 10DErenrich: [C:03+1] Revert "Enable discord preview extension code on testwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338303 (owner: 10Eric Gardner) [22:20:26] !log eevans@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore2004.codfw.wmnet with reason: host reimage [22:20:43] (03CR) 10TrainBranchBot: [C:03+2] "Approved by derenrich@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338303 (owner: 10Eric Gardner) [22:21:44] (03Merged) 10jenkins-bot: Revert "Enable discord preview extension code on testwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338303 (owner: 10Eric Gardner) [22:22:02] !log derenrich@deploy1003 Started scap sync-world: Backport for [[gerrit:1338303|Revert "Enable discord preview extension code on testwiki"]] [22:22:11] (03CR) 10Ladsgroup: [C:03+2] "I was a bit worried that it might get reused in production but blubber config points to a different file altogether." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337196 (owner: 10Samwilson) [22:22:48] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1343.eqiad.wmnet [22:22:50] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1343.eqiad.wmnet [22:22:52] !log rzl@cumin2003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1343.eqiad.wmnet [22:24:19] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore2004.codfw.wmnet with reason: host reimage [22:25:10] (03Merged) 10jenkins-bot: Set development log level to 'debug' [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337196 (owner: 10Samwilson) [22:26:56] !log derenrich@deploy1003 derenrich, egardner: Backport for [[gerrit:1338303|Revert "Enable discord preview extension code on testwiki"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:27:54] !log derenrich@deploy1003 derenrich, egardner: Continuing with deployment [22:29:00] (03CR) 10Ladsgroup: [C:04-1] create wikipedia-ar-arbcom.wikimedia.org (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn) [22:31:39] !log rzl@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1344.eqiad.wmnet [22:31:44] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1344.eqiad.wmnet [22:32:19] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1344.eqiad.wmnet [22:32:24] !log derenrich@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338303|Revert "Enable discord preview extension code on testwiki"]] (duration: 10m 21s) [22:32:53] Ok, we're done for today. We will try again tomorrow [22:33:06] !log rzl@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1344.eqiad.wmnet with OS trixie [22:33:35] !log rzl@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1344 [22:33:56] !log rzl@cumin2003 START - Cookbook sre.dns.netbox [22:36:31] (03CR) 10BCornwall: [C:03+1] Add pki svc IP A and PTR records [dns] - 10https://gerrit.wikimedia.org/r/1338234 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [22:37:39] !log rzl@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1344 - rzl@cumin2003" [22:37:43] !log rzl@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1344 - rzl@cumin2003" [22:37:43] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [22:37:44] !log rzl@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1344.eqiad.wmnet 194.48.64.10.in-addr.arpa 4.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [22:37:46] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1344.eqiad.wmnet 194.48.64.10.in-addr.arpa 4.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [22:37:47] !log rzl@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1344 [22:39:05] !log rzl@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1344 [22:39:05] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1344 [22:45:12] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore2004.codfw.wmnet with OS bookworm [22:48:40] FIRING: [2x] ProbeDown: Service sessionstore2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:51:30] !log rzl@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1344.eqiad.wmnet with reason: host reimage [22:57:58] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1344.eqiad.wmnet with reason: host reimage [23:04:12] (03PS1) 10Southparkfan: profile::mediawiki::scap_client: allow configuring betacluster_udp2log_host [puppet] - 10https://gerrit.wikimedia.org/r/1338305 (https://phabricator.wikimedia.org/T436469) [23:04:48] (03CR) 10CI reject: [V:04-1] profile::mediawiki::scap_client: allow configuring betacluster_udp2log_host [puppet] - 10https://gerrit.wikimedia.org/r/1338305 (https://phabricator.wikimedia.org/T436469) (owner: 10Southparkfan) [23:05:49] (03PS1) 10Southparkfan: LabsServices: adjust udp2log to use service RR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338306 (https://phabricator.wikimedia.org/T436469) [23:09:34] (03PS2) 10Southparkfan: profile::mediawiki::scap_client: allow configuring betacluster_udp2log_host [puppet] - 10https://gerrit.wikimedia.org/r/1338305 (https://phabricator.wikimedia.org/T436469) [23:09:42] jouncebot: nowandnext [23:09:42] No deployments scheduled for the next 2 hour(s) and 50 minute(s) [23:09:42] In 2 hour(s) and 50 minute(s): Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T0200) [23:10:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338142 (https://phabricator.wikimedia.org/T398709) (owner: 10Ladsgroup) [23:11:18] (03Merged) 10jenkins-bot: Move testcommonswiki links tables to x4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338142 (https://phabricator.wikimedia.org/T398709) (owner: 10Ladsgroup) [23:11:39] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1338142|Move testcommonswiki links tables to x4 (T398709)]] [23:11:43] T398709: FY2025-26 WE 6.4.1: Move links tables of commons to a dedicated cluster - https://phabricator.wikimedia.org/T398709 [23:12:02] (03CR) 10CI reject: [V:04-1] profile::mediawiki::scap_client: allow configuring betacluster_udp2log_host [puppet] - 10https://gerrit.wikimedia.org/r/1338305 (https://phabricator.wikimedia.org/T436469) (owner: 10Southparkfan) [23:15:06] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1344.eqiad.wmnet with OS trixie [23:15:43] (03PS3) 10Southparkfan: profile::mediawiki::scap_client: allow configuring betacluster_udp2log_host [puppet] - 10https://gerrit.wikimedia.org/r/1338305 (https://phabricator.wikimedia.org/T436469) [23:15:59] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1338142|Move testcommonswiki links tables to x4 (T398709)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:18:26] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [23:18:40] RESOLVED: [2x] ProbeDown: Service sessionstore2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:18:41] (03CR) 10BCornwall: [V:04-1] "Yeah. Though I'd probably go ahead and test on clouddumps1002 manually - running the script directly and ensuring the output is as expecte" [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins) [23:20:03] (03CR) 10BCornwall: create wikipedia-ar-arbcom.wikimedia.org (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn) [23:22:54] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338142|Move testcommonswiki links tables to x4 (T398709)]] (duration: 11m 15s) [23:22:57] T398709: FY2025-26 WE 6.4.1: Move links tables of commons to a dedicated cluster - https://phabricator.wikimedia.org/T398709 [23:26:45] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1344.eqiad.wmnet [23:26:47] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1344.eqiad.wmnet [23:26:48] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1344.eqiad.wmnet [23:27:22] !log rzl@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1345.eqiad.wmnet [23:27:23] (03CR) 10BCornwall: [C:04-1] P:tofurkey enable Tofurkey for MAGRU [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [23:27:26] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1345.eqiad.wmnet [23:27:59] (03CR) 10BCornwall: [C:03+1] editcheck-headless: Add ingress discovery records [dns] - 10https://gerrit.wikimedia.org/r/1338080 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [23:28:01] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1345.eqiad.wmnet [23:28:14] !log rzl@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1345.eqiad.wmnet with OS trixie [23:28:37] FIRING: [2x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [23:28:43] !log rzl@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1345 [23:28:56] !log rzl@cumin2003 START - Cookbook sre.dns.netbox [23:29:12] (03CR) 10BCornwall: [C:03+1] Add tool-server CNAMEs to k8s-ingress-ml-serve [dns] - 10https://gerrit.wikimedia.org/r/1337904 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [23:32:11] (03CR) 10Santiago Faci: [C:03+2] test-kitchen: Bump chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338150 (https://phabricator.wikimedia.org/T433733) (owner: 10Santiago Faci) [23:32:59] !log rzl@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1345 - rzl@cumin2003" [23:33:03] !log rzl@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1345 - rzl@cumin2003" [23:33:03] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [23:33:04] !log rzl@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1345.eqiad.wmnet 195.48.64.10.in-addr.arpa 5.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [23:33:07] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1345.eqiad.wmnet 195.48.64.10.in-addr.arpa 5.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [23:33:08] !log rzl@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1345 [23:33:39] !log rzl@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1345 [23:33:40] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1345 [23:34:24] (03Merged) 10jenkins-bot: test-kitchen: Bump chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338150 (https://phabricator.wikimedia.org/T433733) (owner: 10Santiago Faci) [23:35:13] (03CR) 10Scott French: "Many thanks for taking a look!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335285 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [23:35:48] (03CR) 10Dzahn: "both of your comments are exactly what I wanted to ask with this review :)" [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn) [23:37:24] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [23:37:30] (03CR) 10Andrea Denisse: [C:03+2] aptrepo: Add grafana-plugins to main apt repo [puppet] - 10https://gerrit.wikimedia.org/r/1338064 (https://phabricator.wikimedia.org/T436056) (owner: 10Andrea Denisse) [23:37:31] (03PS2) 10Dzahn: create wikipedia-ar-arbcom.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) [23:37:36] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [23:37:44] (03PS1) 10MusikAnimal: [CodeMirror] enable for new users (enwiki), new VE integration (global) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) [23:38:04] jouncebot: nowandnext [23:38:04] No deployments scheduled for the next 2 hour(s) and 21 minute(s) [23:38:04] In 2 hour(s) and 21 minute(s): Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T0200) [23:38:14] ack, going to merge a beta-only change here [23:38:29] (03CR) 10Dzahn: create wikipedia-ar-arbcom.wikimedia.org (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn) [23:38:51] (03PS2) 10MusikAnimal: [CodeMirror] enable for new users (enwiki), new VE integration (global) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) [23:38:56] (03PS3) 10Dzahn: create wikipedia-ar-arbcom.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) [23:39:32] (03PS3) 10MusikAnimal: [CodeMirror] enable for new users (enwiki), new VE integration (global) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) [23:39:41] (03CR) 10TrainBranchBot: [C:03+2] "Approved by thcipriani@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338306 (https://phabricator.wikimedia.org/T436469) (owner: 10Southparkfan) [23:40:40] (03Merged) 10jenkins-bot: LabsServices: adjust udp2log to use service RR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338306 (https://phabricator.wikimedia.org/T436469) (owner: 10Southparkfan) [23:41:27] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1338309 [23:41:27] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1338309 (owner: 10TrainBranchBot) [23:43:13] (03PS4) 10MusikAnimal: [CodeMirror] enable for new users (enwiki), new VE integration (global) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) [23:46:09] !log rzl@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1345.eqiad.wmnet with reason: host reimage [23:49:03] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1345.eqiad.wmnet with reason: host reimage [23:51:20] 06SRE, 10LDAP-Access-Requests: Grant Access to <500> for - https://phabricator.wikimedia.org/T437488#12305844 (10Dzahn) Hi @MMilenkovic-WMF are you asking only for basic access to Superset (level 1)? Or additionally for access to private data in it (level 2)? Or in addition to that also fo... [23:51:26] (03PS1) 10Santiago Faci: test-kitchen-next: Disable validate_deployment_windows [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338310 (https://phabricator.wikimedia.org/T433733) [23:52:34] (03PS2) 10Santiago Faci: test-kitchen-next: Disable validate_deployment_windows [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338310 (https://phabricator.wikimedia.org/T433733) [23:52:46] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1338309 (owner: 10TrainBranchBot) [23:52:56] (03CR) 10Ladsgroup: create wikipedia-ar-arbcom.wikimedia.org (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn) [23:52:56] (03PS3) 10Santiago Faci: test-kitchen-next: Disable validate_deployment_windows [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338310 (https://phabricator.wikimedia.org/T433733) [23:53:32] (03CR) 10Ladsgroup: [C:03+1] "I don't know why it's keeping my -1" [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn) [23:53:53] 06SRE, 10LDAP-Access-Requests: Grant Access to <500> for - https://phabricator.wikimedia.org/T437488#12305848 (10Dzahn) by the way: In case you come from some documentation for onboarding or an onboarding buddy told you to make this request; please also give feedback that this distinction betw... [23:54:17] (03PS4) 10Santiago Faci: test-kitchen-next: Disable validate_deployment_windows [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338310 (https://phabricator.wikimedia.org/T433733) [23:54:52] (03PS5) 10Andrea Denisse: grafana: Add tamirsuliman-weathermap-panel to sandbox [puppet] - 10https://gerrit.wikimedia.org/r/1338056 (https://phabricator.wikimedia.org/T436075) [23:55:39] (03CR) 10Dzahn: [C:03+1] create wikipedia-ar-arbcom.wikimedia.org (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn) [23:56:12] (03CR) 10Andrea Denisse: "Thanks Keith, I added some info about it in the commit." [puppet] - 10https://gerrit.wikimedia.org/r/1338056 (https://phabricator.wikimedia.org/T436075) (owner: 10Andrea Denisse) [23:57:36] (03CR) 10Dzahn: [C:03+1] "Mildly agree we should not add something that is deprecated. Let's do another one to remove the old .m. entries." [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn)