[00:00:25] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1318796 (owner: 10TrainBranchBot) [00:01:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node ml-serve2004 has a BGP session which is not in the 'established' state. [00:12:10] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 193306640 and 30 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [00:12:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:14:10] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 13344 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [00:17:25] (03CR) 10Scott French: "Thanks, Blake!" [software/httpbb] - 10https://gerrit.wikimedia.org/r/1318682 (https://phabricator.wikimedia.org/T428972) (owner: 10Blake) [00:25:26] (03CR) 10Scott French: [C:03+1] "Ah, interesting!" [puppet] - 10https://gerrit.wikimedia.org/r/1318302 (https://phabricator.wikimedia.org/T263872) (owner: 10Ahmon Dancy) [00:31:47] (03CR) 10Scott French: [C:03+1] "Wow, that was quick! Thank very much for moving this into integration/config. This LGTM. Please poke me tomorrow and I'll get this merged." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318767 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [00:32:09] (03CR) 10Scott French: [V:03+2 C:03+1] "Trivially builds (there is nothing to build)." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318767 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [00:49:27] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:50:26] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [00:52:57] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306988 (https://phabricator.wikimedia.org/T433314) (owner: 10BPirkle) [00:53:25] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312694 (https://phabricator.wikimedia.org/T422405) (owner: 10Aaron Schulz) [00:58:12] (03CR) 10Ottomata: [C:03+1] [eventstreams] Bump to v0.20.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318784 (https://phabricator.wikimedia.org/T432824) (owner: 10TChin) [01:12:28] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1318807 [01:12:28] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1318807 (owner: 10TrainBranchBot) [01:27:45] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1318807 (owner: 10TrainBranchBot) [01:48:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:08:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:12:25] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:13:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:14:27] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [02:14:37] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:47:11] (03CR) 10Ryan Kemper: [C:03+2] pontoon: add rkemper-kafka stack [puppet] - 10https://gerrit.wikimedia.org/r/1318761 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [03:55:05] FIRING: KubernetesCalicoDown: ml-serve2004.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s-mlserve&var-instance=ml-serve2004.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [03:58:58] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [03:59:04] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [04:01:15] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node ml-serve2004 has a BGP session which is not in the 'established' state. [04:30:07] (03PS1) 10Samwilson: Disallow all audio formats in the format filter [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1318810 (https://phabricator.wikimedia.org/T431671) [04:43:58] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [04:50:26] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [05:06:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-eqord and cr3-ulsfo (198.35.26.128) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [05:08:58] FIRING: [7x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:25:21] (03PS1) 10Ryan Kemper: Transferer: log the reason a copy failed [software/transferpy] - 10https://gerrit.wikimedia.org/r/1318972 (https://phabricator.wikimedia.org/T430880) [05:36:30] (03CR) 10Ryan Kemper: "Small correction because it's bothering me — my commit message said it unwinds the downtime, but it actually (correctly) leaves the downti" [cookbooks] - 10https://gerrit.wikimedia.org/r/1317127 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [05:46:53] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1164.eqiad.wmnet with reason: Reimage [05:47:40] !log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host db1164.eqiad.wmnet with OS trixie [05:48:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:00:04] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T0600) [06:01:23] !log marostegui@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db1164.eqiad.wmnet with reason: host reimage [06:03:55] (03PS1) 10Marostegui: check_private_data_report: Add clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1318973 (https://phabricator.wikimedia.org/T409557) [06:04:57] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Add clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1318973 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [06:05:11] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1164.eqiad.wmnet with reason: host reimage [06:07:55] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309675 (https://phabricator.wikimedia.org/T431830) (owner: 10Dreamrimmer) [06:14:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:14:27] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:14:37] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:26:57] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1164.eqiad.wmnet with OS trixie [06:31:29] (03PS1) 10Slyngshede: data.yaml sync WMF group members with LDAP [puppet] - 10https://gerrit.wikimedia.org/r/1318975 [06:39:43] (03CR) 10Slyngshede: [C:03+1] data.yaml sync WMF group members with LDAP [puppet] - 10https://gerrit.wikimedia.org/r/1318975 (owner: 10Slyngshede) [06:39:45] (03CR) 10Slyngshede: [C:03+2] data.yaml sync WMF group members with LDAP [puppet] - 10https://gerrit.wikimedia.org/r/1318975 (owner: 10Slyngshede) [06:44:13] 10SRE-swift-storage, 06Commons, 06Data-Persistence, 10MediaWiki-File-management, 07Wikimedia-production-error: DBTransactionSizeError when trying to move file - https://phabricator.wikimedia.org/T431616#12165936 (10Marostegui) [06:55:19] (03CR) 10Marostegui: "The thing is, that the new master is very unlikely to have a pt-heartbeat running at that time as it will be a replica of the primary mast" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [06:57:30] (03PS1) 10Ryan Kemper: kafka: converge topic config from hieradata [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) [06:57:51] (03PS1) 10Marostegui: db2232: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1318977 (https://phabricator.wikimedia.org/T433463) [06:58:10] (03CR) 10Ryan Kemper: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [06:58:14] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2160,2232].codfw.wmnet with reason: Reimage [06:58:44] (03CR) 10Marostegui: [C:03+2] db2232: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1318977 (https://phabricator.wikimedia.org/T433463) (owner: 10Marostegui) [06:58:58] !log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host db2232.codfw.wmnet with OS trixie [07:00:04] Amir1, urbanecm, and awight: How many deployers does it take to do UTC morning backport window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T0700). [07:00:04] DreamRimmer: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:15:40] !log marostegui@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db2232.codfw.wmnet with reason: host reimage [07:19:30] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2232.codfw.wmnet with reason: host reimage [07:23:54] (03CR) 10Brouberol: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311441 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [07:31:08] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [07:33:03] (03PS1) 10Marostegui: Revert "db2232: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1318984 [07:33:58] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [07:34:09] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [07:34:26] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [07:36:13] I'm going to reload bird on ganeti2034, only netops VMs are running there, it might just make a bit of noise [07:37:51] (done) [07:42:12] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2232.codfw.wmnet with OS trixie [07:42:53] !log ayounsi@cumin1003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet [07:43:03] (03CR) 10Marostegui: [C:03+2] Revert "db2232: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1318984 (owner: 10Marostegui) [07:46:35] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet [07:46:45] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1163.eqiad.wmnet with reason: Maintenance [07:46:53] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1163 (T431660)', diff saved to https://phabricator.wikimedia.org/P95441 and previous config saved to /var/cache/conftool/dbconfig/20260729-074652-cwilliams.json [07:48:03] (03PS1) 10Btullis: hadoop-test: Switch to using the new namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318988 (https://phabricator.wikimedia.org/T406746) [07:48:28] (03PS1) 10Marostegui: eqiad.yaml: Add clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1318989 (https://phabricator.wikimedia.org/T409557) [07:48:49] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance [07:48:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [07:48:59] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [07:49:06] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [07:49:07] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1156 (T431660)', diff saved to https://phabricator.wikimedia.org/P95442 and previous config saved to /var/cache/conftool/dbconfig/20260729-074906-cwilliams.json [07:49:14] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1157 (T431660)', diff saved to https://phabricator.wikimedia.org/P95443 and previous config saved to /var/cache/conftool/dbconfig/20260729-074914-cwilliams.json [07:49:23] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance [07:49:31] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1190 (T431660)', diff saved to https://phabricator.wikimedia.org/P95444 and previous config saved to /var/cache/conftool/dbconfig/20260729-074930-cwilliams.json [07:50:10] FIRING: [4x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [07:50:18] RECOVERY - Host ml-serve2004 is UP: PING OK - Packet loss = 0%, RTA = 34.46 ms [07:53:22] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1163: Maintenance [07:54:50] RESOLVED: KubernetesCalicoDown: ml-serve2004.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s-mlserve&var-instance=ml-serve2004.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:55:34] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1157: Maintenance [07:55:38] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1156: Maintenance [07:56:00] RESOLVED: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node ml-serve2004 has a BGP session which is not in the 'established' state. [07:57:11] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1190: Maintenance [07:58:31] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2072.codfw.wmnet with OS trixie [07:58:46] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12166088 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2072.codfw.wmnet with OS trixie [07:58:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:02:16] (03PS1) 10Bartosz Wójtowicz: ml-services: Update gpt-oss-safeguard-20b docker image. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319026 (https://phabricator.wikimedia.org/T432933) [08:03:03] (03PS3) 10Trueg: WDQSv2: Scheduling based on resource requests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318733 (https://phabricator.wikimedia.org/T431395) [08:04:13] (03PS30) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [08:04:50] 10ops-codfw, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team: hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476 (10klausman) 03NEW [08:05:09] (03CR) 10Federico Ceratto: "ok, I added the CLI option and updated the tests" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [08:06:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [08:07:18] (03PS1) 10Btullis: hadoop-test: Replace an-test-master namenodes with refreshed hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319027 (https://phabricator.wikimedia.org/T406746) [08:08:24] 10ops-codfw, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team: hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478 (10klausman) 03NEW [08:08:49] 06SRE, 10SRE-swift-storage, 05Goal, 06Machine-Learning-Team (Q1 FY2026-27), 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12166214 (10MatthewVernon) [08:10:28] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12166229 (10klausman) As is tradition during reboots, I also found two machines with... [08:11:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [08:12:07] (03CR) 10Brouberol: [C:03+1] hadoop-test: Replace an-test-master namenodes with refreshed hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319027 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [08:12:34] (03CR) 10Brouberol: [C:03+1] hadoop-test: Switch to using the new namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318988 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [08:12:53] (03CR) 10Marostegui: mysql: update replication source (033 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [08:13:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:14:14] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: eqiad row A&B host migration details request for Data Persistence - https://phabricator.wikimedia.org/T432644#12166236 (10MatthewVernon) I've done this for thanos and ms-{fe,be} hosts. Can I check that this is just physical moving, no renumbering? [08:19:40] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2072.codfw.wmnet with reason: host reimage [08:20:20] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 1098 hosts [08:23:46] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2072.codfw.wmnet with reason: host reimage [08:23:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:24:23] (03PS4) 10Trueg: WDQSv2: Scheduling based on resource requests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318733 (https://phabricator.wikimedia.org/T431395) [08:30:28] !log btullis@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 11 hosts with reason: Replacing the namenodes [08:30:29] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1318228 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [08:34:37] (03CR) 10Marostegui: [C:03+2] eqiad.yaml: Add clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1318989 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [08:37:24] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Update gpt-oss-safeguard-20b docker image. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319026 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [08:38:34] (03PS1) 10JMeybohm: Update maintainer fields of core components to the k8s SIG [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1319032 (https://phabricator.wikimedia.org/T373526) [08:41:20] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1163: Maintenance [08:41:40] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance [08:41:48] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1169 (T431660)', diff saved to https://phabricator.wikimedia.org/P95458 and previous config saved to /var/cache/conftool/dbconfig/20260729-084147-cwilliams.json [08:43:32] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Maintenance [08:43:52] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [08:44:01] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1166 (T431660)', diff saved to https://phabricator.wikimedia.org/P95460 and previous config saved to /var/cache/conftool/dbconfig/20260729-084400-cwilliams.json [08:44:08] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1156: Maintenance [08:44:28] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance [08:44:37] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1182 (T431660)', diff saved to https://phabricator.wikimedia.org/P95462 and previous config saved to /var/cache/conftool/dbconfig/20260729-084436-cwilliams.json [08:45:08] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1190: Maintenance [08:45:27] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: depool magru [reason: router upgrade, T431750] [08:45:27] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance [08:45:29] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: router upgrade, T431750] [08:45:31] T431750: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750 [08:45:35] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1199 (T431660)', diff saved to https://phabricator.wikimedia.org/P95464 and previous config saved to /var/cache/conftool/dbconfig/20260729-084534-cwilliams.json [08:45:41] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2072.codfw.wmnet with OS trixie [08:45:48] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12166327 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2072.codfw.wmnet with OS trixie completed: - ms-be2072 (**PASS*... [08:46:12] !log ayounsi@cumin1003 DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on cr1-magru,cr1-magru IPv6,cr1-magru.mgmt with reason: router upgrade [08:47:19] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1169: Maintenance [08:48:26] (03PS1) 10Ayounsi: Add magru core routers mgmt interface monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1319034 [08:50:19] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1166: Maintenance [08:50:52] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1182: Maintenance [08:51:44] (03CR) 10Cathal Mooney: [C:03+1] Add magru core routers mgmt interface monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1319034 (owner: 10Ayounsi) [08:52:28] (03PS1) 10Ayounsi: cr2-magru: enable BGP RIB sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1319035 (https://phabricator.wikimedia.org/T320264) [08:52:43] (03CR) 10Btullis: [C:03+2] hadoop-test: Switch to using the new namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318988 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [08:52:48] (03CR) 10Ayounsi: [C:03+2] Add magru core routers mgmt interface monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1319034 (owner: 10Ayounsi) [08:53:10] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1199: Maintenance [08:53:28] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: Update gpt-oss-safeguard-20b docker image. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319026 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [08:53:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:53:57] (03CR) 10Cathal Mooney: [C:03+2] cr2-magru: enable BGP RIB sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1319035 (https://phabricator.wikimedia.org/T320264) (owner: 10Ayounsi) [08:55:38] (03Merged) 10jenkins-bot: cr2-magru: enable BGP RIB sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1319035 (https://phabricator.wikimedia.org/T320264) (owner: 10Ayounsi) [08:55:45] (03Merged) 10jenkins-bot: ml-services: Update gpt-oss-safeguard-20b docker image. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319026 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [08:56:31] (03CR) 10Btullis: [C:03+2] hadoop-test: Replace an-test-master namenodes with refreshed hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319027 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [08:58:39] (03PS1) 10Hnowlan: debian: update dependencies to add systemd [debs/pint] - 10https://gerrit.wikimedia.org/r/1319039 [08:58:52] (03Merged) 10jenkins-bot: hadoop-test: Replace an-test-master namenodes with refreshed hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319027 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [09:00:01] !log Dropping renamed tables T426341 [09:00:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:00:05] T426341: Drop DiscussionTools database tables from wikis which don't need them - https://phabricator.wikimedia.org/T426341 [09:00:14] !log ayounsi@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr1-magru,cr1-magru IPv6,cr1-magru.mgmt with reason: router upgrade [09:00:21] 06SRE, 06Infrastructure-Foundations, 10netops: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166353 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a746a25c-706b-4c61-bdf4-214a1c78ff90) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their servi... [09:00:47] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host idp2005.wikimedia.org [09:02:49] (03PS4) 10Blake: mw-pretrain: Add a canary values.yaml. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318671 (https://phabricator.wikimedia.org/T427668) [09:03:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:03:55] (03CR) 10Blake: mw-pretrain: Add a canary values.yaml. (033 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318671 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [09:04:47] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp2005.wikimedia.org [09:07:17] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host idp1005.wikimedia.org [09:11:14] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1005.wikimedia.org [09:11:40] (03PS1) 10JMeybohm: Update maintainer fields of core components to the k8s SIG [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319041 (https://phabricator.wikimedia.org/T373526) [09:11:46] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host idm1001.wikimedia.org [09:11:57] !log ayounsi@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-magru,cr2-magru IPv6,cr2-magru.mgmt with reason: router upgrade [09:12:04] 06SRE, 06Infrastructure-Foundations, 10netops: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166372 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cce20762-fa19-4ada-b944-1a93ea8ed59f) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi... [09:13:09] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [09:13:59] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [09:14:51] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [09:15:38] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [09:15:45] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm1001.wikimedia.org [09:17:17] !log drain cr1-magru - T431750 [09:17:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:17:21] T431750: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750 [09:18:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:21:06] !log reboot cr1-magru - T431750 [09:21:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:21:44] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [09:24:02] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 10/11 UP : OSPFv3: 10/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:24:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b4-magru:et-0/0/48 (Core: cr1-magru:et-0/0/2 {#70128}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b4-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:26:39] FIRING: [2x] CoreBGPDown: Core BGP session down between asw1-b4-magru and cr1-magru (195.200.68.148) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=asw1-b4-magru:9804&var-bgp_group=core&var-bgp_neighbor=cr1-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:28:17] (03CR) 10Klausman: [C:03+1] Add tts-section-generator CNAMEs to k8s-ingress-ml-serve [dns] - 10https://gerrit.wikimedia.org/r/1318685 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:28:37] (03CR) 10Elukey: [C:03+1] Update maintainer fields of core components to the k8s SIG [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1319032 (https://phabricator.wikimedia.org/T373526) (owner: 10JMeybohm) [09:29:16] (03CR) 10Elukey: [C:03+1] Update maintainer fields of core components to the k8s SIG [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319041 (https://phabricator.wikimedia.org/T373526) (owner: 10JMeybohm) [09:29:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1 {#70094}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:30:10] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:30:26] (03PS1) 10Bartosz Wójtowicz: ml-services: Request 1 GPU for gpt-oss-safeguard-20b. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319043 (https://phabricator.wikimedia.org/T432933) [09:30:55] (03PS1) 10Marostegui: eqiad.yaml: Fix hostname [puppet] - 10https://gerrit.wikimedia.org/r/1319044 (https://phabricator.wikimedia.org/T409557) [09:31:58] (03CR) 10JMeybohm: [C:03+2] Update maintainer fields of core components to the k8s SIG [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319041 (https://phabricator.wikimedia.org/T373526) (owner: 10JMeybohm) [09:32:02] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 11/11 UP : OSPFv3: 11/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:32:06] (03CR) 10Marostegui: [C:03+2] eqiad.yaml: Fix hostname [puppet] - 10https://gerrit.wikimedia.org/r/1319044 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [09:32:13] (03CR) 10JMeybohm: [V:03+2 C:03+2] Update maintainer fields of core components to the k8s SIG [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1319032 (https://phabricator.wikimedia.org/T373526) (owner: 10JMeybohm) [09:33:05] all done with cr1-magru, moving on to cr2 [09:33:17] !log marostegui@cumin1003 conftool action : set/weight=100; selector: name=clouddb1033.eqiad.wmnet,service=s5 [09:33:21] !log marostegui@cumin1003 conftool action : set/weight=100; selector: name=clouddb1033.eqiad.wmnet,service=s8 [09:33:27] !log marostegui@cumin1003 conftool action : set/pooled=yes; selector: name=clouddb1033.eqiad.wmnet,service=s5 [09:33:32] !log marostegui@cumin1003 conftool action : set/pooled=yes; selector: name=clouddb1033.eqiad.wmnet,service=s8 [09:33:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:34:12] (03PS1) 10Marostegui: clouddb1033: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1319045 (https://phabricator.wikimedia.org/T409557) [09:34:26] (03PS1) 10Btullis: Move analytics-test-hive and analytics-test-presto to new host [dns] - 10https://gerrit.wikimedia.org/r/1319046 (https://phabricator.wikimedia.org/T406746) [09:34:51] RESOLVED: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1 {#70094}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:34:57] (03CR) 10Marostegui: [C:03+2] clouddb1033: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1319045 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [09:35:10] RESOLVED: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:35:16] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1169: Maintenance [09:35:17] (03CR) 10JMeybohm: "Right. And you also have to configure CI to use the new image after it has been build (https://gerrit.wikimedia.org/r/plugins/gitiles/inte" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318569 (https://phabricator.wikimedia.org/T388390) (owner: 10Jelto) [09:36:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between asw1-b4-magru and cr1-magru (195.200.68.148) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=asw1-b4-magru:9804&var-bgp_group=core&var-bgp_neighbor=cr1-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:36:39] (03CR) 10Brouberol: [C:03+1] Move analytics-test-hive and analytics-test-presto to new host [dns] - 10https://gerrit.wikimedia.org/r/1319046 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [09:37:50] (03CR) 10Hnowlan: "Mostly seems good, but a few pitfalls/concerns, and a few nits which can be ignored if needs be." [puppet] - 10https://gerrit.wikimedia.org/r/1306792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [09:38:15] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1166: Maintenance [09:38:35] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance [09:38:43] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1175 (T431660)', diff saved to https://phabricator.wikimedia.org/P95479 and previous config saved to /var/cache/conftool/dbconfig/20260729-093842-cwilliams.json [09:38:49] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1182: Maintenance [09:39:10] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance [09:39:17] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1188 (T431660)', diff saved to https://phabricator.wikimedia.org/P95481 and previous config saved to /var/cache/conftool/dbconfig/20260729-093917-cwilliams.json [09:41:08] (03Merged) 10jenkins-bot: Update maintainer fields of core components to the k8s SIG [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319041 (https://phabricator.wikimedia.org/T373526) (owner: 10JMeybohm) [09:41:09] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1199: Maintenance [09:41:26] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet [09:41:29] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance [09:41:52] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance [09:42:00] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1221 (T431660)', diff saved to https://phabricator.wikimedia.org/P95483 and previous config saved to /var/cache/conftool/dbconfig/20260729-094200-cwilliams.json [09:42:28] (03CR) 10Btullis: [C:03+2] Move analytics-test-hive and analytics-test-presto to new host [dns] - 10https://gerrit.wikimedia.org/r/1319046 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [09:42:46] !log btullis@dns1004 START - running authdns-update [09:44:54] !log btullis@dns1004 END - running authdns-update [09:45:06] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1175: Maintenance [09:45:06] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1188: Maintenance [09:45:24] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet [09:46:21] (03PS1) 10CWilliams: profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1319029 (https://phabricator.wikimedia.org/T433344) [09:46:52] (03CR) 10Marostegui: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319029 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [09:48:21] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1221: Maintenance [09:48:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:50:19] 06SRE, 10Bitu, 06Infrastructure-Foundations: Move tracking of LDAP access to a new repository and manage it from Bitu - https://phabricator.wikimedia.org/T431111#12166505 (10LSobanski) p:05Triage→03High [09:50:36] 06SRE, 06Infrastructure-Foundations: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12166508 (10SLyngshede-WMF) p:05Triage→03Medium a:03SLyngshede-WMF [09:51:12] (03PS1) 10Btullis: hadoop-test: Update the database server used by hive and druid [puppet] - 10https://gerrit.wikimedia.org/r/1319049 (https://phabricator.wikimedia.org/T406746) [09:51:14] (03PS1) 10Bartosz Wójtowicz: ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 [09:52:08] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12166512 (10AndrewTavis_WMDE) [09:52:39] !log drain cr2-magru - T431750 [09:52:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:52:43] T431750: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750 [09:53:31] !log reboot cr2-magru - T431750 [09:53:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:54:16] (03CR) 10Brouberol: [C:03+1] hadoop-test: Update the database server used by hive and druid [puppet] - 10https://gerrit.wikimedia.org/r/1319049 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [09:56:06] PROBLEM - OSPF status on cr2-eqdfw is CRITICAL: OSPFv2: 6/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:56:23] (03CR) 10Btullis: [C:03+2] hadoop-test: Update the database server used by hive and druid [puppet] - 10https://gerrit.wikimedia.org/r/1319049 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [09:56:39] (03CR) 10Hnowlan: opensearch: add security-plugin specific configuration to opensearch.yml (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1318792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [09:56:39] (03PS2) 10Blake: httpbb: Add a --request-header argument. [software/httpbb] - 10https://gerrit.wikimedia.org/r/1318682 (https://phabricator.wikimedia.org/T428972) [09:56:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b4-magru:et-0/0/50 (Core: cr2-magru:et-0/0/2 {#70150}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b4-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:57:10] FIRING: [2x] BFDdown: BFD session down between cr2-eqdfw and 195.200.68.153 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:57:16] (03CR) 10Kevin Bazira: "Thanks for the patch. The gpt-oss-safeguard-20b isvc requires 2 GPUs to meet its latency targets. We did extensive performance tuning on t" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319043 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [09:59:51] (03PS2) 10Lucas Werkmeister (WMDE): wikidata-query-gui: update staging custom-config.json [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 [10:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1000) [10:00:18] PROBLEM - OSPF status on cr1-magru is CRITICAL: OSPFv2: 1/3 UP : OSPFv3: 1/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:00:32] (03PS1) 10Klausman: base: bump BPO Trixie kernel to 6.19 (only used by big ML GPU hosts atm) [puppet] - 10https://gerrit.wikimedia.org/r/1319051 (https://phabricator.wikimedia.org/T433485) [10:01:39] FIRING: [6x] CoreBGPDown: Core BGP session down between asw1-b4-magru and cr2-magru (195.200.68.144) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [10:01:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/50 (Core: cr2-magru:et-0/0/1 {#70130}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [10:02:10] FIRING: [4x] BFDdown: BFD session down between cr1-magru and 195.200.68.129 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:03:15] (03CR) 10Lucas Werkmeister (WMDE): "Thanks, that seems to work better. (I didn’t know which order “staging” and “wikidata-query-gui” should be in the file name, so thanks for" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 (owner: 10Lucas Werkmeister (WMDE)) [10:03:38] (03CR) 10Bartosz Wójtowicz: "Acknowledged" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319043 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [10:03:58] (03Abandoned) 10Bartosz Wójtowicz: ml-services: Request 1 GPU for gpt-oss-safeguard-20b. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319043 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [10:05:06] RECOVERY - OSPF status on cr2-eqdfw is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:05:18] RECOVERY - OSPF status on cr1-magru is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:06:39] RESOLVED: [6x] CoreBGPDown: Core BGP session down between asw1-b4-magru and cr2-magru (195.200.68.144) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [10:06:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/50 (Core: cr2-magru:et-0/0/1 {#70130}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [10:06:53] (03PS1) 10Filippo Giunchedi: hieradata: enable dumps-nfs.w.o usage [puppet] - 10https://gerrit.wikimedia.org/r/1319053 (https://phabricator.wikimedia.org/T432587) [10:06:58] FIRING: [2x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:07:06] RESOLVED: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/50 (Core: cr2-magru:et-0/0/1 {#70130}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [10:07:07] !ack [10:07:08] 8229 (ACKED) [2x] ProbeDown sre (text-https:443 probes/service magru) [10:07:10] RESOLVED: [4x] BFDdown: BFD session down between cr1-magru and 195.200.68.129 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:07:36] i imagine that's the drain? [10:08:02] (03CR) 10JMeybohm: [C:03+1] golang: add trixie-based golang-1.25 image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313867 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:08:06] XioNoX: ^ [10:09:19] https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes%2Fservice&var-module=$__all&orgId=1&from=now-6h&to=now&timezone=utc&var-site=$__all&var-Filters=&viewPanel=panel-2 [10:09:22] it seems to be magru yes [10:09:35] weird, I'd say transient monitoring packets got lots when cr2-magru came back up [10:09:59] should recover soon on its own [10:10:08] looks like it has, if i'm reading this right [10:10:18] (03PS31) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [10:10:35] (03CR) 10Ladsgroup: [C:03+2] Disallow all audio formats in the format filter [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1318810 (https://phabricator.wikimedia.org/T431671) (owner: 10Samwilson) [10:11:06] (03PS32) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [10:11:40] (03CR) 10Federico Ceratto: "Updated the names to avoid using "source" and "destination" anywhere" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [10:11:58] RESOLVED: [2x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:12:03] good [10:12:23] (03CR) 10Marostegui: "@cwilliams@wikimedia.org can you give this another review?" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [10:12:52] 06SRE, 10SRE-swift-storage, 05Goal, 06Machine-Learning-Team (Q1 FY2026-27), 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12166592 (10Ladsgroup) >>! In T432944#12163528, @MatthewVernon wrote: >... [10:13:50] Hi folks, would anyone be around in about an hour to help me with a database deployment to resolve https://phabricator.wikimedia.org/T433429 ? [10:14:02] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: pool magru [reason: router upgrade, T431750] [10:14:04] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: router upgrade, T431750] [10:14:06] T431750: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750 [10:14:28] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:14:28] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:14:31] HouseOfM: you'd need someone with deployment grants [10:14:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:14:42] 06SRE, 06Infrastructure-Foundations, 10netops: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166601 (10ayounsi) 05Open→03Resolved a:03ayounsi Done. [10:14:59] (03CR) 10David Caro: [C:03+1] "LGTM, emails were sent and such too, will just keep an eye on incoming tasks/irc if anyone did not read them." [puppet] - 10https://gerrit.wikimedia.org/r/1319053 (https://phabricator.wikimedia.org/T432587) (owner: 10Filippo Giunchedi) [10:15:19] @marostegui unfortunately I'm the only one in the team around right now, hence me begging in here for help :) [10:15:35] HouseOfM: :( [10:18:58] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:19:42] (03CR) 10Aklapper: "https://phabricator.wikimedia.org/T374926#12166520 implies that we could merge this. Should afterwards check what broke and whether the Di" [puppet] - 10https://gerrit.wikimedia.org/r/1298763 (https://phabricator.wikimedia.org/T405596) (owner: 10Aklapper) [10:21:04] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance [10:21:12] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1186 (T431660)', diff saved to https://phabricator.wikimedia.org/P95493 and previous config saved to /var/cache/conftool/dbconfig/20260729-102111-cwilliams.json [10:21:38] (03Merged) 10jenkins-bot: Disallow all audio formats in the format filter [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1318810 (https://phabricator.wikimedia.org/T431671) (owner: 10Samwilson) [10:27:30] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1186: Maintenance [10:31:18] (03CR) 10Jelto: "good point! I uploaded Id363603cc995cca2cc12ce66f58c471a39deb07a for the helm-linter version bump in the jjb CI definition." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318569 (https://phabricator.wikimedia.org/T388390) (owner: 10Jelto) [10:31:50] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1188: Maintenance [10:32:03] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12166624 (10AndrewTavis_WMDE) [10:32:10] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance [10:32:18] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1197 (T431660)', diff saved to https://phabricator.wikimedia.org/P95496 and previous config saved to /var/cache/conftool/dbconfig/20260729-103217-cwilliams.json [10:33:03] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1175: Maintenance [10:33:23] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance [10:33:32] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1198 (T431660)', diff saved to https://phabricator.wikimedia.org/P95499 and previous config saved to /var/cache/conftool/dbconfig/20260729-103330-cwilliams.json [10:33:37] (03PS1) 10Btullis: hadoop-test: Put the refreshed nodes into insetup mode prior to decom [puppet] - 10https://gerrit.wikimedia.org/r/1319059 (https://phabricator.wikimedia.org/T406746) [10:35:04] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1221: Maintenance [10:35:25] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance [10:35:33] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1238 (T431660)', diff saved to https://phabricator.wikimedia.org/P95501 and previous config saved to /var/cache/conftool/dbconfig/20260729-103532-cwilliams.json [10:36:54] (03CR) 10Brouberol: "Maybe gather an-test-master 1001 and 1002 into the same `node` declaration? Same for 1003 and 1004." [puppet] - 10https://gerrit.wikimedia.org/r/1319059 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [10:38:21] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1197: Maintenance [10:39:23] !log ran https://phabricator.wikimedia.org/T432509#12149723 in production (T432509) [10:39:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:39:27] T432509: transclude a project namespace page in main space page will view old revision - https://phabricator.wikimedia.org/T432509 [10:39:52] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1198: Maintenance [10:41:46] (03PS2) 10Jelto: golang: add trixie-based golang-1.25 image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313867 (https://phabricator.wikimedia.org/T427402) [10:41:57] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1238: Maintenance [10:44:23] (03PS1) 10Ozge: ml-services: Add jina-embeddings production isvc to llm namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319061 (https://phabricator.wikimedia.org/T432717) [10:44:45] (03CR) 10Ozge: "For production deployment." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319061 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:47:14] PROBLEM - Host ml-serve1015 is DOWN: PING CRITICAL - Packet loss = 100% [10:47:49] (03PS2) 10Btullis: hadoop-test: Put the refreshed nodes into insetup mode prior to decom [puppet] - 10https://gerrit.wikimedia.org/r/1319059 (https://phabricator.wikimedia.org/T406746) [10:48:48] (03PS1) 10Marostegui: Revert "db1197: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1319064 [10:49:35] (03CR) 10Marostegui: [C:03+2] Revert "db1197: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1319064 (owner: 10Marostegui) [10:49:42] RECOVERY - Host ml-serve1015 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [10:51:04] (03CR) 10Btullis: "Good call, thanks." [puppet] - 10https://gerrit.wikimedia.org/r/1319059 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [10:52:30] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Add jina-embeddings production isvc to llm namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319061 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:54:33] !log Dropping renamed tables T425066 [10:54:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:54:38] T425066: Drop AbuseFilter tables from closed wikis - https://phabricator.wikimedia.org/T425066 [10:57:16] (03CR) 10Elukey: [C:03+1] base: bump BPO Trixie kernel to 6.19 (only used by big ML GPU hosts atm) [puppet] - 10https://gerrit.wikimedia.org/r/1319051 (https://phabricator.wikimedia.org/T433485) (owner: 10Klausman) [10:57:48] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Add jina-embeddings production isvc to llm namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319061 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:59:55] (03PS2) 10Bartosz Wójtowicz: ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 [11:00:02] (03Merged) 10jenkins-bot: ml-services: Add jina-embeddings production isvc to llm namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319061 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [11:00:05] mvolz: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Services – Citoid / Zotero. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1100). [11:00:52] 06SRE, 06Infrastructure-Foundations, 10netops: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490 (10cmooney) 03NEW p:05Triage→03Medium [11:03:34] 06SRE, 06Infrastructure-Foundations, 10netops: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490#12166714 (10cmooney) Case 2026-0729-825879 created [11:05:44] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:07:15] (03Abandoned) 10Reedy: CirrusSearch-production: Mark 'CirrusSearch Streaming Updater' as a reserved username [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1300182 (https://phabricator.wikimedia.org/T428687) (owner: 10Reedy) [11:08:20] (03CR) 10Btullis: [C:03+2] hadoop-test: Put the refreshed nodes into insetup mode prior to decom [puppet] - 10https://gerrit.wikimedia.org/r/1319059 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [11:12:26] (03CR) 10Kevin Bazira: "besides the Merge Conflict, the patch LGTM!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 (owner: 10Bartosz Wójtowicz) [11:12:51] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [11:13:01] (03PS3) 10Bartosz Wójtowicz: ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 [11:13:25] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [11:14:12] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1186: Maintenance [11:14:43] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance [11:14:51] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1195 (T431660)', diff saved to https://phabricator.wikimedia.org/P95514 and previous config saved to /var/cache/conftool/dbconfig/20260729-111450-cwilliams.json [11:16:26] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 (owner: 10Bartosz Wójtowicz) [11:18:03] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 (owner: 10Bartosz Wójtowicz) [11:20:16] (03Merged) 10jenkins-bot: ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 (owner: 10Bartosz Wójtowicz) [11:21:16] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1195: Maintenance [11:22:00] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [11:23:51] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:26:37] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1198: Maintenance [11:26:58] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance [11:27:20] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [11:27:28] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1212 (T431660)', diff saved to https://phabricator.wikimedia.org/P95517 and previous config saved to /var/cache/conftool/dbconfig/20260729-112727-cwilliams.json [11:27:48] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [11:28:32] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [11:28:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:29:02] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1238: Maintenance [11:29:11] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance [11:29:19] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1241 (T431660)', diff saved to https://phabricator.wikimedia.org/P95520 and previous config saved to /var/cache/conftool/dbconfig/20260729-112918-cwilliams.json [11:29:45] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [11:30:11] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [11:32:13] (03CR) 10Filippo Giunchedi: "Out of curiosity which base images don't ship systemd ?" [debs/pint] - 10https://gerrit.wikimedia.org/r/1319039 (owner: 10Hnowlan) [11:32:14] (03PS1) 10Elukey: services: bump proton's image to pick up security updates [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319068 [11:33:27] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1212: Maintenance [11:35:43] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1241: Maintenance [11:36:50] (03PS1) 10Btullis: datahub-next: Update the MariaDB database to the refreshed host [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319070 (https://phabricator.wikimedia.org/T406746) [11:37:56] (03CR) 10Brouberol: [C:03+1] datahub-next: Update the MariaDB database to the refreshed host [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319070 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [11:43:34] (03CR) 10Btullis: [C:03+2] datahub-next: Update the MariaDB database to the refreshed host [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319070 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [11:45:42] (03Merged) 10jenkins-bot: datahub-next: Update the MariaDB database to the refreshed host [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319070 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [11:47:48] (03PS1) 10Elukey: profile::thanos: fix search sli's recording rule [puppet] - 10https://gerrit.wikimedia.org/r/1319071 [11:48:45] (03CR) 10Elukey: [C:03+2] services: bump proton's image to pick up security updates [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319068 (owner: 10Elukey) [11:49:19] !log elukey@deploy1003 helmfile [staging] START helmfile.d/services/proton: sync [11:50:06] (03CR) 10Elukey: [C:03+2] profile::thanos: fix search sli's recording rule [puppet] - 10https://gerrit.wikimedia.org/r/1319071 (owner: 10Elukey) [11:50:10] !log elukey@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: sync [11:50:10] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [11:50:44] !log btullis@cumin1003 START - Cookbook sre.hosts.decommission for hosts an-test-coord1001.eqiad.wmnet [11:51:26] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [11:51:43] !log elukey@deploy1003 helmfile [codfw] START helmfile.d/services/proton: sync [11:51:51] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [11:52:53] !log elukey@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: sync [11:54:13] (03CR) 10Audrey Penven: [C:03+1] "the diff for the current patch looks much more reasonable - affecting only the service we mean to change." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 (owner: 10Lucas Werkmeister (WMDE)) [11:55:35] (03CR) 10Jelto: [V:03+2 C:03+2] golang: add trixie-based golang-1.25 image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313867 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [11:55:39] !log btullis@cumin1003 START - Cookbook sre.dns.netbox [11:58:56] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1197: Maintenance [12:00:04] Daimona: Create new tables for the CampaignEvents extension (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1200). Please do the needful. [12:00:08] !log btullis@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-coord1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [12:00:09] !log Rename tables T425074 [12:00:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:00:15] T425074: Drop CheckUser tables from closed wikis - https://phabricator.wikimedia.org/T425074 [12:00:17] 06SRE, 10hCaptcha, 06Product Safety and Integrity, 06ServiceOps new, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12166872 (10JMeybohm) Hey @kostajh could you please elaborate what the impact of this is to the hCaptcha service and what you would... [12:00:59] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1098 hosts [12:03:12] btullis@cumin1003 decommission (PID 3619094) is awaiting input [12:04:17] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance [12:04:25] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1222 (T431660)', diff saved to https://phabricator.wikimedia.org/P95530 and previous config saved to /var/cache/conftool/dbconfig/20260729-120424-cwilliams.json [12:04:43] (03PS2) 10Federico Ceratto: mysql.multiinstance_reboot_test: switch to mocker [cookbooks] - 10https://gerrit.wikimedia.org/r/1314730 [12:04:52] (03CR) 10Klausman: [C:03+2] base: bump BPO Trixie kernel to 6.19 (only used by big ML GPU hosts atm) [puppet] - 10https://gerrit.wikimedia.org/r/1319051 (https://phabricator.wikimedia.org/T433485) (owner: 10Klausman) [12:05:36] (03CR) 10Federico Ceratto: "Rebased" [cookbooks] - 10https://gerrit.wikimedia.org/r/1314730 (owner: 10Federico Ceratto) [12:07:58] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1195: Maintenance [12:08:08] (03CR) 10CI reject: [V:04-1] mysql.multiinstance_reboot_test: switch to mocker [cookbooks] - 10https://gerrit.wikimedia.org/r/1314730 (owner: 10Federico Ceratto) [12:08:19] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance [12:08:40] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [12:08:48] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1196 (T431660)', diff saved to https://phabricator.wikimedia.org/P95533 and previous config saved to /var/cache/conftool/dbconfig/20260729-120847-cwilliams.json [12:11:39] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance [12:11:47] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1159 (T431660)', diff saved to https://phabricator.wikimedia.org/P95534 and previous config saved to /var/cache/conftool/dbconfig/20260729-121146-cwilliams.json [12:11:52] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance [12:12:04] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [12:12:12] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1158 (T431660)', diff saved to https://phabricator.wikimedia.org/P95535 and previous config saved to /var/cache/conftool/dbconfig/20260729-121211-cwilliams.json [12:12:37] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1222: Maintenance [12:14:42] !log Creating new DB tables for the CampaignEvents extension in x1.testwiki, x1.test2wiki, x1.officewiki, and x1.wikishared # T429339 [12:14:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:14:46] T429339: Create DB schema for storing worklists - https://phabricator.wikimedia.org/T429339 [12:15:25] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1196: Maintenance [12:15:51] !log elukey@deploy1003 helmfile [eqiad] START helmfile.d/services/proton: sync [12:17:03] !log elukey@deploy1003 helmfile [eqiad] DONE helmfile.d/services/proton: sync [12:17:29] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1159: Maintenance [12:19:02] I'm done with my tables. [12:19:10] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1212: Maintenance [12:19:29] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance [12:19:38] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1223 (T431660)', diff saved to https://phabricator.wikimedia.org/P95540 and previous config saved to /var/cache/conftool/dbconfig/20260729-121937-cwilliams.json [12:20:21] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1158: Maintenance [12:20:40] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 1051 hosts [12:22:26] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1241: Maintenance [12:22:46] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance [12:22:54] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1242 (T431660)', diff saved to https://phabricator.wikimedia.org/P95544 and previous config saved to /var/cache/conftool/dbconfig/20260729-122254-cwilliams.json [12:23:56] (03PS2) 10Anzx: remove throttle exceptions for concluded events [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319075 [12:25:42] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319075 (owner: 10Anzx) [12:25:45] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1223: Maintenance [12:28:16] 14SRE-Sprint-Week-Sustainability-March2023, 06Traffic, 13Patch-For-Review, 07Sustainability (Incident Followup): Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106#12166920 (10BBlack) Looks "reasonable"? The biggest diff is in the loss in revalidation hits (which makes se... [12:29:28] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1242: Maintenance [12:31:17] (03CR) 10Marostegui: [C:03+1] "The compiler seems to be fine: https://puppet-compiler.wmflabs.org/output/1319029/7392/" [puppet] - 10https://gerrit.wikimedia.org/r/1319029 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [12:32:52] !log klausman@cumin1003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve101[2-5].eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) [12:32:56] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1012.eqiad.wmnet [12:33:16] !log btullis@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-coord1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [12:33:16] !log btullis@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:33:17] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-test-coord1001.eqiad.wmnet [12:33:24] (03PS1) 10STran: SI: Instrument abuse filter hits link [extensions/CheckUser] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319077 (https://phabricator.wikimedia.org/T433053) [12:33:36] (03PS1) 10STran: SI: Instrument abuse filter hits link [extensions/CheckUser] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319078 (https://phabricator.wikimedia.org/T433053) [12:33:47] (03PS1) 10STran: SI: Instrument interaction timeline link [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319079 (https://phabricator.wikimedia.org/T433053) [12:33:57] (03PS1) 10STran: SI: Instrument interaction timeline link [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319080 (https://phabricator.wikimedia.org/T433053) [12:34:06] !log btullis@cumin1003 START - Cookbook sre.hosts.decommission for hosts an-test-master[1001-1002].eqiad.wmnet [12:34:12] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/CheckUser] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319077 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [12:34:20] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/CheckUser] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319078 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [12:34:27] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319079 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [12:34:34] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319080 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [12:35:30] (03CR) 10Ayounsi: [C:03+2] Add missing PTR include for private1-23-ulsfo and private1-604-eqsin (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1318769 (owner: 10Ayounsi) [12:35:53] !log ayounsi@dns1004 START - running authdns-update [12:37:59] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1012.eqiad.wmnet [12:38:03] !log ayounsi@dns1004 END - running authdns-update [12:38:05] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309675 (https://phabricator.wikimedia.org/T431830) (owner: 10Dreamrimmer) [12:39:12] (03CR) 10Ssingh: [V:03+1 C:03+2] service: add definition for urldownloader service [puppet] - 10https://gerrit.wikimedia.org/r/1313947 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [12:40:11] !log cwilliams@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for db-test2001.codfw.wmnet [12:40:52] !log cwilliams@cumin1003 END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test2001.codfw.wmnet [12:41:47] (03CR) 10CWilliams: [C:03+1] "The detection of a single replica is still working." [cookbooks] - 10https://gerrit.wikimedia.org/r/1315819 (https://phabricator.wikimedia.org/T420203) (owner: 10FNegri) [12:44:23] !log btullis@cumin1003 START - Cookbook sre.dns.netbox [12:45:01] !log sudo cumin 'O:url_downloader' 'disable-puppet "merging CR 1313948"': T429175 [12:45:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:45:04] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [12:45:12] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1012.eqiad.wmnet [12:45:13] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1012.eqiad.wmnet [12:45:21] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1013.eqiad.wmnet [12:46:56] (03CR) 10Ssingh: [V:03+1 C:03+2] hiera: url_downloader: enable LVS pool and IPIP [puppet] - 10https://gerrit.wikimedia.org/r/1313948 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [12:48:49] !log btullis@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-master[1001-1002].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [12:50:06] !log sudo cumin 'O:url_downloader' 'run-puppet-agent --enable "merging CR 1313948"': T429175 [12:50:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:50:10] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [12:50:27] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet [12:51:53] btullis@cumin1003 decommission (PID 3626581) is awaiting input [12:51:56] (03PS1) 10Ssingh: service: move urldownloaders to lvs_setup [puppet] - 10https://gerrit.wikimedia.org/r/1319082 (https://phabricator.wikimedia.org/T429175) [12:56:06] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet [12:56:07] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet [12:56:13] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet [12:57:06] (03CR) 10Slyngshede: [C:03+1] "Looks good, matches the documentation :-)" [puppet] - 10https://gerrit.wikimedia.org/r/1319082 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [12:57:13] (03PS1) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) [12:57:48] !log sudo cumin 'A:lvs and (A:eqiad or A:codfw)' 'disable-puppet "adding new service urldownloader"': T429175 [12:57:49] 10ops-codfw, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476#12167051 (10isarantopoulos) [12:57:50] FIRING: KubernetesCalicoDown: ml-serve1013.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-mlserve&var-instance=ml-serve1013.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:57:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:57:52] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [12:57:58] 10ops-codfw, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478#12167052 (10isarantopoulos) [12:58:18] (03CR) 10Ssingh: [C:03+2] service: move urldownloaders to lvs_setup [puppet] - 10https://gerrit.wikimedia.org/r/1319082 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [12:58:33] (03PS2) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) [12:58:40] (03CR) 10Jcrespo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [12:58:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [12:58:51] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12167058 (10isarantopoulos) [12:59:21] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Maintenance [12:59:43] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance [12:59:51] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1229 (T431660)', diff saved to https://phabricator.wikimedia.org/P95559 and previous config saved to /var/cache/conftool/dbconfig/20260729-125950-cwilliams.json [13:00:04] Lucas_WMDE, urbanecm, and TheresNoTime: UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1300). Please do the needful. [13:00:04] anzx, Tran, and DreamRimmer: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:09] o/ [13:00:17] o/ [13:00:24] o/ [13:00:28] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/kartotherian: apply [13:00:29] I’m in a meeting and can’t deploy, sorry [13:00:32] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/kartotherian: apply [13:01:17] I can self-deploy but am also in a meeting so am not sure I can help out after [13:01:19] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [13:01:22] !log sukhe@lvs1020:~$ sudo systemctl restart pybal.service [13:01:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:01:48] (⇒ https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/254) [13:02:24] Tran: could you start with yours self-deploying, and then I can do DreamRimmer and anzx 's if needed [13:02:31] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1196: Maintenance [13:02:50] RESOLVED: KubernetesCalicoDown: ml-serve1013.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-mlserve&var-instance=ml-serve1013.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:02:51] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance [13:02:59] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1206 (T431660)', diff saved to https://phabricator.wikimedia.org/P95562 and previous config saved to /var/cache/conftool/dbconfig/20260729-130258-cwilliams.json [13:03:17] (03CR) 10Jcrespo: [C:04-1] "Hey, Manuel, this is not in a rush, I still have to reimage these hosts, but want to check with you (review request) for a quick sanity ch" [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:03:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - urldownloader_8080: Servers urldownloader1005.wikimedia.org are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [13:04:26] TheresNoTime: I forgot keys rotated and haven't updated. Can you start? I'll self-deploy after. [13:04:36] (03PS3) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) [13:04:39] Okay! [13:04:40] (03CR) 10Jcrespo: [C:04-1] "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:04:43] anzx: starting with yours [13:05:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319075 (owner: 10Anzx) [13:05:15] (03PS4) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) [13:05:18] (03PS1) 10Bartosz Wójtowicz: ml-services: Add minReplicas=1 to gpt-oss and qwen36 deployments. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319085 (https://phabricator.wikimedia.org/T432933) [13:05:20] (03CR) 10Marostegui: "This looks good, just a minor non blocking comment." [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:05:26] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1159: Maintenance [13:05:31] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2073.codfw.wmnet with OS trixie [13:05:34] (03CR) 10Marostegui: [C:03+1] dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:05:42] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt IPs new switches - cmooney@cumin1003" [13:05:47] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance [13:05:47] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167092 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2073.codfw.wmnet with OS trixie [13:05:55] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1229: Maintenance [13:06:00] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1077.eqiad.wmnet with OS trixie [13:06:03] TheresNoTime: ok [13:06:08] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [13:06:14] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167094 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1077.eqiad.wmnet with OS trixie [13:06:17] nothing to test mine [13:06:17] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1161 (T431660)', diff saved to https://phabricator.wikimedia.org/P95564 and previous config saved to /var/cache/conftool/dbconfig/20260729-130616-cwilliams.json [13:06:19] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet [13:06:29] (03Merged) 10jenkins-bot: remove throttle exceptions for concluded events [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319075 (owner: 10Anzx) [13:06:36] anzx: (ack, will just run it straight through) [13:07:02] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1158: Maintenance [13:07:18] (03CR) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:07:18] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt IPs new switches - cmooney@cumin1003" [13:07:18] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:07:22] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [13:07:31] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1170 (T431660)', diff saved to https://phabricator.wikimedia.org/P95566 and previous config saved to /var/cache/conftool/dbconfig/20260729-130730-cwilliams.json [13:07:38] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1319075|remove throttle exceptions for concluded events]] [13:08:53] !log sukhe@lvs2014:~$ sudo systemctl restart pybal.service [13:08:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:09:07] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1206: Maintenance [13:09:17] (03PS1) 10Mpostoronca: WikimediaAntiAbuse: Document required load order after Echo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319086 (https://phabricator.wikimedia.org/T432452) [13:09:45] !log samtar@deploy1003 anzx, samtar: Backport for [[gerrit:1319075|remove throttle exceptions for concluded events]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:10:18] !log samtar@deploy1003 anzx, samtar: Continuing with deployment [13:12:13] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1161: Maintenance [13:12:50] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1223: Maintenance [13:13:21] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet [13:13:22] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet [13:13:27] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet [13:13:38] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1170: Maintenance [13:14:02] (03CR) 10Marostegui: [C:03+1] dbbackups: Setup db1265 & db1285 as new backup sources (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:14:38] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1319075|remove throttle exceptions for concluded events]] (duration: 07m 00s) [13:14:46] (03CR) 10BBlack: [C:03+1] "Try as I might, I can't find an actual issue with deleting the Cookie header here. It feels dangerous in the context of text, but I guess" [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [13:15:23] (03PS5) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) [13:15:25] anzx: live [13:15:39] TheresNoTime: thanks for deploying [13:16:02] noting `13:14:37 Job sudo -u mwdeploy -n -- /usr/bin/rsync -l deploy1003.eqiad.wmnet::common/wikiversions*.{json,php} /srv/mediawiki called with an empty host list.` in the scap log, not seen that before [13:16:05] (03CR) 10Jcrespo: "@Marostegui should be fixed now:" [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:16:11] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1242: Maintenance [13:16:17] DreamRimmer: ready? [13:16:26] yes [13:16:29] !log swfrench@cumin2002 START - Cookbook sre.hosts.reboot-single for host conf2005.codfw.wmnet [13:16:30] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1243.eqiad.wmnet with reason: Maintenance [13:16:38] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309675 (https://phabricator.wikimedia.org/T431830) (owner: 10Dreamrimmer) [13:16:39] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1243 (T431660)', diff saved to https://phabricator.wikimedia.org/P95574 and previous config saved to /var/cache/conftool/dbconfig/20260729-131638-cwilliams.json [13:16:52] (03CR) 10Jcrespo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:17:40] (03CR) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:17:51] (03Merged) 10jenkins-bot: Enable the abuse filter block action on Hindi Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309675 (https://phabricator.wikimedia.org/T431830) (owner: 10Dreamrimmer) [13:18:11] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1309675|Enable the abuse filter block action on Hindi Wikipedia (T431830)]] [13:18:15] T431830: Enable the abuse filter block action on Hindi Wikipedia - https://phabricator.wikimedia.org/T431830 [13:18:17] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for role: url_downloader@eqiad [13:18:19] (03PS1) 10Aqu: Add airflow.job_stats event stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319089 (https://phabricator.wikimedia.org/T405360) [13:18:29] !log sukhe@cumin1003 END (ERROR) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=97) for role: url_downloader@eqiad [13:18:34] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet [13:19:25] (03CR) 10Jcrespo: [C:04-1] "Blocked on reimage" [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:19:25] !log btullis@cumin1003 END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-master[1001-1002].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [13:19:25] !log btullis@cumin1003 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [13:19:26] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts an-test-master[1001-1002].eqiad.wmnet [13:19:37] (03CR) 10Dpogorzelski: "So what you mean is to drop the partman config and instead make a dedicated recipe where lvm extends on ml hosts?" [puppet] - 10https://gerrit.wikimedia.org/r/1307368 (https://phabricator.wikimedia.org/T431017) (owner: 10Dpogorzelski) [13:20:21] !log samtar@deploy1003 dreamrimmer, samtar: Backport for [[gerrit:1309675|Enable the abuse filter block action on Hindi Wikipedia (T431830)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:20:32] checking... [13:21:27] (ack) [13:21:58] good to go [13:22:03] (03CR) 10Scott French: [C:03+1] "Thanks, Blake!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318671 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [13:22:03] !log samtar@deploy1003 dreamrimmer, samtar: Continuing with deployment [13:22:04] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply [13:22:27] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf2005.codfw.wmnet [13:22:46] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply [13:23:04] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1243: Maintenance [13:23:05] (03CR) 10Blake: [C:03+2] mw-pretrain: Add a canary values.yaml. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318671 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [13:23:15] !log root@cumin1003 START - Cookbook sre.hosts.reimage for host db1265.eqiad.wmnet with OS trixie [13:23:35] !log root@cumin1003 START - Cookbook sre.hosts.move-vlan for host db1265 [13:23:35] !log root@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host db1265 [13:24:30] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1077.eqiad.wmnet with reason: host reimage [13:24:34] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet [13:24:36] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet [13:24:36] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve101[2-5].eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) [13:24:42] 06SRE, 10SRE-swift-storage, 05Goal, 06Machine-Learning-Team (Q1 FY2026-27), 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12167121 (10Ottomata) Hello! Some (no-stupid! :) )questions about the us... [13:24:46] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for role: url_downloader@eqiad [13:24:47] (03PS2) 10Jelto: Update cert-manager to 1.19.6 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313866 (https://phabricator.wikimedia.org/T427402) [13:25:09] !log klausman@cumin1003 START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet [13:25:25] (03Merged) 10jenkins-bot: mw-pretrain: Add a canary values.yaml. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318671 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [13:26:08] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1309675|Enable the abuse filter block action on Hindi Wikipedia (T431830)]] (duration: 07m 56s) [13:26:11] (03CR) 10Jelto: [V:03+1] "rebased and build verified locally:" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313866 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [13:26:11] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2073.codfw.wmnet with reason: host reimage [13:26:13] T431830: Enable the abuse filter block action on Hindi Wikipedia - https://phabricator.wikimedia.org/T431830 [13:26:21] DreamRimmer: live [13:26:48] thanks so much :) [13:26:55] 06SRE, 10SRE-swift-storage, 05Goal, 06Machine-Learning-Team (Q1 FY2026-27), 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12167133 (10Ottomata) ...Speaking of 'renderings', you might want to con... [13:27:01] Tran: all yours :) [13:27:07] Thanks! Starting [13:27:09] !log sukhe@cumin1003 END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: url_downloader@eqiad [13:27:33] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [13:27:42] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for role: url_downloader@eqiad [13:27:56] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319079 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [13:27:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319080 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [13:27:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy1003 using scap backport" [extensions/CheckUser] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319077 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [13:27:58] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy1003 using scap backport" [extensions/CheckUser] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319078 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [13:28:08] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1077.eqiad.wmnet with reason: host reimage [13:29:10] !log sukhe@cumin1003 END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: url_downloader@eqiad [13:30:08] (03Merged) 10jenkins-bot: SI: Instrument interaction timeline link [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319079 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [13:30:10] (03Merged) 10jenkins-bot: SI: Instrument interaction timeline link [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319080 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [13:30:12] (03Merged) 10jenkins-bot: SI: Instrument abuse filter hits link [extensions/CheckUser] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319077 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [13:30:15] (03Merged) 10jenkins-bot: SI: Instrument abuse filter hits link [extensions/CheckUser] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319078 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [13:30:35] 06SRE, 10SRE-swift-storage, 05Goal, 06Machine-Learning-Team (Q1 FY2026-27), 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12167146 (10Ottomata) Actually, these comments ^^ would be better placed... [13:30:38] !log stran@deploy1003 Started scap sync-world: Backport for [[gerrit:1319079|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319080|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319077|SI: Instrument abuse filter hits link (T433053)]], [[gerrit:1319078|SI: Instrument abuse filter hits link (T433053)]] [13:30:42] T433053: Add missing pieces of instrumentation - https://phabricator.wikimedia.org/T433053 [13:30:52] !log klausman@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet [13:32:00] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2073.codfw.wmnet with reason: host reimage [13:32:09] PROBLEM - PyBal IPVS diff check on lvs1019 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [13:32:36] !log stran@deploy1003 stran: Backport for [[gerrit:1319079|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319080|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319077|SI: Instrument abuse filter hits link (T433053)]], [[gerrit:1319078|SI: Instrument abuse filter hits link (T433053)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified t [13:32:37] here. [13:33:24] testing now [13:33:41] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/kartotherian: apply [13:34:15] FIRING: HttpdUnreachable: httpd unavailable for deployment mw-pretrain/canary at codfw - https://wikitech.wikimedia.org/wiki/Application_servers - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=257&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-pretrain&var-release=canary - https://alerts.wikimedia.org/?q=alertname%3DHttpdUnreachable [13:34:46] (03CR) 10Hashar: [C:03+1] "I think it can be tried. Given Puppet is not going to automagically remove the resources that will then be undefined." [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [13:34:59] PROBLEM - PyBal connections to etcd on lvs1019 is CRITICAL: CRITICAL: 79 connections established with conf1007.eqiad.wmnet:4001 (min=80) https://wikitech.wikimedia.org/wiki/PyBal [13:35:56] looks good, continuing [13:35:58] !log stran@deploy1003 stran: Continuing with deployment [13:36:31] !log root@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage [13:38:17] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [13:39:05] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12167187 (10cmooney) [13:39:44] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [13:40:00] !log stran@deploy1003 Finished scap sync-world: Backport for [[gerrit:1319079|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319080|SI: Instrument interaction timeline link (T433053)]], [[gerrit:1319077|SI: Instrument abuse filter hits link (T433053)]], [[gerrit:1319078|SI: Instrument abuse filter hits link (T433053)]] (duration: 09m 22s) [13:40:04] T433053: Add missing pieces of instrumentation - https://phabricator.wikimedia.org/T433053 [13:40:11] done [13:44:02] !log root@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1265.eqiad.wmnet with reason: host reimage [13:44:18] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/kartotherian: apply [13:46:28] (03PS2) 10CWilliams: sre.mysql: add MySQL-specific base classes [cookbooks] - 10https://gerrit.wikimedia.org/r/1319088 (https://phabricator.wikimedia.org/T433497) [13:46:49] !log swfrench@cumin2002 START - Cookbook sre.hosts.reimage for host conf2005.codfw.wmnet with OS bookworm [13:47:00] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/kartotherian: apply [13:47:08] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1077.eqiad.wmnet with OS trixie [13:47:16] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167235 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1077.eqiad.wmnet with OS trixie completed: - ms-be1077 (**PASS*... [13:47:52] (03PS1) 10Bking: opensearch: Dynamically update Java security policy based on ECS opt-in [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) [13:48:08] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [13:48:16] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/kartotherian: apply [13:48:30] (03PS2) 10Bking: opensearch: Dynamically update Java security policy based on ECS opt-in [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) [13:49:18] !log jgiannelos@deploy1003 helmfile [eqiad] START helmfile.d/services/kartotherian: apply [13:49:54] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [13:49:59] RECOVERY - PyBal connections to etcd on lvs1019 is OK: OK: 80 connections established with conf1007.eqiad.wmnet:4001 (min=80) https://wikitech.wikimedia.org/wiki/PyBal [13:50:09] RECOVERY - NTP peers and stratum check on dns3003 is OK: NTP OK: Offset 1.4278e-05 secs, stratum=1 https://wikitech.wikimedia.org/wiki/NTP [13:50:09] RECOVERY - Check if ntpsec.service has been restarted after /etc/ntpsec/ntp.conf was changed on dns3003 is OK: OK: ntpsec.service was restarted after /etc/ntpsec/ntp.conf was changed. https://wikitech.wikimedia.org/wiki/NTP%23Monitoring [13:50:23] (03PS3) 10Bking: opensearch: Dynamically update Java security policy based on ECS opt-in [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) [13:50:26] (03PS1) 10Ssingh: service: move urldownloader to production [puppet] - 10https://gerrit.wikimedia.org/r/1319096 (https://phabricator.wikimedia.org/T429175) [13:50:32] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [13:50:33] PROBLEM - PyBal IPVS diff check on lvs2013 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [13:50:42] !log sukhe@lvs2013:~$ sudo systemctl restart pybal.service [13:50:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:50:45] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - urldownloader_8080: Servers urldownloader1005.wikimedia.org are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [13:50:46] !log jgiannelos@deploy1003 helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply [13:50:50] yes, known [13:51:16] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12167264 (10klausman) [13:51:25] FIRING: SystemdUnitFailed: prometheus_amd_rocm_stats.service on ml-serve1015:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:52:07] RECOVERY - PyBal IPVS diff check on lvs1019 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [13:52:43] RECOVERY - PyBal IPVS diff check on lvs2013 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [13:52:51] (03CR) 10Ssingh: [C:03+2] service: move urldownloader to production [puppet] - 10https://gerrit.wikimedia.org/r/1319096 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [13:52:52] (03CR) 10CI reject: [V:04-1] opensearch: Dynamically update Java security policy based on ECS opt-in [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [13:53:07] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1229: Maintenance [13:53:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:53:28] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance [13:53:36] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1233 (T431660)', diff saved to https://phabricator.wikimedia.org/P95587 and previous config saved to /var/cache/conftool/dbconfig/20260729-135335-cwilliams.json [13:54:00] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2073.codfw.wmnet with OS trixie [13:54:09] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167276 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2073.codfw.wmnet with OS trixie completed: - ms-be2073 (**PASS*... [13:54:15] RESOLVED: HttpdUnreachable: httpd unavailable for deployment mw-pretrain/canary at codfw - https://wikitech.wikimedia.org/wiki/Application_servers - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=257&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-pretrain&var-release=canary - https://alerts.wikimedia.org/?q=alertname%3DHttpdUnreachable [13:55:52] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1206: Maintenance [13:56:13] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance [13:56:21] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1218 (T431660)', diff saved to https://phabricator.wikimedia.org/P95590 and previous config saved to /var/cache/conftool/dbconfig/20260729-135621-cwilliams.json [13:57:19] (03PS4) 10Bking: opensearch: Dynamically update Java security policy based on ECS opt-in [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) [13:57:51] (03CR) 10Ssingh: [C:03+2] wmnet: add disc-urldownloader A/A record [dns] - 10https://gerrit.wikimedia.org/r/1313950 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [13:58:04] (03PS1) 10Blake: kubernetes: Add a canary deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1319097 (https://phabricator.wikimedia.org/T427668) [13:58:20] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [13:58:21] !log root@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1265.eqiad.wmnet with OS trixie [13:58:26] !log sukhe@puppetserver1001 conftool action : set/pooled=true; selector: dnsdisc=urldownloader [13:58:58] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Maintenance [13:59:18] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance [13:59:26] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1185 (T431660)', diff saved to https://phabricator.wikimedia.org/P95592 and previous config saved to /var/cache/conftool/dbconfig/20260729-135925-cwilliams.json [13:59:53] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1233: Maintenance [14:00:04] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1400) [14:00:55] !log sukhe@dns1004 START - running authdns-update [14:01:01] !log sukhe@dns1004 START - running authdns-update [14:01:03] (03CR) 10Scott French: [C:03+1] kubernetes: Add a canary deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1319097 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [14:02:29] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1218: Maintenance [14:02:40] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1170: Maintenance [14:03:02] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance [14:03:08] !log sukhe@dns1004 END - running authdns-update [14:03:10] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1174 (T431660)', diff saved to https://phabricator.wikimedia.org/P95595 and previous config saved to /var/cache/conftool/dbconfig/20260729-140309-cwilliams.json [14:04:46] (03PS2) 10CWilliams: profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1319029 (https://phabricator.wikimedia.org/T433344) [14:05:51] !log swfrench@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on conf2005.codfw.wmnet with reason: host reimage [14:06:00] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for role: url_downloader@eqiad [14:06:16] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-07-21-080322 to 2026-07-27-212343 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319098 (https://phabricator.wikimedia.org/T430145) [14:06:18] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-07-21-162301 to 2026-07-29-132540 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319099 (https://phabricator.wikimedia.org/T373413) [14:06:27] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1185: Maintenance [14:08:22] !log sukhe@cumin1003 END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: url_downloader@eqiad [14:09:20] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1174: Maintenance [14:09:41] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2005.codfw.wmnet with reason: host reimage [14:09:48] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1243: Maintenance [14:10:00] !log cwilliams@cumin1003 START - Cookbook sre.mysql.update-replication [14:10:07] !log pt1979@cumin2002 START - Cookbook sre.dns.netbox [14:10:07] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance [14:10:07] (03PS5) 10Bking: opensearch: Dynamically update Java security policy based on ECS opt-in [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) [14:10:07] (03PS8) 10Bking: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [14:10:08] !log cwilliams@cumin1003 END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=1) [14:10:15] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1244 (T431660)', diff saved to https://phabricator.wikimedia.org/P95599 and previous config saved to /var/cache/conftool/dbconfig/20260729-141014-cwilliams.json [14:11:20] 06SRE, 06ServiceOps new, 10VisualEditor, 10VisualEditor Suggestion Mode, and 3 others: New Service Request: Headless VE - https://phabricator.wikimedia.org/T431497#12167389 (10Ottomata) So! IIUC, this is a bit of a chicken and egg problem! Yes, Editing would like to deploy Headless VE. But, ServiceOps i... [14:11:32] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12167393 (10Papaul) [14:12:43] !log cwilliams@cumin1003 START - Cookbook sre.mysql.update-replication [14:12:51] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) [14:13:44] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12167398 (10VRiley-WMF) 05Open→03In progress Dell is onsite and proceeding with the mainboard replacment. [14:14:28] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:14:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:15:24] !log pt1979@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add asw1-604 loopback ipv4 - pt1979@cumin2002" [14:15:29] !log pt1979@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add asw1-604 loopback ipv4 - pt1979@cumin2002" [14:15:29] !log pt1979@cumin2002 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:16:46] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1244: Maintenance [14:16:50] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade evaluators from 2026-07-21-080322 to 2026-07-27-212343 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319098 (https://phabricator.wikimedia.org/T430145) (owner: 10Jforrester) [14:17:36] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:18:04] (03PS7) 10Btullis: cirrus: produce ECS server logs on cloudelastic [puppet] - 10https://gerrit.wikimedia.org/r/1311441 (https://phabricator.wikimedia.org/T324335) [14:18:11] (03PS7) 10Btullis: cirrus: produce ECS server logs on the production clusters [puppet] - 10https://gerrit.wikimedia.org/r/1311439 (https://phabricator.wikimedia.org/T324335) [14:18:19] (03CR) 10CWilliams: [C:03+2] profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1319029 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [14:19:37] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-07-21-080322 to 2026-07-27-212343 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319098 (https://phabricator.wikimedia.org/T430145) (owner: 10Jforrester) [14:21:06] (03PS6) 10Bking: opensearch: Dynamically update Java security policy based on ECS opt-in [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) [14:21:07] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:21:59] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:22:26] !log jforrester@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:23:43] !log jforrester@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:23:48] (03CR) 10Ahmon Dancy: "That's right. As it stands, the message blob purge is happening up to 3 times; once per rebuildLocalisationCache.php per live train branch" [puppet] - 10https://gerrit.wikimedia.org/r/1318302 (https://phabricator.wikimedia.org/T263872) (owner: 10Ahmon Dancy) [14:24:18] !log jforrester@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:24:35] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:24:49] !log jforrester@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:24:55] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [14:25:10] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-07-21-162301 to 2026-07-29-132540 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319099 (https://phabricator.wikimedia.org/T373413) (owner: 10Jforrester) [14:25:11] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12167456 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS t... [14:25:39] 06SRE, 10Maps, 07affects-Kiwix-and-openZIM, 06Content-Platform-Team (Work In Progress), 07Essential-Work: Wikipedia wikis have broken maps URLs in infobox: "Bad GeoJSON - unknown \"type\" property \"ExternalData\"" - https://phabricator.wikimedia.org/T424046#12167461 (10Jgiannelos) 05Open→03Resolv... [14:26:03] (03PS5) 10Trueg: WDQSv2: Scheduling based on resource requests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318733 (https://phabricator.wikimedia.org/T431395) [14:27:20] !log root@cumin1003 START - Cookbook sre.hosts.reimage for host db1285.eqiad.wmnet with OS trixie [14:27:31] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-07-21-162301 to 2026-07-29-132540 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319099 (https://phabricator.wikimedia.org/T373413) (owner: 10Jforrester) [14:27:42] !log root@cumin1003 START - Cookbook sre.hosts.move-vlan for host db1285 [14:27:42] !log root@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host db1285 [14:28:04] (03CR) 10CWilliams: "There is testing output on the ticket. It seemed to behave as expected, although testing was not clean due to a yet-to-be-determined reaso" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [14:29:32] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:29:57] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:30:04] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1400) [14:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1430) [14:30:12] !log jforrester@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:30:59] !log jforrester@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:31:07] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2005.codfw.wmnet with OS bookworm [14:32:13] !log jforrester@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:32:25] (03CR) 10Ebernhardson: [C:03+1] "looks reasonable, one question" [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [14:32:55] !log jforrester@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:32:58] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:37:50] (03PS1) 10Ssingh: O:url_downloader: add LVS/IPIP profiles [puppet] - 10https://gerrit.wikimedia.org/r/1319102 (https://phabricator.wikimedia.org/T429175) [14:38:09] (03CR) 10Blake: [C:03+2] kubernetes: Add a canary deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1319097 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [14:39:16] (03CR) 10Slyngshede: [C:03+1] O:url_downloader: add LVS/IPIP profiles [puppet] - 10https://gerrit.wikimedia.org/r/1319102 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [14:39:21] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2074.codfw.wmnet with OS trixie [14:39:29] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167537 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2074.codfw.wmnet with OS trixie [14:39:31] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1078.eqiad.wmnet with OS trixie [14:39:39] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167539 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1078.eqiad.wmnet with OS trixie [14:40:27] (03CR) 10Ssingh: [C:03+2] O:url_downloader: add LVS/IPIP profiles [puppet] - 10https://gerrit.wikimedia.org/r/1319102 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [14:40:59] !log root@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db1285.eqiad.wmnet with reason: host reimage [14:41:47] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:41:50] (03CR) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [14:42:13] (03CR) 10Jcrespo: "This is ready, but the intention is to deploy tomorrow." [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [14:42:32] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:43:05] !log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:44:28] !log root@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1285.eqiad.wmnet with reason: host reimage [14:45:54] !log jhancock@cumin2002 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:46:15] !log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:46:16] PROBLEM - Host ml-serve1015 is DOWN: PING CRITICAL - Packet loss = 100% [14:46:25] RESOLVED: SystemdUnitFailed: prometheus_amd_rocm_stats.service on ml-serve1015:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:46:48] !log bking@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs2014\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main [14:46:54] !log bking@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs2013\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main [14:47:00] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1233: Maintenance [14:47:22] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance [14:47:30] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1254 (T431660)', diff saved to https://phabricator.wikimedia.org/P95613 and previous config saved to /var/cache/conftool/dbconfig/20260729-144729-cwilliams.json [14:48:20] !log dancy@deploy1003 Started deploy [zuul/deploy@22703a6]: Deploying https://gerrit.wikimedia.org/r/c/integration/zuul/+/1311501 (T432491) [14:48:24] T432491: Restrict depth of stacked commits processing in Zuul - https://phabricator.wikimedia.org/T432491 [14:48:26] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2015.codfw.wmnet with OS bookworm [14:48:35] !log dancy@deploy1003 Finished deploy [zuul/deploy@22703a6]: Deploying https://gerrit.wikimedia.org/r/c/integration/zuul/+/1311501 (T432491) (duration: 00m 15s) [14:48:48] RECOVERY - Host ml-serve1015 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [14:48:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [14:48:50] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2021.codfw.wmnet with OS bookworm [14:48:51] (03CR) 10Hnowlan: "I discovered this with our base `docker-registry.wikimedia.org/bookworm:latest` image" [debs/pint] - 10https://gerrit.wikimedia.org/r/1319039 (owner: 10Hnowlan) [14:49:11] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Maintenance [14:49:32] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance [14:49:33] jhancock@cumin2002 provision (PID 2294220) is awaiting input [14:49:47] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1219 (T431660)', diff saved to https://phabricator.wikimedia.org/P95616 and previous config saved to /var/cache/conftool/dbconfig/20260729-144946-cwilliams.json [14:50:50] FIRING: KubernetesCalicoDown: ml-serve1015.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-mlserve&var-instance=ml-serve1015.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:51:47] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2015 [14:51:47] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2015 [14:52:18] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2021 [14:52:47] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12167619 (10Jhancock.wm) [14:53:09] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1185: Maintenance [14:53:29] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance [14:53:29] !log bking@cumin2003 START - Cookbook sre.dns.netbox [14:53:35] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1254: Maintenance [14:53:37] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1200 (T431660)', diff saved to https://phabricator.wikimedia.org/P95618 and previous config saved to /var/cache/conftool/dbconfig/20260729-145336-cwilliams.json [14:54:17] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12167629 (10jcrespo) One request, @VRiley-WMF if all goes well and it goes back to "normal" (bootable state), could you leave the host shutted down for the night (just powerable through remot... [14:54:41] (03CR) 10Bking: opensearch: Dynamically update Java security policy based on ECS opt-in (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [14:55:06] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12167633 (10VRiley-WMF) Of course @jcrespo I will make sure I do that. Thanks! [14:55:32] !log sukhe@cumin1003 START - Cookbook sre.dns.wipe-cache url-downloader1005.wikimedia.org on all recursors [14:55:36] !log sukhe@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) url-downloader1005.wikimedia.org on all recursors [14:55:38] !log sukhe@cumin1003 START - Cookbook sre.dns.wipe-cache url-downloader1006.wikimedia.org on all recursors [14:55:42] !log sukhe@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) url-downloader1006.wikimedia.org on all recursors [14:55:50] RESOLVED: KubernetesCalicoDown: ml-serve1015.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-mlserve&var-instance=ml-serve1015.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:55:56] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1219: Maintenance [14:56:01] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1174: Maintenance [14:56:22] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance [14:56:30] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1181 (T431660)', diff saved to https://phabricator.wikimedia.org/P95621 and previous config saved to /var/cache/conftool/dbconfig/20260729-145629-cwilliams.json [14:57:03] jhancock@cumin2002 provision (PID 2294220) is awaiting input [14:58:04] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1078.eqiad.wmnet with reason: host reimage [14:58:07] !log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:58:28] !log jhancock@cumin2002 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be2097'] [14:58:34] !log jhancock@cumin2002 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ms-be2097'] [14:58:42] !log jhancock@cumin2002 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be2098'] [14:58:47] !log jhancock@cumin2002 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ms-be2098'] [14:58:52] !log sukhe@cumin1003 START - Cookbook sre.hosts.reboot-single for host urldownloader1005.wikimedia.org [14:59:10] !log mwscript-k8s -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --wiki=commonswiki --key '360p.mpeg4.mov' --throttle --video --missing (T358266) [14:59:14] !log root@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1285.eqiad.wmnet with OS trixie [14:59:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:59:15] T358266: TimedMediaHandler: Improve video quality on older iPhones (MPEG-4 Visual) - https://phabricator.wikimedia.org/T358266 [14:59:32] bking@cumin2003 reimage (PID 1418933) is awaiting input [14:59:32] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2074.codfw.wmnet with reason: host reimage [14:59:51] (03CR) 10Btullis: [C:04-1] opensearch: Dynamically update Java security policy based on ECS opt-in (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [15:00:04] Lucas_WMDE: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for T231755 deployment . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1500). [15:00:05] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [15:00:14] hiii [15:00:25] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1200: Maintenance [15:00:57] !log jhancock@cumin2002 START - Cookbook sre.hosts.reimage for host ms-be2097.codfw.wmnet with OS bullseye [15:01:10] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12167670 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2097.codfw.wmnet with OS bullseye [15:01:27] !log jhancock@cumin2002 START - Cookbook sre.hosts.reimage for host ms-be2098.codfw.wmnet with OS bullseye [15:01:37] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12167681 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2098.codfw.wmnet with OS bullseye [15:01:50] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318704 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [15:02:21] (03CR) 10Bking: opensearch: Dynamically update Java security policy based on ECS opt-in (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1319095 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [15:02:43] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1181: Maintenance [15:03:18] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1005.wikimedia.org [15:04:37] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1078.eqiad.wmnet with reason: host reimage [15:04:42] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1244: Maintenance [15:04:47] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [15:04:51] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance [15:04:59] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1247 (T431660)', diff saved to https://phabricator.wikimedia.org/P95625 and previous config saved to /var/cache/conftool/dbconfig/20260729-150459-cwilliams.json [15:08:02] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2074.codfw.wmnet with reason: host reimage [15:09:30] I’ve started tracking results for my deployment window in https://www.wikidata.org/wiki/User:Lucas_Werkmeister/translatable_language_names_performance_(T231755) [15:09:31] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [15:10:41] !log klausman@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on ml-serve2002.codfw.wmnet with reason: T433476 [15:10:46] T433476: hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476 [15:10:47] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2015.codfw.wmnet with reason: host reimage [15:11:17] !log klausman@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on ml-serve2004.codfw.wmnet with reason: T433478 [15:11:21] T433478: hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478 [15:11:33] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1247: Maintenance [15:14:12] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2015.codfw.wmnet with reason: host reimage [15:14:25] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:14:26] (03Merged) 10jenkins-bot: Add i18n/LanguageNames to MessagesDirs [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318704 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [15:14:46] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1318704|Add i18n/LanguageNames to MessagesDirs (T231755)]] [15:14:50] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [15:15:07] (this scap will take a hot second, because l10n cache) [15:15:32] (shouldn’t scap/spiderpig have asked me about that? 🤔) [15:17:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:18:18] “0 languages rebuilt out of 549” o_O [15:18:34] oh wait, that was wmf.12. wmf.13 did rebuild 549 languages [15:19:00] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2021 - bking@cumin2003" [15:19:04] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2021 - bking@cumin2003" [15:19:04] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:19:05] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs2021.codfw.wmnet 210.48.192.10.in-addr.arpa 0.1.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [15:19:08] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2021.codfw.wmnet 210.48.192.10.in-addr.arpa 0.1.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [15:19:09] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs2021 [15:19:39] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2021 [15:19:39] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2021 [15:20:50] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for iberker - https://phabricator.wikimedia.org/T433258#12167769 (10jijiki) a:03IBerker-WMF @IBerker-WMF is this request solely for accessing the superset dashboards ? if that is so, adding yourself to the 'wmf' ldap group should... [15:21:17] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12167772 (10leila) @Bethany happy to help here. A couple of questions: - Can you confirm that you have read https://phabricator.wikimedia.org/L3 and agree to its terms... [15:21:47] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12167773 (10leila) [15:22:14] !log swfrench@cumin2002 START - Cookbook sre.hosts.reboot-single for host conf2006.codfw.wmnet [15:22:32] Lucas_WMDE: In the original implementation that warned of possible l10n rebuild, I had extension.json in the list of triggers.. but later someone said that it was not appropriate (https://phabricator.wikimedia.org/T430750) so I removed it. Clearly there needs to be some additional logic to handle the extension.json situation better. [15:22:53] I see, thanks [15:23:19] (yeah I got bit by a few false positives of extension.json too, but this time it would have been a true positive ^^) [15:24:14] I guess to determine that with certainty, you need a diff of extension.json (or really, full JSON before and after, so you can parse it), which is more annoying to get [15:24:55] Nod. A bit more work, but not a big deal. I guess it could look for file diffs that mention i18n [15:25:32] (or split the condition into “likely” and “possibly” with separate messages, perhaps… easier to implement, slightly less helpful for deployers ^^) [15:25:36] !log jhancock@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage [15:25:48] (03PS1) 10Elukey: profile::thanos: update search-lag's recording rules [puppet] - 10https://gerrit.wikimedia.org/r/1319110 [15:25:50] !log jhancock@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2098.codfw.wmnet with reason: host reimage [15:25:50] anyway in this case it was fine, I was just wondering because I’d seen the warning before. now I know [15:25:59] Nod [15:26:02] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1078.eqiad.wmnet with OS trixie [15:26:09] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167780 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1078.eqiad.wmnet with OS trixie completed: - ms-be1078 (**PASS*... [15:26:35] (03CR) 10RLazarus: [C:03+1] profile::thanos: update search-lag's recording rules [puppet] - 10https://gerrit.wikimedia.org/r/1319110 (owner: 10Elukey) [15:28:16] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf2006.codfw.wmnet [15:28:36] !log mvernon@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host ms-be2074.codfw.wmnet with OS trixie [15:28:42] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167789 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2074.codfw.wmnet with OS trixie completed: - ms-be2074 (**FAIL*... [15:28:45] (03PS1) 10Elukey: docker_registry: add s3cmd and skopeo packages [puppet] - 10https://gerrit.wikimedia.org/r/1319112 (https://phabricator.wikimedia.org/T427175) [15:28:49] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167790 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2074.codfw.wmnet with OS trixie executed with errors: - ms-be20... [15:28:51] (03CR) 10Elukey: [C:03+2] profile::thanos: update search-lag's recording rules [puppet] - 10https://gerrit.wikimedia.org/r/1319110 (owner: 10Elukey) [15:29:05] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319112 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [15:29:22] !log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage [15:29:29] (03CR) 10CI reject: [V:04-1] docker_registry: add s3cmd and skopeo packages [puppet] - 10https://gerrit.wikimedia.org/r/1319112 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [15:32:19] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12167806 (10jijiki) @CDiggs-WMF let us know if you have worked things out :) [15:32:52] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde: Backport for [[gerrit:1318704|Add i18n/LanguageNames to MessagesDirs (T231755)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:32:57] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [15:33:16] ok, testing [15:33:28] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12167815 (10VRiley-WMF) Dell has come on site and replaced the mainboard. I have logged the new MAC addresses (if needed) into Netbox. I am collecting a TSR report for Dell and then will powe... [15:33:30] !log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2098.codfw.wmnet with reason: host reimage [15:33:33] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2074.codfw.wmnet with OS trixie [15:33:34] (03PS2) 10Elukey: docker_registry: add s3cmd and skopeo packages [puppet] - 10https://gerrit.wikimedia.org/r/1319112 (https://phabricator.wikimedia.org/T427175) [15:33:56] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167820 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2074.codfw.wmnet with OS trixie [15:34:08] (03CR) 10BCornwall: [C:03+1] Add tts-section-generator CNAMEs to k8s-ingress-ml-serve [dns] - 10https://gerrit.wikimedia.org/r/1318685 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [15:34:32] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319112 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [15:35:23] seems to have made performance not meaningfully worse, deploying [15:35:26] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde: Continuing with deployment [15:35:37] (the next patches will be much more interesting, performance-wise. this one just slightly bloated the l10n cache) [15:36:21] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2074.codfw.wmnet with reason: host reimage [15:36:28] !log swfrench@cumin2002 START - Cookbook sre.hosts.reimage for host conf2006.codfw.wmnet with OS bookworm [15:37:33] (03PS3) 10Elukey: docker_registry: add s3cmd and skopeo packages [puppet] - 10https://gerrit.wikimedia.org/r/1319112 (https://phabricator.wikimedia.org/T427175) [15:37:41] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319112 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [15:38:38] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage [15:38:57] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2015.codfw.wmnet with OS bookworm [15:39:12] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations: Move Docker images under the /v2/releng prefix to S3 - https://phabricator.wikimedia.org/T432829#12167849 (10elukey) Copied the image:tag combinations specified in T432829#12144758, plus all the more recent tags up to the latest ones. I am going on... [15:39:27] (03CR) 10Elukey: [C:03+2] docker_registry: add s3cmd and skopeo packages [puppet] - 10https://gerrit.wikimedia.org/r/1319112 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [15:40:40] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1254: Maintenance [15:41:00] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance [15:41:08] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1259 (T431660)', diff saved to https://phabricator.wikimedia.org/P95638 and previous config saved to /var/cache/conftool/dbconfig/20260729-154107-cwilliams.json [15:41:24] !log bking@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [15:41:28] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [15:41:32] !log bking@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 18s) [15:41:50] (03CR) 10Majavah: [C:03+2] P:toolforge::bastion: Add script to migrate OpenSearch indexes [puppet] - 10https://gerrit.wikimedia.org/r/1318228 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [15:42:23] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards [15:43:02] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1219: Maintenance [15:43:22] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance [15:43:31] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1232 (T431660)', diff saved to https://phabricator.wikimedia.org/P95640 and previous config saved to /var/cache/conftool/dbconfig/20260729-154330-cwilliams.json [15:43:58] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2074.codfw.wmnet with reason: host reimage [15:44:40] (03PS9) 10Bking: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [15:45:05] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] "starting gate-and-submit ahead of deployment (I’m testing on mwdebug how this affects performance)" [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318705 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [15:45:06] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] "starting gate-and-submit ahead of deployment (I’m testing on mwdebug how this affects performance)" [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318706 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [15:46:45] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318704|Add i18n/LanguageNames to MessagesDirs (T231755)]] (duration: 31m 59s) [15:46:50] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [15:47:08] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1200: Maintenance [15:47:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318705 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [15:47:09] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318706 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [15:47:14] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12167894 (10VRiley-WMF) System is powered off, but it is reachable. Should be good to go. @jcrespo should we be okay to close this ticket? [15:47:28] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance [15:47:29] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12167896 (10VRiley-WMF) 05In progress→03Open [15:47:31] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1259: Maintenance [15:47:36] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1207 (T431660)', diff saved to https://phabricator.wikimedia.org/P95643 and previous config saved to /var/cache/conftool/dbconfig/20260729-154735-cwilliams.json [15:47:37] (03CR) 10FNegri: [C:03+2] sre.mysql.multiinstance_reboot: fix host type detection [cookbooks] - 10https://gerrit.wikimedia.org/r/1315819 (https://phabricator.wikimedia.org/T420203) (owner: 10FNegri) [15:48:03] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2021.codfw.wmnet with reason: host reimage [15:49:49] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1232: Maintenance [15:50:20] (03PS1) 10Aaron Schulz: Remove some API Gateway specific template files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319114 (https://phabricator.wikimedia.org/T428625) [15:50:26] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [15:50:38] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1181: Maintenance [15:50:54] (03CR) 10JHathaway: [C:03+1] Account blocking: Only unset email if requested [software/bitu] - 10https://gerrit.wikimedia.org/r/1318083 (https://phabricator.wikimedia.org/T431114) (owner: 10Slyngshede) [15:50:54] (03CR) 10Bking: [C:03+2] cirrus: produce ECS server logs on cloudelastic [puppet] - 10https://gerrit.wikimedia.org/r/1311441 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [15:50:58] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance [15:51:05] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1191 (T431660)', diff saved to https://phabricator.wikimedia.org/P95646 and previous config saved to /var/cache/conftool/dbconfig/20260729-155104-cwilliams.json [15:51:23] (03Merged) 10jenkins-bot: sre.mysql.multiinstance_reboot: fix host type detection [cookbooks] - 10https://gerrit.wikimedia.org/r/1315819 (https://phabricator.wikimedia.org/T420203) (owner: 10FNegri) [15:51:32] (03PS8) 10Btullis: cirrus: produce ECS server logs on the production clusters [puppet] - 10https://gerrit.wikimedia.org/r/1311439 (https://phabricator.wikimedia.org/T324335) [15:51:36] (03CR) 10Bking: [C:03+2] cirrus: produce ECS server logs on the production clusters [puppet] - 10https://gerrit.wikimedia.org/r/1311439 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [15:52:34] (03CR) 10CI reject: [V:04-1] Remove some API Gateway specific template files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319114 (https://phabricator.wikimedia.org/T428625) (owner: 10Aaron Schulz) [15:53:16] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1207: Maintenance [15:53:49] (03CR) 10Bking: [C:03+2] "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1311439 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [15:54:24] (03PS1) 10Majavah: P:toolforge::bastion: Fix execution permissions [puppet] - 10https://gerrit.wikimedia.org/r/1319115 [15:55:36] !log swfrench@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on conf2006.codfw.wmnet with reason: host reimage [15:55:57] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1079.eqiad.wmnet with OS trixie [15:56:08] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167976 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1079.eqiad.wmnet with OS trixie [15:57:05] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [15:57:07] (03Merged) 10jenkins-bot: Load language names from JSON instead of PHP [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318705 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [15:57:13] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1191: Maintenance [15:57:30] (03PS1) 10Ottomata: mw-page-html-feature-counts-change-enrich: produce to kafka main-eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319116 (https://phabricator.wikimedia.org/T433507) [15:57:36] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2074.codfw.wmnet with OS trixie [15:57:42] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167985 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2074.codfw.wmnet with OS trixie completed: - ms-be2074 (**PASS*... [15:57:44] (03PS2) 10Aaron Schulz: Remove some API Gateway specific template files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319114 (https://phabricator.wikimedia.org/T428625) [15:57:47] (03CR) 10Majavah: [C:03+2] P:toolforge::bastion: Fix execution permissions [puppet] - 10https://gerrit.wikimedia.org/r/1319115 (owner: 10Majavah) [15:59:20] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [15:59:29] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1247: Maintenance [15:59:34] (03PS1) 10Ottomata: eventstreams: expose page_html_feature_counts_change.v1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319117 (https://phabricator.wikimedia.org/T433507) [15:59:38] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2006.codfw.wmnet with reason: host reimage [15:59:49] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance [15:59:56] (03CR) 10CI reject: [V:04-1] Remove some API Gateway specific template files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319114 (https://phabricator.wikimedia.org/T428625) (owner: 10Aaron Schulz) [15:59:58] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1248 (T431660)', diff saved to https://phabricator.wikimedia.org/P95651 and previous config saved to /var/cache/conftool/dbconfig/20260729-155956-cwilliams.json [16:00:27] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply [16:00:44] (03Merged) 10jenkins-bot: Update rebuild.php to also write message JSON files [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318706 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [16:00:48] (03CR) 10Scott French: [V:03+2 C:03+2] "Thanks again, Ahmon!" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318767 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [16:00:50] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply [16:01:12] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1318705|Load language names from JSON instead of PHP (T231755)]], [[gerrit:1318706|Update rebuild.php to also write message JSON files (T231755)]] [16:01:39] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [16:05:21] (03PS3) 10Aaron Schulz: Remove some API Gateway specific template files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319114 (https://phabricator.wikimedia.org/T428625) [16:06:03] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12168030 (10CDiggs-WMF) @jijiki @Dzahn I'm not sure exactly but I've gotten into idm.wikimedia.org now with the developer account (I didn't realize I had one!?), and have... [16:06:28] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1248: Maintenance [16:07:06] sync-testservers-k8s is taking a while… [16:07:15] (03CR) 10CI reject: [V:04-1] Remove some API Gateway specific template files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319114 (https://phabricator.wikimedia.org/T428625) (owner: 10Aaron Schulz) [16:07:26] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde: Backport for [[gerrit:1318705|Load language names from JSON instead of PHP (T231755)]], [[gerrit:1318706|Update rebuild.php to also write message JSON files (T231755)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:07:30] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [16:07:30] yay, testing [16:08:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:09:57] (03PS1) 10Aaron Schulz: [DNM] Remove some logic specific to the old api-gateway service [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319122 (https://phabricator.wikimedia.org/T428625) [16:10:31] performance is absolutely fine, deploying [16:10:35] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde: Continuing with deployment [16:12:23] !log hot-swap line card in FPC0 on cr1-eqiad with replacement MPC10E from Juniper T426343 [16:12:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:13:16] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] "starting gate-and-submit ahead of deployment (I’m testing on mwdebug how this affects performance)" [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318707 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [16:13:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:23] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1079.eqiad.wmnet with reason: host reimage [16:14:28] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:15:53] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2021.codfw.wmnet with OS bookworm [16:16:47] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12168065 (10MatthewVernon) [16:16:48] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318705|Load language names from JSON instead of PHP (T231755)]], [[gerrit:1318706|Update rebuild.php to also write message JSON files (T231755)]] (duration: 15m 36s) [16:16:51] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [16:17:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318707 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [16:17:14] 10ops-eqiad, 06DC-Ops, 10decommission-hardware, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): decommission an-test-coord1001.eqiad.wmnet - https://phabricator.wikimedia.org/T433494#12168068 (10BTullis) a:05BTullis→03None I still have a puppet cleanup patch to supply and some keytabs to remove, but ple... [16:17:59] (03CR) 10TChin: [C:03+2] [eventstreams] Bump to v0.20.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318784 (https://phabricator.wikimedia.org/T432824) (owner: 10TChin) [16:18:00] 10ops-eqiad, 06DC-Ops, 10decommission-hardware, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): decommission an-test-master100[1-2] - https://phabricator.wikimedia.org/T433495#12168080 (10BTullis) a:05BTullis→03None I still have a puppet cleanup patch to supply and some keytabs to remove, but please fee... [16:18:10] 10ops-eqiad, 06DC-Ops, 10decommission-hardware, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): decommission an-test-master100[1-2] - https://phabricator.wikimedia.org/T433495#12168095 (10BTullis) [16:18:19] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1079.eqiad.wmnet with reason: host reimage [16:18:33] 10ops-eqiad, 06DC-Ops, 10decommission-hardware, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): decommission an-test-coord1001.eqiad.wmnet - https://phabricator.wikimedia.org/T433494#12168096 (10BTullis) [16:19:11] 10ops-eqiad, 06DC-Ops, 10decommission-hardware, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): decommission an-test-master100[1-2] - https://phabricator.wikimedia.org/T433495#12168100 (10BTullis) [16:19:40] !log Deploying Refinery at 56695674 as part of weekly train [16:19:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:20:20] (03Merged) 10jenkins-bot: [eventstreams] Bump to v0.20.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318784 (https://phabricator.wikimedia.org/T432824) (owner: 10TChin) [16:20:53] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2006.codfw.wmnet with OS bookworm [16:21:13] !log akhatun@deploy1003 Started deploy [analytics/refinery@5669567] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@56695674] [16:22:02] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:23:43] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1051 hosts [16:23:52] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12168123 (10Dzahn) @CDiggs-WMF Glad to hear you got in. Personally I did not do anything about the developer account itself. It's possible that ITS created it for you but I... [16:24:58] 06SRE, 06ServiceOps new, 10VisualEditor, 10VisualEditor Suggestion Mode, and 3 others: New Service Request: Headless VE - https://phabricator.wikimedia.org/T431497#12168130 (10Esanders) [16:25:47] (03Merged) 10jenkins-bot: Optimize language name loading with fallbacks [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318707 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [16:26:05] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1318707|Optimize language name loading with fallbacks (T231755)]] [16:26:09] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [16:28:06] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde: Backport for [[gerrit:1318707|Optimize language name loading with fallbacks (T231755)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:28:11] !log akhatun@deploy1003 Finished deploy [analytics/refinery@5669567] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@56695674] (duration: 06m 57s) [16:28:28] 06SRE, 10SRE-tools, 06Infrastructure-Foundations: sre.ganeti.makevm: Create machine types - https://phabricator.wikimedia.org/T344972#12168164 (10LSobanski) p:05Medium→03Low [16:28:33] (03PS1) 10Dzahn: admin: document ldap_only access for Chandler Diggs [puppet] - 10https://gerrit.wikimedia.org/r/1319123 (https://phabricator.wikimedia.org/T433302) [16:28:37] testing once more [16:28:54] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12168169 (10Bethany) @leila , I signed the L3 Wikimedia Server Access Responsibilities document on July 11, 2022, at 9:11 p.m., and I reaffirm my acceptance of those r... [16:29:06] (03CR) 10CI reject: [V:04-1] admin: document ldap_only access for Chandler Diggs [puppet] - 10https://gerrit.wikimedia.org/r/1319123 (https://phabricator.wikimedia.org/T433302) (owner: 10Dzahn) [16:29:45] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests, 13Patch-For-Review: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12168174 (10Dzahn) @jijiki I think even though it's self-service we must still document it in the repo. At least when it's a contractor with an expiry... [16:29:51] (03PS2) 10Dzahn: admin: document ldap_only access for Chandler Diggs [puppet] - 10https://gerrit.wikimedia.org/r/1319123 (https://phabricator.wikimedia.org/T433302) [16:30:15] (03PS1) 10Scott French: hieradata: Move etcd replication back to conf2005 [puppet] - 10https://gerrit.wikimedia.org/r/1319124 (https://phabricator.wikimedia.org/T428495) [16:30:35] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319124 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [16:30:35] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12168185 (10Bethany) [16:30:53] (03CR) 10CI reject: [V:04-1] admin: document ldap_only access for Chandler Diggs [puppet] - 10https://gerrit.wikimedia.org/r/1319123 (https://phabricator.wikimedia.org/T433302) (owner: 10Dzahn) [16:31:02] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations: Cookbook sre.hardware.upgrade-firmware fails to get firmwares from Dell's website - https://phabricator.wikimedia.org/T357756#12168189 (10LSobanski) Untagging #infrastructure-foundations. @wiki_willy could you take a look at updating the documentation ment... [16:32:32] (03PS1) 10Elukey: admin: allow dcops to run the downtime cookbook [puppet] - 10https://gerrit.wikimedia.org/r/1319125 (https://phabricator.wikimedia.org/T433409) [16:32:39] (03PS1) 10Scott French: Revert "hieradata: Temporarily point codfw PyBals at eqiad etcd" [puppet] - 10https://gerrit.wikimedia.org/r/1319126 (https://phabricator.wikimedia.org/T428495) [16:32:55] (03PS2) 10Scott French: Revert "hieradata: Temporarily point codfw PyBals at eqiad etcd" [puppet] - 10https://gerrit.wikimedia.org/r/1319126 (https://phabricator.wikimedia.org/T428495) [16:33:14] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12168226 (10Dzahn) >>! In T433313#12167772, @leila wrote: > .. https://phabricator.wikimedia.org/L3 and agree to its terms? (You may have already signed this form, bu... [16:33:27] (03PS1) 10Scott French: Revert "wmnet: Temporarily direct codfw, eqsin, ulsfo etcd clients to eqiad" [dns] - 10https://gerrit.wikimedia.org/r/1319128 (https://phabricator.wikimedia.org/T428495) [16:33:40] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319126 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [16:34:14] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1259: Maintenance [16:35:07] 10SRE-Access-Requests, 06Infrastructure-Foundations, 10LDAP-Access-Requests, 13Patch-For-Review: Superset data access request for abibendall - https://phabricator.wikimedia.org/T430938#12168235 (10Dzahn) a:05LSobanski→03ABendall-WMF [16:36:04] hm. puzzling performance results [16:36:26] I’ll just think out loud a bit in case anyone wants to chime in [16:36:37] so my `ab` results are at https://www.wikidata.org/wiki/User:Lucas_Werkmeister/translatable_language_names_performance_(T231755) [16:36:37] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [16:36:52] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1232: Maintenance [16:36:56] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1079.eqiad.wmnet with OS trixie [16:36:56] for the baseline, first, and second deployment, I got pretty low standard deviations after rerunning `ab` enough times (which I interpret as the server(s) “warming up”) [16:37:05] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12168269 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1079.eqiad.wmnet with OS trixie completed: - ms-be1079 (**PASS*... [16:37:12] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance [16:37:13] 60, 50, 40-something milliseconds [16:37:20] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1234 (T431660)', diff saved to https://phabricator.wikimedia.org/P95664 and previous config saved to /var/cache/conftool/dbconfig/20260729-163719-cwilliams.json [16:37:28] and then for the third deployment, the standard deviation won’t go below 100 no matter how often I rerun `ab` [16:37:39] I got *one* run where the median time was still better than in the other deployments [16:37:53] !log btullis@cumin1003 START - Cookbook sre.hosts.reboot-single for host an-test-master1004.eqiad.wmnet [16:37:57] but a lot of other runs have a higher median time, often substantially so, above 200, compared to 140-ish for the other deploys) [16:38:07] (03PS3) 10Dzahn: admin: document ldap_only access for Chandler Diggs [puppet] - 10https://gerrit.wikimedia.org/r/1319123 (https://phabricator.wikimedia.org/T433302) [16:38:23] and I’m not sure if this means the last patch (third deployment) made the performance a lot more erratic… or if mwdebug is just behaving differently for some other reason [16:38:57] !log tchin@deploy1003 helmfile [staging] START helmfile.d/services/eventstreams: apply [16:39:02] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1207: Maintenance [16:39:12] (wow, just now I got a run where the longest request took 6784 ms…) [16:39:16] (03CR) 10Dzahn: "thanks for this! I was thinking I could also remove just the proxy class first and then the other to make the diff smaller." [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [16:39:24] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance [16:39:30] !log tchin@deploy1003 helmfile [staging] DONE helmfile.d/services/eventstreams: apply [16:39:33] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1230 (T431660)', diff saved to https://phabricator.wikimedia.org/P95667 and previous config saved to /var/cache/conftool/dbconfig/20260729-163932-cwilliams.json [16:39:57] !log tchin@deploy1003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [16:40:05] !log tchin@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [16:40:12] I think the responsible thing to do is to assume that the last change doesn’t improve performance… yet [16:40:28] I think it might fare better once the other l10n cache improvements (bvibber et al.) go in [16:40:30] !log tchin@deploy1003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [16:40:35] so I might just try it again then [16:40:42] (and then I’ll also have a new baseline to compare against) [16:40:59] (03CR) 10Blake: [C:03+1] hieradata: Move etcd replication back to conf2005 [puppet] - 10https://gerrit.wikimedia.org/r/1319124 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [16:41:01] but for now I’ll abort this deploy and revert the change on wmf.13 [16:41:19] and then -1 this one change on master, and rebase the rest around so the rest of the chain can still be merged [16:41:19] !log tchin@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [16:41:26] so we’ll still have translatable language names on translatewiki.net, with one implementation [16:41:35] and in a few weeks? months? we can try the other implementation again [16:43:10] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-master1004.eqiad.wmnet [16:43:15] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests, 13Patch-For-Review: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12168318 (10CDiggs-WMF) Thanks for your help here team! [16:43:15] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde: Rolling back deployment [16:43:25] !log tchin@deploy1003 helmfile [eqiad] START helmfile.d/services/eventstreams: apply [16:43:42] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1234: Maintenance [16:43:44] !log btullis@cumin1003 START - Cookbook sre.hosts.reboot-single for host an-test-master1003.eqiad.wmnet [16:43:47] (03PS1) 10Dzahn: ci: stop including the legacy jenkins proxy config [puppet] - 10https://gerrit.wikimedia.org/r/1319130 (https://phabricator.wikimedia.org/T418521) [16:43:55] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1191: Maintenance [16:44:15] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance [16:44:16] !log tchin@deploy1003 helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply [16:44:22] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1194 (T431660)', diff saved to https://phabricator.wikimedia.org/P95669 and previous config saved to /var/cache/conftool/dbconfig/20260729-164422-cwilliams.json [16:44:28] (03CR) 10Dzahn: "just like I1987adf717297c4 but only half of it to reduce the diff" [puppet] - 10https://gerrit.wikimedia.org/r/1319130 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [16:45:00] (03CR) 10Dzahn: "removing just the proxy config first and then doing the second part here" [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [16:45:14] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1230: Maintenance [16:45:20] (03CR) 10JHathaway: [C:03+1] data.yaml offboarding for suecarmol [puppet] - 10https://gerrit.wikimedia.org/r/1318055 (owner: 10Slyngshede) [16:46:05] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [16:46:27] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12168390 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2097.codfw.wmnet with OS bullseye execute... [16:46:44] !log jhancock@cumin2002 START - Cookbook sre.hosts.reimage for host ms-be2097.codfw.wmnet with OS bullseye [16:46:46] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [16:46:51] (03PS1) 10Lucas Werkmeister (WMDE): Revert "Optimize language name loading with fallbacks" [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319131 (https://phabricator.wikimedia.org/T231755) [16:46:52] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12168396 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2097.codfw.wmnet with OS bullseye [16:47:00] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] Revert "Optimize language name loading with fallbacks" [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319131 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [16:47:30] (03Abandoned) 10Lucas Werkmeister (WMDE): Remove most of LocalNames [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318708 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [16:47:32] (03Abandoned) 10Lucas Werkmeister (WMDE): Remove T231755.php again [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318709 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [16:47:47] !log silenced EtcdReplicationDown 57b2b421-1cc9-4e38-9276-94f223fd231c - T428495 [16:47:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:47:52] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [16:48:04] * Lucas_WMDE waits for the revert to go through gate-and-submit before releasing the deployment lock [16:48:49] (03CR) 10Scott French: [C:03+2] hieradata: Move etcd replication back to conf2005 [puppet] - 10https://gerrit.wikimedia.org/r/1319124 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [16:50:16] (03CR) 10Dzahn: [V:03+1 C:03+2] "just removing apache config as step 1:" [puppet] - 10https://gerrit.wikimedia.org/r/1319130 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [16:50:28] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1194: Maintenance [16:50:53] (03CR) 10JHathaway: [C:03+1] admin: allow dcops to run the downtime cookbook [puppet] - 10https://gerrit.wikimedia.org/r/1319125 (https://phabricator.wikimedia.org/T433409) (owner: 10Elukey) [16:52:01] (03CR) 10Dzahn: [C:03+1] "Surprised they did not already have this since a long time" [puppet] - 10https://gerrit.wikimedia.org/r/1319125 (https://phabricator.wikimedia.org/T433409) (owner: 10Elukey) [16:53:11] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1248: Maintenance [16:53:32] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1249.eqiad.wmnet with reason: Maintenance [16:53:40] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1249 (T431660)', diff saved to https://phabricator.wikimedia.org/P95674 and previous config saved to /var/cache/conftool/dbconfig/20260729-165339-cwilliams.json [16:54:32] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards [16:54:36] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [16:55:08] (03PS3) 10Dzahn: ci: remove jenkins profile from role::ci [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) [16:55:30] (03CR) 10CI reject: [V:04-1] ci: remove jenkins profile from role::ci [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [16:55:56] (03CR) 10Dzahn: "Thanks! Can adjust the comment accordingly." [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [16:57:30] (03PS4) 10Dzahn: ci: remove jenkins profile from role::ci [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) [16:58:58] FIRING: [2x] JobUnavailable: Reduced availability for job etcdmirror in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:59:53] !log jhancock@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage [17:00:04] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1249: Maintenance [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1700) [17:00:28] I’m technically still holding the deployment lock (should be done in a moment) [17:00:46] (03PS5) 10Dzahn: ci: remove jenkins profile from role::ci [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) [17:02:21] (03Merged) 10jenkins-bot: Revert "Optimize language name loading with fallbacks" [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319131 (https://phabricator.wikimedia.org/T231755) (owner: 10Lucas Werkmeister (WMDE)) [17:02:30] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318707|Optimize language name loading with fallbacks (T231755)]] (duration: 36m 25s) [17:02:35] T231755: Local language name should be translatable in translatewiki.net - https://phabricator.wikimedia.org/T231755 [17:02:38] * Lucas_WMDE done deploying, sorry for the slight overrun [17:02:55] no problem :) [17:03:16] dancy: are you good to proceed with the scap.cfg change? [17:03:23] Yep, let's do it [17:03:30] * swfrench-wmf thumbs up [17:03:48] !log CI: contint1002/contint2002 - rm /etc/apache2/jenkins_proxy - removing legacy jenkins proxy config - jenkins is on new dedicated machines and uses jenkins_proxy_ext config T418521 [17:03:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:03:53] T418521: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521 [17:03:55] (03CR) 10Scott French: [C:03+2] "Great, thanks for confirming, Ahmon." [puppet] - 10https://gerrit.wikimedia.org/r/1318302 (https://phabricator.wikimedia.org/T263872) (owner: 10Ahmon Dancy) [17:04:15] !log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage [17:05:12] !log CI: contint1002/contint2002 - restarted httpd to be extra sure all is cleaned up - https://integration.wikimedia.org/ci/ is up and running T418521 [17:05:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:05:53] * swfrench-wmf is now waiting for puppet on the deployment host [17:07:28] oh, I should also `git pull` the revert on deploy1003, shouldn’t I [17:08:34] done, I think [17:09:59] Lucas_WMDE: just so we know what to expect, when dancy runs a smoke-test scap deployment in a couple of minutes, what should we expect as side-effects of your revert? (e.g., will there be l10n changes?) [17:10:18] there should be nothing special [17:10:27] I did several deployments, only the first of which changed the l10n cache, and only the last was reverted [17:10:43] the others seemed to be fine performance-wise and stay in place [17:10:57] ah, great - thanks for clarifying, as I wasn't closely following along :) [17:10:58] (though ideally I’d get them merged on the master branch before next week, otherwise they’ll vanish again with the next train :D) [17:12:45] heh, indeed [17:13:06] dancy: puppet run is done, scap.cfg is ready for you [17:13:53] !log dancy@deploy1003 Started scap sync-world: Testing delay_messageblobstore_purge: true [17:16:41] !log akhatun@deploy1003 Started deploy [analytics/refinery@5669567] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@56695674] [17:17:04] !log akhatun@deploy1003 Finished deploy [analytics/refinery@5669567] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@56695674] (duration: 00m 22s) [17:17:25] !log akhatun@deploy1003 Started deploy [analytics/refinery@5669567]: Regular analytics weekly train [analytics/refinery@56695674] [17:18:20] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12168622 (10leila) @Bethany and @Dzahn: thank you. I approve bgwiki's request for Level 2 access to analytics-privatedata-users. Thank you. [17:18:58] FIRING: [2x] JobUnavailable: Reduced availability for job etcdmirror in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:20:22] !log dancy@deploy1003 Finished scap sync-world: Testing delay_messageblobstore_purge: true (duration: 06m 29s) [17:20:43] swfrench-wmf: Looks good [17:20:52] dancy: awesome, thanks! [17:22:03] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:23:45] !log akhatun@deploy1003 Finished deploy [analytics/refinery@5669567]: Regular analytics weekly train [analytics/refinery@56695674] (duration: 06m 20s) [17:24:28] !log akhatun@deploy1003 Started deploy [analytics/refinery@5669567] (thin): Regular analytics weekly train THIN [analytics/refinery@56695674] [17:24:32] (03PS4) 10Aaron Schulz: Remove some API Gateway specific template files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319114 (https://phabricator.wikimedia.org/T428625) [17:26:30] !log akhatun@deploy1003 Finished deploy [analytics/refinery@5669567] (thin): Regular analytics weekly train THIN [analytics/refinery@56695674] (duration: 02m 02s) [17:26:49] (03CR) 10CI reject: [V:04-1] Remove some API Gateway specific template files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319114 (https://phabricator.wikimedia.org/T428625) (owner: 10Aaron Schulz) [17:28:31] (03PS10) 10Btullis: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) [17:28:31] (03PS8) 10Btullis: cirrus: remove the log4j syslog appender log shipping path [puppet] - 10https://gerrit.wikimedia.org/r/1311426 (https://phabricator.wikimedia.org/T324335) [17:28:46] (03PS1) 10Btullis: opensearch: fix log4j hot-reloading under the Java Security Manager [puppet] - 10https://gerrit.wikimedia.org/r/1319135 (https://phabricator.wikimedia.org/T324335) [17:28:52] (03PS1) 10Btullis: opensearch: drop the legacy syslog appender security policy grants [puppet] - 10https://gerrit.wikimedia.org/r/1319136 (https://phabricator.wikimedia.org/T324335) [17:30:24] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1234: Maintenance [17:30:43] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance [17:30:51] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1235 (T431660)', diff saved to https://phabricator.wikimedia.org/P95684 and previous config saved to /var/cache/conftool/dbconfig/20260729-173051-cwilliams.json [17:31:09] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1230: Maintenance [17:31:54] (03CR) 10Bking: [C:04-1] opensearch: ship the cirrus ECS server logs via rsyslog imfile (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [17:32:25] 06SRE, 06Infrastructure-Foundations, 06ServiceOps new, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12168681 (10bd808) [17:33:54] (03CR) 10Dzahn: [C:03+2] "verifying on secondary machine first" [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [17:37:15] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1235: Maintenance [17:37:27] (03CR) 10RobH: [C:03+1] admin: allow dcops to run the downtime cookbook [puppet] - 10https://gerrit.wikimedia.org/r/1319125 (https://phabricator.wikimedia.org/T433409) (owner: 10Elukey) [17:37:32] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1194: Maintenance [17:37:51] (03CR) 10BCornwall: [C:03+1] Revert "hieradata: Temporarily point codfw PyBals at eqiad etcd" [puppet] - 10https://gerrit.wikimedia.org/r/1319126 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [17:37:51] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance [17:38:00] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1202 (T431660)', diff saved to https://phabricator.wikimedia.org/P95688 and previous config saved to /var/cache/conftool/dbconfig/20260729-173759-cwilliams.json [17:38:15] (03CR) 10BCornwall: [C:03+1] Revert "wmnet: Temporarily direct codfw, eqsin, ulsfo etcd clients to eqiad" [dns] - 10https://gerrit.wikimedia.org/r/1319128 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [17:41:11] !log Deployed refinery using scap, then deployed onto hdfs [17:41:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:44:08] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12168699 (10cmooney) @VRiley to confirm what the next steps are here of most importance to unblock my side of the work: # Cable the mgmt port on each of the... [17:44:08] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1202: Maintenance [17:44:53] (03CR) 10Scott French: [C:03+2] Revert "wmnet: Temporarily direct codfw, eqsin, ulsfo etcd clients to eqiad" [dns] - 10https://gerrit.wikimedia.org/r/1319128 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [17:45:10] (03CR) 10BCornwall: [C:03+2] Revert "hieradata: Temporarily point codfw PyBals at eqiad etcd" [puppet] - 10https://gerrit.wikimedia.org/r/1319126 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [17:45:11] !log swfrench@dns1004 START - running authdns-update [17:45:17] !log bking@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs2015\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main [17:46:17] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: eqiad row A&B host migration details request for Data Persistence - https://phabricator.wikimedia.org/T432644#12168702 (10RobH) >>! In T432644#12166236, @MatthewVernon wrote: > I've done this for thanos and ms-{fe,be} hosts. Can I check that this is just ph... [17:46:45] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1249: Maintenance [17:47:06] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance [17:47:14] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1252 (T431660)', diff saved to https://phabricator.wikimedia.org/P95692 and previous config saved to /var/cache/conftool/dbconfig/20260729-174713-cwilliams.json [17:47:19] (03PS1) 10Cathal Mooney: Enable OSPF on HE drmrs <-> eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1319140 (https://phabricator.wikimedia.org/T424839) [17:47:25] !log bking@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [17:47:29] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [17:47:29] !log swfrench@dns1004 END - running authdns-update [17:47:32] !log authdns-update to direct codfw, eqsin, ulsfo etcd clients back to codfw - T428495 [17:47:33] !log bking@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 14s) [17:47:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:47:36] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [17:47:50] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2021.codfw.wmnet, repooling source-only afterwards [17:50:13] !log brett@cumin2002 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-codfw and A:lvs (T428495) [17:50:42] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2022.codfw.wmnet with OS bookworm [17:51:34] (03PS2) 10Cathal Mooney: Enable OSPF on HE drmrs <-> eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1319140 (https://phabricator.wikimedia.org/T424839) [17:53:13] swfrench-wmf: pybal restarted on A:lvs-codfw. lvs2013 is struggling but that's related to another issue [17:53:24] (i.e. the cookbook that's running will fail but it's not related) [17:53:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:53:31] * swfrench-wmf looks at icinga [17:53:45] (03CR) 10Papaul: [C:03+1] Enable OSPF on HE drmrs <-> eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1319140 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [17:54:11] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2022 [17:54:33] PROBLEM - PyBal connections to etcd on lvs2014 is CRITICAL: CRITICAL: 0 connections established with conf2004.codfw.wmnet:4001 (min=97) https://wikitech.wikimedia.org/wiki/PyBal [17:54:33] brett: ah, I see ... yeah, I guess this is one of those situations where the polling for green will time out since there's a preexisting issue [17:54:51] ^ transiently expected [17:55:40] !log begin rolling restart of confd in codfw, eqsin, ulsfo - T428495 [17:55:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:55:44] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [17:56:43] !log brett@cumin2002 END (FAIL) - Cookbook sre.loadbalancer.restart-pybal (exit_code=1) rolling-restart of pybal on A:lvs-codfw and A:lvs (T428495) [17:57:10] !log bking@cumin2003 START - Cookbook sre.dns.netbox [17:57:56] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [debs/pint] - 10https://gerrit.wikimedia.org/r/1319039 (owner: 10Hnowlan) [17:59:28] (03CR) 10BCornwall: "> Out of curiosity which base images don't ship systemd ?" [debs/pint] - 10https://gerrit.wikimedia.org/r/1319039 (owner: 10Hnowlan) [18:00:05] dduvall and dancy: Time to do the MediaWiki train - Utc-7 Version deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1800). [18:02:48] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2022 - bking@cumin2003" [18:02:53] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2022 - bking@cumin2003" [18:02:53] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:02:54] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs2022.codfw.wmnet 211.48.192.10.in-addr.arpa 1.1.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:02:54] (03CR) 10Cathal Mooney: [C:03+2] Enable OSPF on HE drmrs <-> eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1319140 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [18:02:57] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2022.codfw.wmnet 211.48.192.10.in-addr.arpa 1.1.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:02:58] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs2022 [18:02:59] !log restarted navtiming on webperf2003 - T428495 [18:03:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:03:06] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [18:03:10] dancy: o/ [18:04:29] (03Merged) 10jenkins-bot: Enable OSPF on HE drmrs <-> eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1319140 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [18:04:53] (03CR) 10BCornwall: [C:03+1] wikimedia.org: add TXT record for BIMI [dns] - 10https://gerrit.wikimedia.org/r/1314966 (https://phabricator.wikimedia.org/T311685) (owner: 10Ssingh) [18:05:08] obvious-dust and Krinkle i'm about to roll the wmf.13 train to group1 if you're still wanting to observe [18:05:24] (it's a very similar process to what we did yesterday) [18:06:02] bking@cumin2003 reimage (PID 1463407) is awaiting input [18:07:19] swfrench-wmf: how are things? [18:07:44] brett: moving along well. how's pybal doing? [18:08:39] (03PS1) 10Ahmon Dancy: buildkitd: Pass -v when removing the container [puppet] - 10https://gerrit.wikimedia.org/r/1319144 (https://phabricator.wikimedia.org/T433505) [18:08:50] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2022 [18:08:50] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2022 [18:08:54] I'm going to be leaving the errors where they were [18:09:12] 06SRE, 06ServiceOps new, 10VisualEditor, 10VisualEditor Suggestion Mode, and 3 others: Deploy Headless VE in k8s for technical pilot - https://phabricator.wikimedia.org/T431497#12168751 (10Ottomata) [18:09:24] (03CR) 10Ahmon Dancy: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319144 (https://phabricator.wikimedia.org/T433505) (owner: 10Ahmon Dancy) [18:09:31] (03PS1) 10TrainBranchBot: group1 to 1.47.0-wmf.13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319145 (https://phabricator.wikimedia.org/T430832) [18:09:34] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dduvall@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319145 (https://phabricator.wikimedia.org/T430832) (owner: 10TrainBranchBot) [18:09:35] (03CR) 10Dzahn: [C:03+2] "it does not look that much anymore in the actual puppet run:" [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [18:09:56] brett: which lvs hosts have you done? [18:10:04] swfrench-wmf: All of them [18:10:19] *A:lvs-codfw [18:10:22] `Active: active (running) since Wed 2026-07-29 13:21:19 UTC; 4h 48min ago` [18:10:35] on lvs2014 for example [18:10:46] (03CR) 10Dzahn: [C:03+2] "so it's only rsyslog and the 2 firewall rules that actively got removed. the rest is not actively removed by puppet or not affected." [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [18:11:20] (03Merged) 10jenkins-bot: group1 to 1.47.0-wmf.13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319145 (https://phabricator.wikimedia.org/T430832) (owner: 10TrainBranchBot) [18:11:45] brett: yeah, it looks like lvs2014 could use a bonk [18:11:48] (03PS2) 10Ahmon Dancy: buildkitd: Pass -v when removing the container [puppet] - 10https://gerrit.wikimedia.org/r/1319144 (https://phabricator.wikimedia.org/T433505) [18:11:58] (03CR) 10Ahmon Dancy: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319144 (https://phabricator.wikimedia.org/T433505) (owner: 10Ahmon Dancy) [18:12:20] (03CR) 10Dzahn: "broke puppet in beta:" [puppet] - 10https://gerrit.wikimedia.org/r/1319102 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [18:12:31] but of course a cookbook didn't work [18:12:39] oh, right, because 2013 failed [18:12:50] ah, so it stopped in its tracks [18:13:10] !log brett@cumin2002 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs (T428495) [18:13:15] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [18:14:14] bonked [18:15:02] !log CI: contint1002/contint2002: apt-get remove --purge jenkins - jenkins be gone - T418521 [18:15:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:15:06] T418521: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521 [18:15:17] !log brett@cumin2002 END (ERROR) - Cookbook sre.loadbalancer.restart-pybal (exit_code=97) rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs (T428495) [18:15:18] similar failures expected [18:15:27] jouncebot: nowandnext [18:15:27] For the next 1 hour(s) and 44 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1800) [18:15:27] In 1 hour(s) and 44 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T2000) [18:15:40] brett: awesome, thanks! at your convenience, libericas are ready to go. copy-paste-able command in https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Move_other_clients if that's handy [18:15:53] 06SRE, 06ServiceOps new, 10VisualEditor, 10VisualEditor Suggestion Mode, and 3 others: Deploy Headless VE in k8s for technical pilot - https://phabricator.wikimedia.org/T431497#12168766 (10Ottomata) [18:15:57] brett: that'd be ulsfo and eqsin, specifically [18:16:16] ack, on it [18:16:17] !log removing jenkins during the train - living on the edge - no, just kidding, jenkins has migrated to dedicated machines, nothing should happen [18:16:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:16:25] (03PS4) 10Aaron Schulz: Add CacheAbstractContentFragment to high_traffic_jobs with concurrency=1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314852 (https://phabricator.wikimedia.org/T430898) (owner: 10Genoveva Galarza) [18:16:25] (03PS1) 10Ahmon Dancy: buildkitd: Pin the snapshotter to overlayfs [puppet] - 10https://gerrit.wikimedia.org/r/1319146 (https://phabricator.wikimedia.org/T433505) [18:16:55] (03CR) 10Aaron Schulz: [C:03+1] Add CacheAbstractContentFragment to high_traffic_jobs with concurrency=1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314852 (https://phabricator.wikimedia.org/T430898) (owner: 10Genoveva Galarza) [18:17:19] !log dduvall@deploy1003 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.13 refs T430832 [18:17:24] T430832: 1.47.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T430832 [18:17:25] (03PS2) 10Ahmon Dancy: buildkitd: Pin the snapshotter to overlayfs [puppet] - 10https://gerrit.wikimedia.org/r/1319146 (https://phabricator.wikimedia.org/T433505) [18:17:33] !log brett@cumin2002 START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T428495) [18:17:41] RECOVERY - PyBal connections to etcd on lvs2014 is OK: OK: 97 connections established with conf2004.codfw.wmnet:4001 (min=97) https://wikitech.wikimedia.org/wiki/PyBal [18:17:55] (03CR) 10Ahmon Dancy: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319146 (https://phabricator.wikimedia.org/T433505) (owner: 10Ahmon Dancy) [18:17:57] "The following packages were automatically installed and are no longer required: .. puppet" wtf :) [18:18:10] mutante: Sounds like a good thing to me [18:18:13] haha [18:18:35] lol [18:19:13] !log brett@cumin2002 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T428495) [18:19:17] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [18:19:27] !log brett@cumin2002 START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin (T428495) [18:21:13] PROBLEM - OSPF status on cr1-eqiad is CRITICAL: OSPFv2: 8/9 UP : OSPFv3: 8/9 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:22:05] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations: Cookbook sre.hardware.upgrade-firmware fails to get firmwares from Dell's website - https://phabricator.wikimedia.org/T357756#12168795 (10wiki_willy) a:03RobH Chatted with @RobH, who will follow up on this one. Thanks, Willy [18:22:25] !log brett@cumin2002 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin (T428495) [18:23:56] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1235: Maintenance [18:24:21] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1251.eqiad.wmnet with reason: Maintenance [18:24:21] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12168813 (10RobH) a:05elukey→03MatthewVernon @MatthewVernon & @KOfori, This sretest2010.codfw.wmnet host is now ready in codfw for ms-be type testing of... [18:24:26] !log brett@cumin2002 START - Cookbook sre.hosts.remove-downtime for lvs2014.codfw.wmnet [18:24:27] !log brett@cumin2002 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2014.codfw.wmnet [18:24:29] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1251 (T431660)', diff saved to https://phabricator.wikimedia.org/P95698 and previous config saved to /var/cache/conftool/dbconfig/20260729-182428-cwilliams.json [18:25:13] RECOVERY - OSPF status on cr1-eqiad is OK: OSPFv2: 9/9 UP : OSPFv3: 9/9 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:27:09] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12168832 (10VRiley-WMF) 05Open→03In progress Moving this now. [18:27:40] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2022.codfw.wmnet with reason: host reimage [18:30:46] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1251: Maintenance [18:30:50] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1202: Maintenance [18:31:03] brett: awesome, looks like we're all done. thanks so much for your help! [18:31:09] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance [18:31:18] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1227 (T431660)', diff saved to https://phabricator.wikimedia.org/P95701 and previous config saved to /var/cache/conftool/dbconfig/20260729-183117-cwilliams.json [18:31:32] Thank you! [18:31:53] train is looking good [18:34:18] 10ops-codfw, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476#12168856 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm tobias downtimed the server for me and... [18:34:38] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2022.codfw.wmnet with reason: host reimage [18:35:53] 10ops-codfw, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478#12168862 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm learned lesson from T433476. same error... [18:37:08] (03PS1) 10Perryprog: Use os.environ.get for checking DJANGO_DEBUGGING [software/bitu] - 10https://gerrit.wikimedia.org/r/1319150 [18:37:24] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1227: Maintenance [18:44:04] (03PS1) 10Ahmon Dancy: buildkitd: Bump buildkit image to wmf-v0.32.0 [puppet] - 10https://gerrit.wikimedia.org/r/1319151 (https://phabricator.wikimedia.org/T433520) [18:45:04] (03PS2) 10Ahmon Dancy: buildkitd: Bump buildkit image to wmf-v0.32.0 [puppet] - 10https://gerrit.wikimedia.org/r/1319151 (https://phabricator.wikimedia.org/T433520) [18:46:28] (03CR) 10Daniel Kinzler: [DNM] Remove some logic specific to the old api-gateway service (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319122 (https://phabricator.wikimedia.org/T428625) (owner: 10Aaron Schulz) [18:47:01] (03PS10) 10Cwhite: profile: add initial opensearch security plugin config [puppet] - 10https://gerrit.wikimedia.org/r/1306792 (https://phabricator.wikimedia.org/T350516) [18:47:01] (03PS4) 10Cwhite: opensearch: add security-plugin specific configuration to opensearch.yml [puppet] - 10https://gerrit.wikimedia.org/r/1318792 (https://phabricator.wikimedia.org/T350516) [18:47:33] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1252 (T431660)', diff saved to https://phabricator.wikimedia.org/P95704 and previous config saved to /var/cache/conftool/dbconfig/20260729-184732-cwilliams.json [18:48:39] (03CR) 10Scott French: [C:03+1] "Seems reasonable to me! Ping me if you need a merge." [puppet] - 10https://gerrit.wikimedia.org/r/1319144 (https://phabricator.wikimedia.org/T433505) (owner: 10Ahmon Dancy) [18:49:04] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:50:04] (03CR) 10Cwhite: profile: add initial opensearch security plugin config (0315 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1306792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [18:50:24] (03CR) 10Dzahn: [C:03+2] buildkitd: Bump buildkit image to wmf-v0.32.0 [puppet] - 10https://gerrit.wikimedia.org/r/1319151 (https://phabricator.wikimedia.org/T433520) (owner: 10Ahmon Dancy) [18:50:50] (03CR) 10Cwhite: opensearch: add security-plugin specific configuration to opensearch.yml (036 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1318792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [18:51:25] (03CR) 10Dzahn: [C:03+2] buildkitd: Pass -v when removing the container [puppet] - 10https://gerrit.wikimedia.org/r/1319144 (https://phabricator.wikimedia.org/T433505) (owner: 10Ahmon Dancy) [18:53:28] (03CR) 10Daniel Kinzler: [DNM] Remove some logic specific to the old api-gateway service (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319122 (https://phabricator.wikimedia.org/T428625) (owner: 10Aaron Schulz) [18:55:00] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 539159384 and 46 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [18:57:12] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12168920 (10VRiley-WMF) Moved unit into C8 U26 cable ID: 5346 Port 26 Updating netbox now. [18:57:41] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P95707 and previous config saved to /var/cache/conftool/dbconfig/20260729-185740-cwilliams.json [18:58:00] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 77328 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [18:58:54] Hi dduvall, I think we have a train blocker: [18:58:54] https://phabricator.wikimedia.org/T433457 [18:58:54] I added it to wmf.13 task. [18:59:00] cc akhatun [19:01:04] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2021.codfw.wmnet, repooling source-only afterwards [19:01:09] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [19:01:16] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2022.codfw.wmnet with OS bookworm [19:04:28] (and dduvall sorry I don't check IRC much these days so I might not see pings here unless I'm looking, will see slack or task comments though) [19:06:40] (03PS1) 10JHathaway: lists.wikimedia.org: dmarc, set policy to reject [dns] - 10https://gerrit.wikimedia.org/r/1319154 (https://phabricator.wikimedia.org/T379517) [19:07:00] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 232483088 and 20 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:07:49] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P95709 and previous config saved to /var/cache/conftool/dbconfig/20260729-190748-cwilliams.json [19:09:00] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2834472 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:09:04] 06SRE, 06collaboration-services, 10Wikimedia-Mailing-lists, 13Patch-For-Review: Further improve DMARC compatibility on lists.wikimedia.org - https://phabricator.wikimedia.org/T379517#12168955 (10jhathaway) patch out for review: https://gerrit.wikimedia.org/r/c/operations/dns/+/1319154 However, there is no... [19:09:19] ottomata: thanks for filing that! please advise on rollback [19:09:52] (03PS1) 10Scott French: mw-*: Revert temporary mw.mail_timeout increase [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319157 (https://phabricator.wikimedia.org/T383047) [19:11:24] !log rolling back wmf.13 to group0 due to T433457 (cc T430832) [19:11:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:11:30] T433457: `mediawiki.api-request` validation errors from Extension:OAuth `context.oauth_consumer_*` logging context - https://phabricator.wikimedia.org/T433457 [19:11:30] T430832: 1.47.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T430832 [19:11:35] (03PS1) 10TrainBranchBot: group0 to 1.47.0-wmf.13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319158 (https://phabricator.wikimedia.org/T430832) [19:11:38] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dduvall@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319158 (https://phabricator.wikimedia.org/T430832) (owner: 10TrainBranchBot) [19:12:00] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 234619624 and 28 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:13:29] (03Merged) 10jenkins-bot: group0 to 1.47.0-wmf.13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319158 (https://phabricator.wikimedia.org/T430832) (owner: 10TrainBranchBot) [19:15:00] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 58272 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:15:39] Krinkle, obvious-dust: just fyi there is now a train blocker and i have performed a rollback. in spiderpig, it is quite easy to do: select the previous group and click "roll back" [19:16:52] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [19:17:12] there are some edge cases where rolling back can actually cause more issues but they are rare and are typically related to a lack of versioning in serialized objects or something like that [19:17:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:17:57] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1252 (T431660)', diff saved to https://phabricator.wikimedia.org/P95711 and previous config saved to /var/cache/conftool/dbconfig/20260729-191756-cwilliams.json [19:18:44] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1251: Maintenance [19:19:31] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations: Cookbook sre.hardware.upgrade-firmware fails to get firmwares from Dell's website - https://phabricator.wikimedia.org/T357756#12168996 (10RobH) I've made a first run at this to remove the wholly inaccurate info so its not there anymore but will keep this t... [19:19:33] !log dduvall@deploy1003 rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.13 refs T430832 [19:19:38] T430832: 1.47.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T430832 [19:21:15] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1048] - vriley@cumin1003" [19:21:20] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1048] - vriley@cumin1003" [19:21:20] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [19:22:11] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 [19:22:27] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1048 [19:22:51] (03PS1) 10Zabe: Add throttle exemption for Black Cultural Archives UK [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319159 (https://phabricator.wikimedia.org/T433529) [19:23:45] (03PS2) 10Zabe: Add throttle exemption for Black Cultural Archives UK [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319159 (https://phabricator.wikimedia.org/T433529) [19:24:26] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1227: Maintenance [19:24:47] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance [19:24:54] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1231 (T431660)', diff saved to https://phabricator.wikimedia.org/P95714 and previous config saved to /var/cache/conftool/dbconfig/20260729-192454-cwilliams.json [19:27:57] (03CR) 10Zabe: [C:03+2] Add throttle exemption for Black Cultural Archives UK [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319159 (https://phabricator.wikimedia.org/T433529) (owner: 10Zabe) [19:29:01] (03Merged) 10jenkins-bot: Add throttle exemption for Black Cultural Archives UK [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319159 (https://phabricator.wikimedia.org/T433529) (owner: 10Zabe) [19:29:31] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1319159|Add throttle exemption for Black Cultural Archives UK (T433529)]] [19:29:35] T433529: Temporary IP Lift Request for Black Cultural Archives UK - https://phabricator.wikimedia.org/T433529 [19:31:01] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1231: Maintenance [19:31:31] !log zabe@deploy1003 zabe: Backport for [[gerrit:1319159|Add throttle exemption for Black Cultural Archives UK (T433529)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:32:19] !log zabe@deploy1003 zabe: Continuing with deployment [19:34:34] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12169045 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2097.codfw.wmnet with OS bullseye execute... [19:34:35] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12169046 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2098.codfw.wmnet with OS bullseye execute... [19:36:19] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1319159|Add throttle exemption for Black Cultural Archives UK (T433529)]] (duration: 06m 49s) [19:36:27] T433529: Temporary IP Lift Request for Black Cultural Archives UK - https://phabricator.wikimedia.org/T433529 [19:37:40] !log zabe@deploy1003:~$ mwscript-k8s --comment='T433529' --follow -- resetAuthenticationThrottle.php --wiki=aawiki --signup --ip=89.36.114.94 [19:37:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:39:03] (03PS1) 10Ozge: ml-services: set jina embeddings staging to ml-serve1012 temporarily. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319160 [19:39:29] (03CR) 10JHathaway: [C:03+1] mw-*: Revert temporary mw.mail_timeout increase [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319157 (https://phabricator.wikimedia.org/T383047) (owner: 10Scott French) [19:39:51] (03CR) 10Ozge: [V:03+2 C:03+2] "temporary patch to test jina in ml-serve1012" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319160 (owner: 10Ozge) [19:41:08] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [19:43:49] !log pt1979@cumin2002 START - Cookbook sre.hosts.reimage for host ms-be2097.codfw.wmnet with OS bullseye [19:43:59] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12169064 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host ms-be2097.codfw.wmnet with OS bullseye [19:46:23] (03PS1) 10Ozge: ml-services: revert temporary change (set jina embeddings staging to ml-serve1012) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319161 (https://phabricator.wikimedia.org/T433361) [19:46:50] (03CR) 10Ozge: [V:03+2 C:03+2] "reverting the temp change." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319161 (https://phabricator.wikimedia.org/T433361) (owner: 10Ozge) [19:47:12] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12169083 (10VRiley-WMF) Updating firmware on cloudvirt1048 [19:47:24] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [19:47:26] (03PS1) 10Ladsgroup: Remove $wmg hack for UploadStashMaxAge [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319162 (https://phabricator.wikimedia.org/T119117) [19:50:26] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [19:52:02] (03CR) 10Zabe: [C:03+1] Remove $wmg hack for UploadStashMaxAge [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319162 (https://phabricator.wikimedia.org/T119117) (owner: 10Ladsgroup) [19:54:01] 10ops-eqiad, 06SRE, 06DC-Ops: wmcs PXE issues - https://phabricator.wikimedia.org/T433538 (10Jhancock.wm) 03NEW [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: May I have your attention please! UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T2000) [20:00:05] AaronSchulz: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:09] Thanks mutante! [20:04:45] !log pt1979@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage [20:05:00] * AaronSchulz looks around [20:06:41] who's running the deployment window then? [20:06:50] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1315010 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:07:47] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1315004 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:07:51] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1315005 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:07:52] I guess I'll go. It's just my stuf. [20:07:55] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1315006 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:07:59] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1315003 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:08:03] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1315019 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:08:06] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1315011 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:08:18] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1315013 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:08:21] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1315012 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:09:22] !log pt1979@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2097.codfw.wmnet with reason: host reimage [20:09:38] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1048.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [20:09:56] (03CR) 10TrainBranchBot: [C:03+2] "Approved by aaron@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306988 (https://phabricator.wikimedia.org/T433314) (owner: 10BPirkle) [20:10:06] (03CR) 10Ahmon Dancy: [C:03+1] "Tested manually on runner-1040.gitlab-runners" [puppet] - 10https://gerrit.wikimedia.org/r/1319146 (https://phabricator.wikimedia.org/T433505) (owner: 10Ahmon Dancy) [20:10:51] (03CR) 10Ahmon Dancy: [C:03+1] "PCC results: https://puppet-compiler.wmflabs.org/output/1319146/7405/" [puppet] - 10https://gerrit.wikimedia.org/r/1319146 (https://phabricator.wikimedia.org/T433505) (owner: 10Ahmon Dancy) [20:11:26] (03Merged) 10jenkins-bot: REST: use RestExternalModules config variable [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306988 (https://phabricator.wikimedia.org/T433314) (owner: 10BPirkle) [20:11:30] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1048.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [20:11:33] (03CR) 10Aaron Schulz: REST: use RestExternalModules config variable (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306988 (https://phabricator.wikimedia.org/T433314) (owner: 10BPirkle) [20:11:46] !log aaron@deploy1003 Started scap sync-world: Backport for [[gerrit:1306988|REST: use RestExternalModules config variable (T433314 T428375)]] [20:11:52] T433314: Migrate our config from RestSandboxSpecs to ExternalRestModules - https://phabricator.wikimedia.org/T433314 [20:11:53] T428375: REST: External Modules - https://phabricator.wikimedia.org/T428375 [20:12:24] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1048.eqiad.wmnet with OS trixie [20:12:42] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12169163 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie [20:13:46] !log aaron@deploy1003 bpirkle, aaron: Backport for [[gerrit:1306988|REST: use RestExternalModules config variable (T433314 T428375)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:15:48] !log aaron@deploy1003 bpirkle, aaron: Continuing with deployment [20:16:12] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 96779648 and 17 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [20:17:43] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1231: Maintenance [20:18:03] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance [20:18:11] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1253 (T431660)', diff saved to https://phabricator.wikimedia.org/P95719 and previous config saved to /var/cache/conftool/dbconfig/20260729-201810-cwilliams.json [20:19:53] !log aaron@deploy1003 Finished scap sync-world: Backport for [[gerrit:1306988|REST: use RestExternalModules config variable (T433314 T428375)]] (duration: 08m 07s) [20:19:59] T433314: Migrate our config from RestSandboxSpecs to ExternalRestModules - https://phabricator.wikimedia.org/T433314 [20:20:00] T428375: REST: External Modules - https://phabricator.wikimedia.org/T428375 [20:23:12] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3474352 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [20:24:14] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1253: Maintenance [20:25:21] (03PS2) 10Aaron Schulz: Use the openapi.json endpoint for the wikibase-rest/v1 REST module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312694 (https://phabricator.wikimedia.org/T422405) [20:25:30] (03CR) 10CI reject: [V:04-1] Use the openapi.json endpoint for the wikibase-rest/v1 REST module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312694 (https://phabricator.wikimedia.org/T422405) (owner: 10Aaron Schulz) [20:26:00] (03PS3) 10Aaron Schulz: Use the openapi.json endpoint for the wikibase-rest/v1 REST module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312694 (https://phabricator.wikimedia.org/T422405) [20:26:32] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12169221 (10VRiley-WMF) @fgiunchedi I was able to get through a good majority of this process, but I'm getting a hangup from the reimage. It seems like it's seeing... [20:29:01] (03CR) 10TrainBranchBot: [C:03+2] "Approved by aaron@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312694 (https://phabricator.wikimedia.org/T422405) (owner: 10Aaron Schulz) [20:30:21] (03Merged) 10jenkins-bot: Use the openapi.json endpoint for the wikibase-rest/v1 REST module [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312694 (https://phabricator.wikimedia.org/T422405) (owner: 10Aaron Schulz) [20:30:39] !log aaron@deploy1003 Started scap sync-world: Backport for [[gerrit:1312694|Use the openapi.json endpoint for the wikibase-rest/v1 REST module (T422405)]] [20:30:44] T422405: Surface externally managed Wikibase OAD in the REST Sandbox for projects where it is enabled - https://phabricator.wikimedia.org/T422405 [20:32:39] !log aaron@deploy1003 aaron: Backport for [[gerrit:1312694|Use the openapi.json endpoint for the wikibase-rest/v1 REST module (T422405)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:35:51] (03CR) 10Perryprog: Use os.environ.get for checking DJANGO_DEBUGGING (031 comment) [software/bitu] - 10https://gerrit.wikimedia.org/r/1319150 (owner: 10Perryprog) [20:41:06] hrm [20:48:14] !log aaron@deploy1003 aaron: Continuing with deployment [20:52:36] !log aaron@deploy1003 Finished scap sync-world: Backport for [[gerrit:1312694|Use the openapi.json endpoint for the wikibase-rest/v1 REST module (T422405)]] (duration: 21m 57s) [20:52:41] T422405: Surface externally managed Wikibase OAD in the REST Sandbox for projects where it is enabled - https://phabricator.wikimedia.org/T422405 [20:54:18] (03PS1) 10Aaron Schulz: Make the "wikibase-rest/v1" external module published [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319175 (https://phabricator.wikimedia.org/T422405) [20:55:57] (03PS2) 10Aaron Schulz: Make the "wikibase-rest/v1" external module published [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319175 (https://phabricator.wikimedia.org/T422405) [20:57:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by aaron@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319175 (https://phabricator.wikimedia.org/T422405) (owner: 10Aaron Schulz) [20:59:16] (03Merged) 10jenkins-bot: Make the "wikibase-rest/v1" external module published [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319175 (https://phabricator.wikimedia.org/T422405) (owner: 10Aaron Schulz) [20:59:34] !log aaron@deploy1003 Started scap sync-world: Backport for [[gerrit:1319175|Make the "wikibase-rest/v1" external module published (T422405)]] [20:59:38] T422405: Surface externally managed Wikibase OAD in the REST Sandbox for projects where it is enabled - https://phabricator.wikimedia.org/T422405 [21:00:05] Deploy window Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T2100) [21:01:45] !log aaron@deploy1003 aaron: Backport for [[gerrit:1319175|Make the "wikibase-rest/v1" external module published (T422405)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:02:12] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 256140936 and 13 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [21:05:38] (03CR) 10Dzahn: [C:03+2] "compiler output looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1319146 (https://phabricator.wikimedia.org/T433505) (owner: 10Ahmon Dancy) [21:08:25] !log aaron@deploy1003 aaron: Continuing with deployment [21:10:57] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1253: Maintenance [21:11:12] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 434720 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [21:12:07] !log bking@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs2015\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main [21:12:27] !log aaron@deploy1003 Finished scap sync-world: Backport for [[gerrit:1319175|Make the "wikibase-rest/v1" external module published (T422405)]] (duration: 12m 53s) [21:12:31] T422405: Surface externally managed Wikibase OAD in the REST Sandbox for projects where it is enabled - https://phabricator.wikimedia.org/T422405 [21:12:39] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2022.codfw.wmnet, repooling source-only afterwards [21:12:43] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [21:14:52] ok, done [21:16:09] !log zabe@deploy1003 mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=metawiki 'Mental Health Resource Center' 'Safety Resource Center/Mental Health' Zabe --reason 'per request [[:phab:T433118|T433118]]' [21:16:13] T433118: Request to move translatable page: Mental_Health_Resource_Center - https://phabricator.wikimedia.org/T433118 [21:16:30] (03PS11) 10Bking: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:17:09] (03CR) 10Bking: opensearch: ship the cirrus ECS server logs via rsyslog imfile (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:17:21] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:19:14] (03PS1) 10Clare Ming: Test Kitchen UI: Deploy v1.5.0 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319178 (https://phabricator.wikimedia.org/T432304) [21:19:28] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:21:57] 06SRE, 06ServiceOps new, 10VisualEditor, 10VisualEditor Suggestion Mode, and 3 others: Deploy Headless VE in k8s for technical pilot - https://phabricator.wikimedia.org/T431497#12169461 (10Scott_French) Thanks for the follow-up @Ottomata. >>! In T431497#12167389, @Ottomata wrote: > IIUC, there are 2 requ... [21:24:53] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12169463 (10Papaul) @MatthewVernon both nodes are failing with the same reason.Can you please check this? thank you ` 15:52:05 err '/usr/sbin/mkfs -t xfs... [21:25:02] !log pt1979@cumin2002 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2097.codfw.wmnet with OS bullseye [21:25:10] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12169464 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host ms-be2097.codfw.wmnet with OS bullseye executed... [21:25:16] 06SRE, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 10ServiceOps-Mediawiki, and 2 others: [SPIKE] Validate index.php Apache Behavior for canonical API Routing - https://phabricator.wikimedia.org/T433547 (10HCoplin-WMF) 03NEW [21:28:05] 06SRE, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 10ServiceOps-Mediawiki, and 2 others: [SPIKE] Validate index.php Apache Behavior for canonical API Routing - https://phabricator.wikimedia.org/T433547#12169488 (10HCoplin-WMF) [21:28:54] (03PS1) 10Cory Massaro: wikifunctions: Upgrade orchestrator from 2026-07-21-162301 to 2026-07-29-180129 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319180 [21:29:28] (03CR) 10Bking: opensearch: ship the cirrus ECS server logs via rsyslog imfile (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:30:52] 06SRE, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 10ServiceOps-Mediawiki, and 2 others: [SPIKE] Validate index.php Apache Behavior for canonical API Routing - https://phabricator.wikimedia.org/T433547#12169491 (10HCoplin-WMF) [21:32:36] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1048.eqiad.wmnet with OS trixie [21:32:51] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12169496 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie executed with er... [21:33:13] 06SRE, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 10ServiceOps-Mediawiki, and 2 others: [SPIKE] Validate index.php Apache Behavior for canonical API Routing - https://phabricator.wikimedia.org/T433547#12169497 (10HCoplin-WMF) [21:33:28] (03PS12) 10Bking: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:37:27] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:46:42] (03PS13) 10Bking: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:48:50] (03PS14) 10Bking: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:48:57] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:49:45] (03CR) 10Bking: "Updated the `hosts:` line since PCC didn't try to run against the prod cirrussearch hosts." [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:52:49] (03CR) 10Cory Massaro: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-07-21-162301 to 2026-07-29-180129 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319180 (owner: 10Cory Massaro) [21:53:01] (03CR) 10Bking: [C:03+1] opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [21:53:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:55:01] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-07-21-162301 to 2026-07-29-180129 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319180 (owner: 10Cory Massaro) [21:55:12] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 1649213416 and 116 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [21:55:55] (03PS2) 10Clare Ming: Test Kitchen UI: Deploy v1.5.0 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319178 (https://phabricator.wikimedia.org/T432304) [21:58:05] !log apine@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [21:58:25] !log apine@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [21:58:59] !log apine@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [21:59:38] !log apine@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T2200) [22:00:39] !log apine@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [22:01:22] !log apine@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [22:02:12] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 84952 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [22:20:25] !log brett@cumin2002 START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: Switch upgrade maintenance window, T433097] [22:20:29] T433097: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097 [22:20:42] !log brett@cumin2002 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: Switch upgrade maintenance window, T433097] [22:25:39] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2015.codfw.wmnet -> wdqs2022.codfw.wmnet, repooling source-only afterwards [22:25:44] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [22:33:13] 06SRE, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 10ServiceOps-Mediawiki, and 2 others: [SPIKE] Validate index.php Apache Behavior for canonical API Routing - https://phabricator.wikimedia.org/T433547#12169588 (10KineticPelagic) As part of my Clinic Duty rotation, I am moving this task because it... [22:49:04] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [22:53:04] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [22:53:34] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [22:53:35] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [22:54:05] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [23:05:12] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 67086528 and 8 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [23:06:14] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3771032 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [23:17:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:42:31] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1319187 [23:42:31] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1319187 (owner: 10TrainBranchBot) [23:42:50] PROBLEM - Host asw1-eqsin is DOWN: PING CRITICAL - Packet loss = 100% [23:43:14] RECOVERY - Host asw1-eqsin is UP: PING OK - Packet loss = 0%, RTA = 211.48 ms [23:46:16] PROBLEM - Host ps1-604-eqsin is DOWN: PING CRITICAL - Packet loss = 100% [23:46:34] RECOVERY - Host ps1-604-eqsin is UP: PING OK - Packet loss = 0%, RTA = 211.72 ms [23:46:50] PROBLEM - Host scs-eqsin is DOWN: PING CRITICAL - Packet loss = 100% [23:48:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [23:50:26] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [23:51:52] RECOVERY - Host scs-eqsin is UP: PING OK - Packet loss = 0%, RTA = 211.07 ms [23:54:24] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1319187 (owner: 10TrainBranchBot) [23:56:06] (03PS1) 10Aleksandar Mastilovic: Change severity of data eng Airflow alerts to "page" [alerts] - 10https://gerrit.wikimedia.org/r/1319189 (https://phabricator.wikimedia.org/T432312) [23:57:42] !log pt1979@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mr1-eqsin,mr1-eqsin IPv6,mr1-eqsin.oob,mr1-eqsin.oob IPv6 with reason: connection issue [23:57:52] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12169750 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0fbc87b8-270e-465d-9d10-b229a17d4554) set by pt1979@cumin1003 for 2:00:00... [23:58:42] (03CR) 10CI reject: [V:04-1] Change severity of data eng Airflow alerts to "page" [alerts] - 10https://gerrit.wikimedia.org/r/1319189 (https://phabricator.wikimedia.org/T432312) (owner: 10Aleksandar Mastilovic) [23:58:44] RESOLVED: RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable