[00:12:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:09:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 14.36% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:10:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1134:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1134 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [01:13:40] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:15:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1134:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1134 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [01:19:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:21:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1134:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1134 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [01:26:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1134:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1134 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [02:00:17] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:08:50] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 08m 33s) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:17:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab-replica-b.discovery.wmnet in eqsin #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqsin&var-cluster=text&var-origin=gitlab-replica-b.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [02:22:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab-replica-b.discovery.wmnet in eqsin #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqsin&var-cluster=text&var-origin=gitlab-replica-b.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [02:50:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [02:50:38] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:12:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:18:25] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:00:23] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:11:32] (03PS3) 10Tryvix1509: Adjust extendedconfirmed calculation to first edit on viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) [06:11:55] (03CR) 10Tryvix1509: "Done" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) (owner: 10Tryvix1509) [06:14:46] !log installing Chromium security updates [06:14:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:50:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:58:33] hello o/ [07:00:04] Amir1, urbanecm, and awight: OwO what's this, a deployment window?? UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T0700). nyaa~ [07:00:05] abijeet and Hamishcz: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:47] I'll be deploying abijeet's change. [07:01:11] abijeet: not sure why scap is showing dependency warning. It seems it is wrong. [07:02:04] We've wmf.19, warning shows that patch isn't in wmf.17 :/ [07:02:42] kart_, [07:02:55] that's weird [07:03:01] I will deploy it and let's test it. [07:03:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kartik@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329314 (https://phabricator.wikimedia.org/T433483) (owner: 10Abijeet Patro) [07:04:05] (03PS1) 10Muehlenhoff: proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337443 [07:04:38] (03Merged) 10jenkins-bot: ArticleGuidance: Configure feedback links to local talk pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329314 (https://phabricator.wikimedia.org/T433483) (owner: 10Abijeet Patro) [07:05:12] !log kartik@deploy1003 Started scap sync-world: Backport for [[gerrit:1329314|ArticleGuidance: Configure feedback links to local talk pages (T433483)]] [07:05:15] T433483: Redirect "suggest a new type" to per-wiki talk page with localised message template - https://phabricator.wikimedia.org/T433483 [07:06:39] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2196 crashed - https://phabricator.wikimedia.org/T437015#12293577 (10Marostegui) >>! In T437015#12290030, @Jhancock.wm wrote: > @Marostegui I'll get this one added to the thread and start updating firmwares. Let me know when this has been done and all the required r... [07:14:52] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:xe-1/0/1:0 (Transport: cr2-eqord:xe-0/1/5 (Arelion, IC-314533 24ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:17:00] (03CR) 10Dpogorzelski: [C:03+2] admin_ng: add liftwing-studio namespace to dse-k8s-eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335898 (owner: 10Dpogorzelski) [07:23:40] !log kartik@deploy1003 abi, kartik: Backport for [[gerrit:1329314|ArticleGuidance: Configure feedback links to local talk pages (T433483)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:23:44] T433483: Redirect "suggest a new type" to per-wiki talk page with localised message template - https://phabricator.wikimedia.org/T433483 [07:23:52] abijeet: can you test please? [07:24:16] (03PS1) 10Marostegui: control-mariadb-10.11-bookworm: New version [software] - 10https://gerrit.wikimedia.org/r/1337444 (https://phabricator.wikimedia.org/T436082) [07:25:39] (03CR) 10Marostegui: [C:03+2] control-mariadb-10.11-bookworm: New version [software] - 10https://gerrit.wikimedia.org/r/1337444 (https://phabricator.wikimedia.org/T436082) (owner: 10Marostegui) [07:26:11] (03Merged) 10jenkins-bot: control-mariadb-10.11-bookworm: New version [software] - 10https://gerrit.wikimedia.org/r/1337444 (https://phabricator.wikimedia.org/T436082) (owner: 10Marostegui) [07:29:54] 06SRE, 10SRE-Access-Requests: Requesting access to production for derenrich - https://phabricator.wikimedia.org/T436965#12293594 (10Volans) [07:32:05] abijeet: We need to test the patch, are you around? [07:33:16] kart_, sorry, on it [07:33:21] cool. [07:34:28] 06SRE, 10Wikimedia-Mailing-lists: Create new mailing lists: wikidata-admins@lists.wikimedia.org - https://phabricator.wikimedia.org/T435638#12293604 (10Jony) @Ladsgroup thanks! [07:34:36] kart_, ok looks good [07:34:41] nice! [07:34:45] Deploying.. [07:34:51] !log kartik@deploy1003 abi, kartik: Continuing with deployment [07:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.32% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:42:49] (03CR) 10Muehlenhoff: [C:03+2] proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337443 (owner: 10Muehlenhoff) [07:42:58] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337443 (owner: 10Muehlenhoff) [07:45:43] !log jmm@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [07:46:35] !log jmm@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [07:46:44] (03CR) 10Arnaudb: [C:03+2] releases-jenkins: do not use HTTP proxy outside of wmnet [puppet] - 10https://gerrit.wikimedia.org/r/1335868 (owner: 10Jaime Nuche) [07:47:03] !log kartik@deploy1003 Finished scap sync-world: Backport for [[gerrit:1329314|ArticleGuidance: Configure feedback links to local talk pages (T433483)]] (duration: 41m 51s) [07:47:06] T433483: Redirect "suggest a new type" to per-wiki talk page with localised message template - https://phabricator.wikimedia.org/T433483 [07:50:44] !log jmm@deploy1003 helmfile [codfw] START helmfile.d/services/proton: apply [07:51:58] !log jmm@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: apply [07:52:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:53:16] (03PS1) 10Marostegui: mariadb: Move db1241 to s6 [puppet] - 10https://gerrit.wikimedia.org/r/1337447 (https://phabricator.wikimedia.org/T434782) [07:54:36] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1180: Cloning [07:54:41] (03CR) 10Marostegui: [C:03+2] mariadb: Move db1241 to s6 [puppet] - 10https://gerrit.wikimedia.org/r/1337447 (https://phabricator.wikimedia.org/T434782) (owner: 10Marostegui) [07:54:50] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1180: Cloning [07:55:03] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1241: Cloning [07:55:36] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1241: Cloning [07:56:17] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Cloning [07:57:02] thanks, kart_ [07:57:49] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1180.eqiad.wmnet onto db1241.eqiad.wmnet [07:58:17] abijeet: :) [08:00:46] !log jmm@deploy1003 helmfile [eqiad] START helmfile.d/services/proton: apply [08:02:04] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [08:02:06] !log jmm@deploy1003 helmfile [eqiad] DONE helmfile.d/services/proton: apply [08:03:02] (03PS1) 10STran: SI: Use new InfoChip text class in case status updater [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337450 (https://phabricator.wikimedia.org/T437020) [08:03:27] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [08:03:35] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 07 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337450 (https://phabricator.wikimedia.org/T437020) (owner: 10STran) [08:03:50] (03CR) 10Dpogorzelski: [C:03+2] admin_ng: add liftwing-studio to the ceph-rbd CSI tenant namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335899 (owner: 10Dpogorzelski) [08:03:59] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host aqs1022.eqiad.wmnet with OS bookworm [08:08:07] FIRING: [2x] ProbeDown: Service aqs1022-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:09:12] (03CR) 10EMcFarland: [C:03+1] "I can't do the final +2, but this is approved on my end." [puppet] - 10https://gerrit.wikimedia.org/r/1333291 (https://phabricator.wikimedia.org/T436659) (owner: 10Urbanecm) [08:12:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:13:07] FIRING: [4x] ProbeDown: Service aqs1022-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:14:35] (03CR) 10Muehlenhoff: [C:03+2] New r/o LDAP replicas on Trixie/MDB [puppet] - 10https://gerrit.wikimedia.org/r/1335830 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [08:15:06] (03Merged) 10jenkins-bot: admin_ng: add liftwing-studio to the ceph-rbd CSI tenant namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335899 (owner: 10Dpogorzelski) [08:20:51] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [08:22:03] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [08:23:07] RESOLVED: [4x] ProbeDown: Service aqs1022-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:27:49] (03CR) 10Dpogorzelski: [C:03+2] lw studio: add dedicated pgsql [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335793 (owner: 10Dpogorzelski) [08:28:21] !log jmm@cumin1004 START - Cookbook sre.ganeti.makevm for new host ldap-replica1005.wikimedia.org [08:28:23] !log jmm@cumin1004 START - Cookbook sre.dns.netbox [08:30:12] (03Merged) 10jenkins-bot: lw studio: add dedicated pgsql [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335793 (owner: 10Dpogorzelski) [08:31:35] (03PS1) 10Gerrit maintenance bot: mariadb: Promote db2157 to s5 master [puppet] - 10https://gerrit.wikimedia.org/r/1337452 (https://phabricator.wikimedia.org/T437188) [08:32:36] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM, nice!" [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [08:33:58] jmm@cumin1004 makevm (PID 858122) is awaiting input [08:34:35] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s5 T437188 [08:34:37] (03PS2) 10Trueg: WDQS: Improved resource limits for Qlever on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335771 (https://phabricator.wikimedia.org/T436295) [08:34:38] T437188: Switchover s5 master (db2213 -> db2157) - https://phabricator.wikimedia.org/T437188 [08:34:49] !log marostegui@cumin1003 dbctl commit (dc=all): 'Set db2157 with weight 0 T437188', diff saved to https://phabricator.wikimedia.org/P96353 and previous config saved to /var/cache/conftool/dbconfig/20260907-083448-marostegui.json [08:34:52] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:xe-1/0/1:0 (Transport: cr2-eqord:xe-0/1/5 (Arelion, IC-314533 24ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [08:35:21] (03PS1) 10Ayounsi: Add support for multiple host queries coma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 [08:35:47] (03CR) 10Marostegui: [C:03+2] mariadb: Promote db2157 to s5 master [puppet] - 10https://gerrit.wikimedia.org/r/1337452 (https://phabricator.wikimedia.org/T437188) (owner: 10Gerrit maintenance bot) [08:36:02] (03CR) 10Ayounsi: "recheck" [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 (owner: 10Ayounsi) [08:36:52] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1337454 [08:36:52] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1337454 (owner: 10TrainBranchBot) [08:37:13] jouncebot: nowandnext [08:37:13] No deployments scheduled for the next 1 hour(s) and 22 minute(s) [08:37:13] In 1 hour(s) and 22 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1000) [08:37:40] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica1005.wikimedia.org - jmm@cumin1004" [08:37:43] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica1005.wikimedia.org - jmm@cumin1004" [08:37:43] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:37:43] !log jmm@cumin1004 START - Cookbook sre.dns.wipe-cache ldap-replica1005.wikimedia.org on all recursors [08:37:46] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ldap-replica1005.wikimedia.org on all recursors [08:38:02] !log Starting s5 codfw failover from db2213 to db2157 - T437188 [08:38:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:38:17] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica1005.wikimedia.org - jmm@cumin1004" [08:38:21] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica1005.wikimedia.org - jmm@cumin1004" [08:38:26] !log marostegui@cumin1003 dbctl commit (dc=all): 'Promote db2157 to s5 primary T437188', diff saved to https://phabricator.wikimedia.org/P96354 and previous config saved to /var/cache/conftool/dbconfig/20260907-083825-marostegui.json [08:38:38] (03PS1) 10Urbanecm: Mentorship: Don't renew mentors' away status on every cleaner run [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337455 (https://phabricator.wikimedia.org/T436659) [08:38:49] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337455 (https://phabricator.wikimedia.org/T436659) (owner: 10Urbanecm) [08:38:59] !log jmm@cumin1004 START - Cookbook sre.hosts.reimage for host ldap-replica1005.wikimedia.org with OS trixie [08:39:05] !log marostegui@cumin1003 dbctl commit (dc=all): 'Depool db2213 T437188', diff saved to https://phabricator.wikimedia.org/P96355 and previous config saved to /var/cache/conftool/dbconfig/20260907-083904-marostegui.json [08:39:13] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12293831 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host ldap-replica1005.wikimedia.org with... [08:39:26] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover [08:40:25] !log marostegui@cumin1003 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2213: Repooling after switchover [08:40:35] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover [08:40:56] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 2 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [08:40:57] (03PS1) 10Filippo Giunchedi: Revert "site: temp decom for cloudvirt10(6[567]|7[234]) ahead of rack move" [puppet] - 10https://gerrit.wikimedia.org/r/1337456 [08:41:01] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [08:41:30] PROBLEM - pt-heartbeat-wikimedia process on db2213 is CRITICAL: PROCS CRITICAL: 0 processes with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [08:42:04] (03PS1) 10Trueg: wdqs: extend envoy rq_time histogram buckets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) (owner: 10Gmodena) [08:42:04] (03CR) 10Trueg: [C:03+1] "AFAICT you merely add new buckets for query times above 2500ms, that looks good to me." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) (owner: 10Gmodena) [08:42:15] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [08:42:23] (03CR) 10Trueg: [C:03+2] WDQS: Do not keep any index backups on the PVC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335776 (https://phabricator.wikimedia.org/T436802) (owner: 10Trueg) [08:42:46] (03PS1) 10Marostegui: mariadb: Change s5 codfw candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1337457 (https://phabricator.wikimedia.org/T436078) [08:43:21] mvernon@cumin2003 reimage (PID 4085360) is awaiting input [08:43:32] (03CR) 10Marostegui: "Changed in dbctl" [puppet] - 10https://gerrit.wikimedia.org/r/1337457 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [08:43:35] (03CR) 10Marostegui: [C:03+2] mariadb: Change s5 codfw candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1337457 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [08:44:15] (03PS1) 10Urbanecm: Ignore PersonalDashboard stubs [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337458 (https://phabricator.wikimedia.org/T428679) [08:44:23] (03CR) 10CI reject: [V:04-1] Mentorship: Don't renew mentors' away status on every cleaner run [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337455 (https://phabricator.wikimedia.org/T436659) (owner: 10Urbanecm) [08:44:41] (03PS2) 10Urbanecm: Mentorship: Don't renew mentors' away status on every cleaner run [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337455 (https://phabricator.wikimedia.org/T436659) [08:44:41] (03CR) 10Filippo Giunchedi: "Sample prom file: https://phabricator.wikimedia.org/P96357" [puppet] - 10https://gerrit.wikimedia.org/r/1331487 (https://phabricator.wikimedia.org/T436252) (owner: 10Filippo Giunchedi) [08:44:49] (03CR) 10Urbanecm: [C:03+2] Ignore PersonalDashboard stubs [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337458 (https://phabricator.wikimedia.org/T428679) (owner: 10Urbanecm) [08:44:52] (03CR) 10Trueg: [C:03+2] wdqs: Utilize the wdqs-proxy's readiness probe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333130 (https://phabricator.wikimedia.org/T432331) (owner: 10Trueg) [08:44:54] (03CR) 10Urbanecm: [C:03+2] Mentorship: Don't renew mentors' away status on every cleaner run [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337455 (https://phabricator.wikimedia.org/T436659) (owner: 10Urbanecm) [08:45:18] (03CR) 10Trueg: [C:03+2] WDQS: Simplification of qlever proxy values [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335712 (https://phabricator.wikimedia.org/T429175) (owner: 10Trueg) [08:45:28] (03Merged) 10jenkins-bot: WDQS: Do not keep any index backups on the PVC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335776 (https://phabricator.wikimedia.org/T436802) (owner: 10Trueg) [08:46:10] (03CR) 10TrainBranchBot: "Approved by urbanecm@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337455 (https://phabricator.wikimedia.org/T436659) (owner: 10Urbanecm) [08:46:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337458 (https://phabricator.wikimedia.org/T428679) (owner: 10Urbanecm) [08:46:13] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for eventgate-logging-external in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334801 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:46:24] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for eventgate-logging-external in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334802 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:46:44] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for eventgate-analytics* in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334803 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:46:59] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for eventgate-analytics* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334804 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:47:17] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for eventgate-main in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334805 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:47:26] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1337454 (owner: 10TrainBranchBot) [08:47:31] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for eventgate-main in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334806 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:47:49] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (NOOP 3): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9368/console" [puppet] - 10https://gerrit.wikimedia.org/r/1335928 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [08:47:59] (03Merged) 10jenkins-bot: wdqs: Utilize the wdqs-proxy's readiness probe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333130 (https://phabricator.wikimedia.org/T432331) (owner: 10Trueg) [08:48:05] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1022.eqiad.wmnet with reason: host reimage [08:48:21] (03Merged) 10jenkins-bot: WDQS: Simplification of qlever proxy values [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335712 (https://phabricator.wikimedia.org/T429175) (owner: 10Trueg) [08:48:47] (03CR) 10Volans: "It could be a nice addition. One optional easy expansion inline." [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 (owner: 10Ayounsi) [08:49:36] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1180: Pool db1180.eqiad.wmnet in after cloning [08:52:30] (03PS3) 10Filippo Giunchedi: cloudnfs: add textfile exporter for extra nfsd metrics [puppet] - 10https://gerrit.wikimedia.org/r/1331487 (https://phabricator.wikimedia.org/T436252) [08:52:35] !log jmm@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-replica1005.wikimedia.org with reason: host reimage [08:52:42] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1022.eqiad.wmnet with reason: host reimage [08:53:28] (03Merged) 10jenkins-bot: Ignore PersonalDashboard stubs [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337458 (https://phabricator.wikimedia.org/T428679) (owner: 10Urbanecm) [08:53:31] (03Merged) 10jenkins-bot: Mentorship: Don't renew mentors' away status on every cleaner run [extensions/GrowthExperiments] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337455 (https://phabricator.wikimedia.org/T436659) (owner: 10Urbanecm) [08:54:51] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1337455|Mentorship: Don't renew mentors' away status on every cleaner run (T436659)]], [[gerrit:1337458|Ignore PersonalDashboard stubs (T428679)]] [08:54:57] T436659: Unnecessary mentor status updates - https://phabricator.wikimedia.org/T436659 [08:54:57] T428679: PersonalDashboard: Support Test Kitchen experiment enrollment and variant-aware instrumentation - https://phabricator.wikimedia.org/T428679 [08:55:38] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync [08:55:39] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync [08:56:05] (03PS6) 10Elukey: docker_registry: move away from Swift completely [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) [08:56:23] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-replica1005.wikimedia.org with reason: host reimage [08:56:35] (03CR) 10CI reject: [V:04-1] Add Python 3.14 support [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 (owner: 10Ayounsi) [08:56:54] !log jmm@cumin1004 START - Cookbook sre.hosts.reboot-single for host maps-test2001.codfw.wmnet [08:57:45] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [08:59:02] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [09:00:15] (03CR) 10Slyngshede: [C:03+2] Varnish: Reject non-thumb requests to thumbs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [09:00:37] (03Abandoned) 10Slyngshede: P:debmonitor::server Add CDN endpoint check. [puppet] - 10https://gerrit.wikimedia.org/r/1003409 (https://phabricator.wikimedia.org/T350694) (owner: 10Slyngshede) [09:00:52] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12294046 (10MoritzMuehlenhoff) [09:01:07] !log urbanecm@deploy1003 urbanecm: Backport for [[gerrit:1337455|Mentorship: Don't renew mentors' away status on every cleaner run (T436659)]], [[gerrit:1337458|Ignore PersonalDashboard stubs (T428679)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:01:12] T436659: Unnecessary mentor status updates - https://phabricator.wikimedia.org/T436659 [09:01:12] T428679: PersonalDashboard: Support Test Kitchen experiment enrollment and variant-aware instrumentation - https://phabricator.wikimedia.org/T428679 [09:02:07] !log installing giflib security updates [09:02:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:03:16] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps-test2001.codfw.wmnet [09:04:06] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [09:04:39] (03Abandoned) 10Slyngshede: P:idp More precise base_dn for user lookup [puppet] - 10https://gerrit.wikimedia.org/r/1060396 (https://phabricator.wikimedia.org/T371930) (owner: 10Slyngshede) [09:06:35] (03CR) 10Slyngshede: [C:03+1] "I'm voting OK on this one. It's much simpler." [puppet] - 10https://gerrit.wikimedia.org/r/1298885 (owner: 10BCornwall) [09:08:15] !log urbanecm@deploy1003 urbanecm: Continuing with deployment [09:10:01] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1022.eqiad.wmnet with OS bookworm [09:10:04] !log ayounsi@cumin1003 START - Cookbook sre.network.peering with action 'configure' for AS: 139009 [09:10:19] !log rebuild software RAID following disk replacement T437036 [09:10:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:10:22] T437036: Broken disk on maps-test2001 - https://phabricator.wikimedia.org/T437036 [09:10:31] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-replica1005.wikimedia.org with OS trixie [09:10:31] !log jmm@cumin1004 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ldap-replica1005.wikimedia.org [09:10:44] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12294065 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host ldap-replica1005.wikimedia.org with OS t... [09:12:15] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 139009 [09:13:27] (03CR) 10David Caro: [C:03+1] "LGTM, just some questions, feel free to merge as is if the answers point that way :)" [puppet] - 10https://gerrit.wikimedia.org/r/1331487 (https://phabricator.wikimedia.org/T436252) (owner: 10Filippo Giunchedi) [09:13:37] (03CR) 10Brouberol: "adding @bking@wikimedia.org and @rkemper@wikimedia.org as reviewers, as they are our resident opensearch experts" [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (https://phabricator.wikimedia.org/T435265) (owner: 10Herron) [09:13:56] (03CR) 10Brouberol: [C:03+1] Puppet 8: Replace unscoped legacy facts in module query_service [puppet] - 10https://gerrit.wikimedia.org/r/1335930 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [09:13:59] (03PS1) 10Arnaudb: gitlab: exclude gitlab-replicas from ATS paging [alerts] - 10https://gerrit.wikimedia.org/r/1337543 (https://phabricator.wikimedia.org/T437189) [09:15:03] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337455|Mentorship: Don't renew mentors' away status on every cleaner run (T436659)]], [[gerrit:1337458|Ignore PersonalDashboard stubs (T428679)]] (duration: 20m 12s) [09:15:08] T436659: Unnecessary mentor status updates - https://phabricator.wikimedia.org/T436659 [09:15:09] T428679: PersonalDashboard: Support Test Kitchen experiment enrollment and variant-aware instrumentation - https://phabricator.wikimedia.org/T428679 [09:15:09] (03Abandoned) 10Slyngshede: Remove ip_reputation_vendors from puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/1230914 (https://phabricator.wikimedia.org/T365798) (owner: 10Muehlenhoff) [09:15:23] (03CR) 10Jelto: [C:03+1] "lgtm, thank you" [alerts] - 10https://gerrit.wikimedia.org/r/1337543 (https://phabricator.wikimedia.org/T437189) (owner: 10Arnaudb) [09:15:39] (03CR) 10Arnaudb: [C:03+2] gitlab: exclude gitlab-replicas from ATS paging [alerts] - 10https://gerrit.wikimedia.org/r/1337543 (https://phabricator.wikimedia.org/T437189) (owner: 10Arnaudb) [09:16:23] (03Abandoned) 10Slyngshede: Function:htpasswd Avoid initializing variable multiple times. [puppet] - 10https://gerrit.wikimedia.org/r/936213 (https://phabricator.wikimedia.org/T228966) (owner: 10Slyngshede) [09:16:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:17:45] (03Merged) 10jenkins-bot: gitlab: exclude gitlab-replicas from ATS paging [alerts] - 10https://gerrit.wikimedia.org/r/1337543 (https://phabricator.wikimedia.org/T437189) (owner: 10Arnaudb) [09:18:40] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:19:32] (03CR) 10Elukey: [C:03+2] mediawiki::maintenance::growthexperiments: Document interval expectations [puppet] - 10https://gerrit.wikimedia.org/r/1333291 (https://phabricator.wikimedia.org/T436659) (owner: 10Urbanecm) [09:23:40] !log blake@cumin1003 START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-codfw [09:24:11] (03CR) 10Btullis: [C:03+2] Add the upstream ceph-squid repository bookworm-wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/1335835 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [09:25:42] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2213: Repooling after switchover [09:31:57] (03CR) 10Ayounsi: [C:03+2] makevm: add --force to reimage cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1334770 (owner: 10Ayounsi) [09:32:19] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [09:32:33] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [09:34:39] (03Merged) 10jenkins-bot: makevm: add --force to reimage cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1334770 (owner: 10Ayounsi) [09:34:45] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1180: Pool db1180.eqiad.wmnet in after cloning [09:36:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:37:23] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [09:37:29] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [09:38:26] (03CR) 10JMeybohm: [C:03+1] "🎉" [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) (owner: 10Elukey) [09:38:58] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [09:40:20] jouncebot: nowandnext [09:40:20] No deployments scheduled for the next 0 hour(s) and 19 minute(s) [09:40:20] In 0 hour(s) and 19 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1000) [09:41:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337379 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [09:41:32] !log aokoth@deploy1003 helmfile [codfw] START helmfile.d/services/miscweb: apply [09:42:10] (03Merged) 10jenkins-bot: Enable thumb.wikimedia.org everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337379 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [09:42:24] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1337379|Enable thumb.wikimedia.org everywhere (T427465)]] [09:42:27] T427465: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465 [09:43:36] jmm@cumin1004 makevm (PID 873441) is awaiting input [09:44:42] !log aokoth@deploy1003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [09:45:45] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [09:46:56] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1337379|Enable thumb.wikimedia.org everywhere (T427465)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:47:20] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12294166 (10MoritzMuehlenhoff) bullseye-security was restored on the Debian archive side, please check if there are still any issues. [09:47:31] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Q1:rack/setup/install dse-k8s-worker10[29-38] - https://phabricator.wikimedia.org/T436964#12294167 (10BTullis) a:05BTullis→03None I have updated preseed.yaml and site.pp for these new nodes, so I think that I'm done for now.... [09:47:45] (03PS2) 10Ayounsi: Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 [09:47:48] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for eventgate-logging-external in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334801 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [09:47:56] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [09:48:01] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [09:48:20] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [09:48:20] (03PS1) 10Arnaudb: gitlab: silence ATSBackendErrorsHigh during primary upgrades [cookbooks] - 10https://gerrit.wikimedia.org/r/1337547 (https://phabricator.wikimedia.org/T437189) [09:48:49] !log aokoth@deploy1003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [09:49:00] !log jmm@cumin1004 START - Cookbook sre.ganeti.makevm for new host ldap-replica1006.wikimedia.org [09:49:02] !log jmm@cumin1004 START - Cookbook sre.dns.netbox [09:50:03] (03Abandoned) 10Muehlenhoff: thumbor-plugins: Stop using pkg_resources [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1243135 (owner: 10Muehlenhoff) [09:50:20] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw [09:50:27] (03Abandoned) 10Muehlenhoff: Rebuild for trixie [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1306219 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [09:51:26] !log aokoth@deploy1003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [09:51:38] (03CR) 10Samwilson: [C:03+1] Clean up requirements and add a dependency bump path [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333251 (owner: 10Ladsgroup-claude) [09:51:57] (03CR) 10Ayounsi: Add support for multiple host queries comma separated (032 comments) [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 (owner: 10Ayounsi) [09:52:36] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337379|Enable thumb.wikimedia.org everywhere (T427465)]] (duration: 10m 11s) [09:52:38] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [09:52:39] T427465: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465 [09:53:05] (03CR) 10Ladsgroup: [C:03+2] Clean up requirements and add a dependency bump path [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333251 (owner: 10Ladsgroup-claude) [09:53:31] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica1006.wikimedia.org - jmm@cumin1004" [09:53:43] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [09:54:40] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [09:54:40] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw [09:56:00] (03CR) 10Volans: [C:03+1] "We've chatted offline with Arzhel, I support this approach. LGTM" [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 (owner: 10Ayounsi) [09:56:19] !log aokoth@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply [09:56:31] (03CR) 10Clément Goubert: [C:03+1] "LGTM pending puppet + dns changes" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [09:56:34] jmm@cumin1004 makevm (PID 873441) is awaiting input [09:56:46] !log aokoth@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply [09:57:02] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica1006.wikimedia.org - jmm@cumin1004" [09:57:02] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:57:02] !log jmm@cumin1004 START - Cookbook sre.dns.wipe-cache ldap-replica1006.wikimedia.org on all recursors [09:57:05] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ldap-replica1006.wikimedia.org on all recursors [09:57:36] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica1006.wikimedia.org - jmm@cumin1004" [09:57:39] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica1006.wikimedia.org - jmm@cumin1004" [09:57:45] !log jmm@cumin1004 START - Cookbook sre.hosts.reimage for host ldap-replica1006.wikimedia.org with OS trixie [09:57:58] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12294181 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host ldap-replica1006.wikimedia.org with... [09:58:05] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [09:58:50] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [09:59:38] (03CR) 10Majavah: [C:03+2] cloudnfs: Remove TWL NFS configuration [puppet] - 10https://gerrit.wikimedia.org/r/1325912 (https://phabricator.wikimedia.org/T402054) (owner: 10Majavah) [09:59:48] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [09:59:54] (03Merged) 10jenkins-bot: Clean up requirements and add a dependency bump path [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333251 (owner: 10Ladsgroup-claude) [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1000) [10:00:07] (03PS1) 10Muehlenhoff: Depool puppetserver2002 for reboot [dns] - 10https://gerrit.wikimedia.org/r/1337549 [10:00:20] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for eventgate-logging-external in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334802 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [10:00:36] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [10:00:37] (03PS2) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-logging-external in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334802 (https://phabricator.wikimedia.org/T420436) [10:00:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:02:26] !log mvernon@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs2001.codfw.wmnet [10:03:03] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9370/console" [puppet] - 10https://gerrit.wikimedia.org/r/1335901 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [10:03:20] (03PS2) 10Blake: memcached: continue pki rollout [puppet] - 10https://gerrit.wikimedia.org/r/1337548 (https://phabricator.wikimedia.org/T353511) [10:03:39] (03CR) 10Majavah: [V:03+1 C:03+1] Puppet 8: Replace unscoped legacy facts in module galera [puppet] - 10https://gerrit.wikimedia.org/r/1335901 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [10:04:07] (03CR) 10CI reject: [V:04-1] Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 (owner: 10Ayounsi) [10:04:21] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [10:05:01] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9371/console" [puppet] - 10https://gerrit.wikimedia.org/r/1335917 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [10:05:26] (03CR) 10Majavah: [V:03+1 C:03+1] Puppet 8: Replace unscoped legacy facts in module openstack [puppet] - 10https://gerrit.wikimedia.org/r/1335917 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [10:05:46] (03CR) 10Muehlenhoff: "Works like a charm" [cookbooks] - 10https://gerrit.wikimedia.org/r/1334770 (owner: 10Ayounsi) [10:06:09] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [10:06:35] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [10:06:37] (03PS1) 10Zabe: rdbms: Discourage setting db domain to false for remote virtual domains [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337550 (https://phabricator.wikimedia.org/T422940) [10:06:45] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for eventgate-logging-external in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334802 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [10:07:33] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad [10:07:51] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [10:08:02] (03PS2) 10Brouberol: Deduplicate the proxy configuration value [deployment-charts] - 10https://gerrit.wikimedia.org/r/1328403 [10:08:56] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [10:09:41] (03PS3) 10Trueg: WDQS: Improved resource limits for Qlever on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335771 (https://phabricator.wikimedia.org/T436295) [10:09:55] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [10:10:47] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [10:10:48] (03CR) 10FNegri: [V:03+1] "LGTM. I tried running PCC, but it does not do erb compilation... Please merge this and I'll double check the generated file on a live host" [puppet] - 10https://gerrit.wikimedia.org/r/1335918 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [10:11:42] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [10:11:42] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad [10:12:51] !log jmm@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-replica1006.wikimedia.org with reason: host reimage [10:12:55] (03CR) 10FNegri: [V:03+1] "LGTM. PCC is clean, but I don't remember if PCC actually does erb compilation... Please merge this and I'll double check the generated fil" [puppet] - 10https://gerrit.wikimedia.org/r/1335918 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [10:12:57] (03PS1) 10Hashar: Print log file for any action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337551 [10:13:02] (03PS3) 10Ayounsi: Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 [10:13:02] !log blake@cumin1003 END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-codfw [10:15:09] mvernon@cumin2003 upgrade-firmware (PID 4110475) is awaiting input [10:15:28] (03CR) 10Blake: [C:03+2] memcached: continue pki rollout [puppet] - 10https://gerrit.wikimedia.org/r/1337548 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [10:16:01] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host aqs2001.codfw.wmnet [10:17:46] (03CR) 10CI reject: [V:04-1] Print log file for any action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337551 (owner: 10Hashar) [10:18:02] (03PS1) 10Majavah: openstack: wikireplica_dns: Add user-facing names for x4 databases [puppet] - 10https://gerrit.wikimedia.org/r/1337552 (https://phabricator.wikimedia.org/T437208) [10:18:17] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-replica1006.wikimedia.org with reason: host reimage [10:19:25] FIRING: [3x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:20:05] (03PS2) 10Hashar: Print log file for any action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337551 [10:20:37] (03CR) 10FNegri: [C:03+1] openstack: wikireplica_dns: Add user-facing names for x4 databases [puppet] - 10https://gerrit.wikimedia.org/r/1337552 (https://phabricator.wikimedia.org/T437208) (owner: 10Majavah) [10:21:31] (03PS1) 10Trueg: WDQS: Bump wdqs-proxy version to 0.8.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337553 (https://phabricator.wikimedia.org/T436916) [10:24:25] FIRING: [4x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:25:06] (03CR) 10Samwilson: "I think this could do with some docs in README about how to run `unit-test`, i.e. that it doesn't require docker and what it does require." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333252 (owner: 10Ladsgroup-claude) [10:25:17] (03CR) 10CI reject: [V:04-1] Print log file for any action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337551 (owner: 10Hashar) [10:25:29] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2001.codfw.wmnet [10:25:31] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs2001.codfw.wmnet [10:25:54] !log mvernon@cumin2003 START - Cookbook sre.hosts.provision for host aqs2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:26:03] !log btullis@deploy1003 Started scap sync-world: Attempting to update mediawiki-cli for T436913 [10:26:06] T436913: 7za is OOMkilled when dumps are running on a Bookworm-based image - https://phabricator.wikimedia.org/T436913 [10:26:22] (03CR) 10Jelto: [C:03+1] "lgtm, we could think about deleting the downtime again at the end of the cookbook or use `with`. But I'm also fine just letting the downti" [cookbooks] - 10https://gerrit.wikimedia.org/r/1337547 (https://phabricator.wikimedia.org/T437189) (owner: 10Arnaudb) [10:26:29] !log cgoubert@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1340.eqiad.wmnet [10:26:34] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1340.eqiad.wmnet [10:27:09] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1340.eqiad.wmnet [10:27:16] (03PS3) 10Hashar: Print log file for any action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337551 [10:27:23] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1340.eqiad.wmnet with OS trixie [10:27:40] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:27:53] !log cgoubert@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1340 [10:29:07] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host aqs2001.codfw.wmnet with OS bookworm [10:29:25] RESOLVED: [4x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:29:29] (03PS1) 10Blake: memcached: continue pki rollout [puppet] - 10https://gerrit.wikimedia.org/r/1337554 (https://phabricator.wikimedia.org/T353511) [10:29:47] (03CR) 10Blake: [C:03+2] memcached: continue pki rollout [puppet] - 10https://gerrit.wikimedia.org/r/1337554 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [10:30:48] (03CR) 10Ladsgroup: [C:03+1] openstack: wikireplica_dns: Add user-facing names for x4 databases [puppet] - 10https://gerrit.wikimedia.org/r/1337552 (https://phabricator.wikimedia.org/T437208) (owner: 10Majavah) [10:30:57] cgoubert@cumin2003 renumber-node (PID 4115522) is awaiting input [10:31:33] (03CR) 10Majavah: [C:03+2] openstack: wikireplica_dns: Add user-facing names for x4 databases [puppet] - 10https://gerrit.wikimedia.org/r/1337552 (https://phabricator.wikimedia.org/T437208) (owner: 10Majavah) [10:32:44] (03CR) 10CI reject: [V:04-1] Print log file for any action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337551 (owner: 10Hashar) [10:33:00] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [10:33:24] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-replica1006.wikimedia.org with OS trixie [10:33:25] !log jmm@cumin1004 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ldap-replica1006.wikimedia.org [10:33:39] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12294328 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host ldap-replica1006.wikimedia.org with OS t... [10:34:55] FIRING: [4x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:34:59] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1241: Pool db1241.eqiad.wmnet in after cloning [10:36:15] (03CR) 10Samwilson: "Is it possible to run this in Docker as well? It looks like ruff needs to be installed in the container. (I think we should maintain the s" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 (owner: 10Ladsgroup-claude) [10:37:10] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1340 - cgoubert@cumin2003" [10:38:14] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1181.eqiad.wmnet onto db1190.eqiad.wmnet [10:38:17] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1181: Depool db1181.eqiad.wmnet to then clone it to db1190.eqiad.wmnet - marostegui@cumin1003 [10:38:41] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1181: Depool db1181.eqiad.wmnet to then clone it to db1190.eqiad.wmnet - marostegui@cumin1003 [10:38:42] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1190: Depool db1190.eqiad.wmnet - marostegui@cumin1003 [10:39:06] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1190: Depool db1190.eqiad.wmnet - marostegui@cumin1003 [10:39:10] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1340 - cgoubert@cumin2003" [10:39:10] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:39:11] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1340.eqiad.wmnet 164.32.64.10.in-addr.arpa 4.6.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [10:39:14] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1340.eqiad.wmnet 164.32.64.10.in-addr.arpa 4.6.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [10:39:15] !log cgoubert@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1340 [10:39:17] (03PS1) 10Marostegui: mariadb: Move db1190 to s7 [puppet] - 10https://gerrit.wikimedia.org/r/1337557 (https://phabricator.wikimedia.org/T434782) [10:39:50] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1340 [10:39:50] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1340 [10:40:45] (03CR) 10Marostegui: [C:03+2] mariadb: Move db1190 to s7 [puppet] - 10https://gerrit.wikimedia.org/r/1337557 (https://phabricator.wikimedia.org/T434782) (owner: 10Marostegui) [10:40:52] !log urbanecm@deploy1003 mwscript-k8s job started: extensions/GrowthExperiments/maintenance/cleanMentorList.php --wiki=hrwiki # T436659 [10:40:55] T436659: Unnecessary mentor status updates - https://phabricator.wikimedia.org/T436659 [10:40:59] PROBLEM - Postfix SMTP on crm2001 is CRITICAL: CRITICAL - Certificate crm2001.codfw.wmnet expires in 15 day(s) (Wed 23 Sep 2026 10:40:00 AM GMT +0000). https://wikitech.wikimedia.org/wiki/Mail%23Troubleshooting [10:47:04] !log urbanecm@deploy1003 mwscript-k8s job started: extensions/GrowthExperiments/maintenance/cleanMentorList.php --wiki=frwiki # T436659 [10:47:07] T436659: Unnecessary mentor status updates - https://phabricator.wikimedia.org/T436659 [10:48:15] (03PS2) 10Muehlenhoff: Depool puppetserver2001 for reboot [dns] - 10https://gerrit.wikimedia.org/r/1337549 [10:49:55] RESOLVED: [4x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:50:25] (03CR) 10Muehlenhoff: [C:03+2] Depool puppetserver2001 for reboot [dns] - 10https://gerrit.wikimedia.org/r/1337549 (owner: 10Muehlenhoff) [10:50:38] !log jmm@dns1004 START - running authdns-update [10:51:56] !log cgoubert@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1340.eqiad.wmnet with reason: host reimage [10:53:08] !log jmm@dns1004 END - running authdns-update [10:54:59] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1340.eqiad.wmnet with reason: host reimage [10:55:04] mvernon@cumin2003 reimage (PID 4115771) is awaiting input [10:55:22] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12294381 (10MoritzMuehlenhoff) [10:56:23] (03CR) 10Ladsgroup: "All of the dependencies are tracked under requirements-dev.txt and ruff is added to that files in this patch. I can make make targets like" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 (owner: 10Ladsgroup-claude) [10:56:49] (03PS1) 10Hashar: Normalize root path to avoid extra . or / [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337559 [10:57:14] jouncebot: nowandnext [10:57:15] For the next 0 hour(s) and 2 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1000) [10:57:15] In 2 hour(s) and 2 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1300) [10:58:13] (03CR) 10Zabe: [C:03+2] rdbms: Discourage setting db domain to false for remote virtual domains [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337550 (https://phabricator.wikimedia.org/T422940) (owner: 10Zabe) [10:58:14] (03CR) 10Zabe: [C:03+2] Migrate querying categorylinks to virtual domain [extensions/UploadWizard] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337310 (https://phabricator.wikimedia.org/T405812) (owner: 10Zabe) [10:58:31] !log btullis@deploy1003 Finished scap sync-world: Attempting to update mediawiki-cli for T436913 (duration: 35m 20s) [10:58:35] T436913: 7za is OOMkilled when dumps are running on a Bookworm-based image - https://phabricator.wikimedia.org/T436913 [10:58:46] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.12 point update - https://phabricator.wikimedia.org/T403852#12294383 (10MoritzMuehlenhoff) [10:58:59] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.12 point update - https://phabricator.wikimedia.org/T403852#12294384 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff All done [10:59:12] (03PS4) 10Ayounsi: Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 [11:02:14] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2001.codfw.wmnet with reason: host reimage [11:03:20] (03CR) 10Ayounsi: Add support for multiple host queries comma separated (032 comments) [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 (owner: 10Ayounsi) [11:03:38] (03Merged) 10jenkins-bot: rdbms: Discourage setting db domain to false for remote virtual domains [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337550 (https://phabricator.wikimedia.org/T422940) (owner: 10Zabe) [11:03:43] (03Merged) 10jenkins-bot: Migrate querying categorylinks to virtual domain [extensions/UploadWizard] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337310 (https://phabricator.wikimedia.org/T405812) (owner: 10Zabe) [11:04:07] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1337550|rdbms: Discourage setting db domain to false for remote virtual domains (T422940)]], [[gerrit:1337310|Migrate querying categorylinks to virtual domain (T405812)]] [11:04:13] T422940: Figure out how to handle virtual domains on foreign wikis - https://phabricator.wikimedia.org/T422940 [11:04:13] T405812: Migrate categorylinks to virtual domain - https://phabricator.wikimedia.org/T405812 [11:06:47] !log jmm@cumin2003 START - Cookbook sre.puppet.disable-merges [11:06:48] (03CR) 10Clément Goubert: [C:03+1] "Thanks for taking a look. If the reimage went fine, I will repool the server." [software/spicerack] - 10https://gerrit.wikimedia.org/r/1336116 (owner: 10JHathaway) [11:06:48] !log jmm@cumin2003 END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) [11:07:43] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1339.eqiad.wmnet [11:07:45] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1339.eqiad.wmnet [11:07:48] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host puppetserver2001.codfw.wmnet [11:08:56] (03CR) 10CI reject: [V:04-1] Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 (owner: 10Ayounsi) [11:09:35] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2001.codfw.wmnet with reason: host reimage [11:09:51] (03PS8) 10Ladsgroup-claude: Add a unit test layer that runs without the container [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333252 [11:09:52] (03PS6) 10Ladsgroup-claude: Replace flake8 with ruff and apply its fixes [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 [11:09:52] (03PS5) 10Ladsgroup-claude: Apply ruff format [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333254 [11:09:53] (03PS2) 10Ladsgroup-claude: [WIP] Run the Swift result storage read off the event loop [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337312 (https://phabricator.wikimedia.org/T431767) [11:09:54] (03PS2) 10Ladsgroup-claude: [WIP] Stop blocking the loop on the failure-counter memcached write [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337313 (https://phabricator.wikimedia.org/T431767) [11:09:55] (03PS2) 10Ladsgroup-claude: [WIP] Read originals off the event loop in the file loader [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337314 (https://phabricator.wikimedia.org/T431767) [11:09:58] (03PS2) 10Ladsgroup-claude: [WIP] Collect timing headers on the context instead of the request handler [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337315 (https://phabricator.wikimedia.org/T431767) [11:10:02] (03PS2) 10Ladsgroup-claude: [WIP] Give thumbor's engine thread pool a worker [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337316 (https://phabricator.wikimedia.org/T431767) [11:10:06] (03PS2) 10Ladsgroup-claude: [WIP] Run engine.load() on the engine thread pool too [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337317 (https://phabricator.wikimedia.org/T431767) [11:10:18] !log zabe@deploy1003 zabe: Backport for [[gerrit:1337550|rdbms: Discourage setting db domain to false for remote virtual domains (T422940)]], [[gerrit:1337310|Migrate querying categorylinks to virtual domain (T405812)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:10:23] T422940: Figure out how to handle virtual domains on foreign wikis - https://phabricator.wikimedia.org/T422940 [11:10:23] T405812: Migrate categorylinks to virtual domain - https://phabricator.wikimedia.org/T405812 [11:10:51] !log zabe@deploy1003 zabe: Continuing with deployment [11:11:30] !log jmm@cumin2003 START - Cookbook sre.puppet.disable-merges [11:11:31] !log jmm@cumin2003 END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) [11:11:55] (03CR) 10Ladsgroup: "I did this for both unit tests and ruff (the patch above it). I'm not super sure about explicitly mentioning the dependencies in the readm" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333252 (owner: 10Ladsgroup-claude) [11:13:23] (03PS1) 10Hashar: Log info to stdout when run interactively [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337562 [11:13:34] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2001.codfw.wmnet [11:14:16] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1340.eqiad.wmnet with OS trixie [11:14:30] (03PS5) 10Ayounsi: Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 [11:17:14] (03PS1) 10Btullis: Tone down the error messages for the ceph clusters [alerts] - 10https://gerrit.wikimedia.org/r/1337564 (https://phabricator.wikimedia.org/T437206) [11:17:25] FIRING: [2x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:18:15] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337550|rdbms: Discourage setting db domain to false for remote virtual domains (T422940)]], [[gerrit:1337310|Migrate querying categorylinks to virtual domain (T405812)]] (duration: 14m 08s) [11:18:20] T422940: Figure out how to handle virtual domains on foreign wikis - https://phabricator.wikimedia.org/T422940 [11:18:21] T405812: Migrate categorylinks to virtual domain - https://phabricator.wikimedia.org/T405812 [11:19:07] FIRING: [2x] ProbeDown: Service aqs2001-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:20:09] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1241: Pool db1241.eqiad.wmnet in after cloning [11:20:10] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1180.eqiad.wmnet onto db1241.eqiad.wmnet [11:22:21] (03PS2) 10Ayounsi: Add Python 3.14 support [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 [11:22:32] (03PS4) 10Hashar: Print log file for any action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337551 [11:24:07] FIRING: [4x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:27:06] (03PS3) 10Ayounsi: Add Python 3.14 support [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 [11:27:35] (03CR) 10Clément Goubert: mw-on-k8s: Add runbook link using official URL shortener (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1335636 (https://phabricator.wikimedia.org/T437010) (owner: 10Andrea Denisse) [11:28:02] !log jmm@cumin1004 START - Cookbook sre.hosts.decommission for hosts ldap-replica1006.wikimedia.org [11:28:40] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2001.codfw.wmnet with OS bookworm [11:32:56] !log jmm@cumin1004 START - Cookbook sre.dns.netbox [11:34:07] FIRING: [4x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:34:22] FIRING: [4x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:35:12] !log mvernon@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs2002.codfw.wmnet [11:37:16] (03PS1) 10Muehlenhoff: Revert "Depool puppetserver2001 for reboot" [dns] - 10https://gerrit.wikimedia.org/r/1337569 [11:37:58] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ldap-replica1006.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1004" [11:39:07] RESOLVED: [4x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:39:33] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ldap-replica1006.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin1004" [11:39:33] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:39:34] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ldap-replica1006.wikimedia.org [11:39:50] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12294486 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin1004 for hosts: `ldap-replica1006.wikimedia.org` - ldap-r... [11:39:56] !log jmm@cumin1004 START - Cookbook sre.ganeti.makevm for new host ldap-replica2007.wikimedia.org [11:39:58] !log jmm@cumin1004 START - Cookbook sre.dns.netbox [11:39:59] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [11:40:24] (03CR) 10Muehlenhoff: [C:03+2] Revert "Depool puppetserver2001 for reboot" [dns] - 10https://gerrit.wikimedia.org/r/1337569 (owner: 10Muehlenhoff) [11:41:51] !log installing bash updates from bookworm point release [11:41:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:44:05] (03PS5) 10Blake: memcached: Clean up pki migration switch. [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) [11:44:43] cgoubert@cumin2003 renumber-node (PID 4115522) is awaiting input [11:45:07] (03CR) 10Zabe: "The x4 master is currently just a replica of a s4 host, right?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [11:45:22] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica2007.wikimedia.org - jmm@cumin1004" [11:45:25] (03PS6) 10Blake: memcached: Clean up pki migration switch. [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) [11:45:25] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica2007.wikimedia.org - jmm@cumin1004" [11:45:25] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:45:25] !log jmm@cumin1004 START - Cookbook sre.dns.wipe-cache ldap-replica2007.wikimedia.org on all recursors [11:45:29] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ldap-replica2007.wikimedia.org on all recursors [11:46:03] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica2007.wikimedia.org - jmm@cumin1004" [11:46:06] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica2007.wikimedia.org - jmm@cumin1004" [11:46:11] !log jmm@cumin1004 START - Cookbook sre.hosts.reimage for host ldap-replica2007.wikimedia.org with OS trixie [11:46:13] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [11:46:29] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12294502 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host ldap-replica2007.wikimedia.org with... [11:46:59] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host aqs2002.codfw.wmnet [11:50:10] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.13 point update - https://phabricator.wikimedia.org/T414205#12294507 (10MoritzMuehlenhoff) [11:50:14] (03CR) 10Ladsgroup: [C:03+1] "x4 master from mediawiki point of view right now is just s4 master: https://noc.wikimedia.org/dbconfig/eqiad.json (should be the same for " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [11:50:37] FIRING: [7x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:52:19] !log jmm@dns1004 START - running authdns-update [11:53:03] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns2004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:03] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns2006 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:03] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns1006 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:03] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns1005 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:05] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns6002 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:05] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns6001 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:05] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns7001 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:05] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns7002 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:07] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns5004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:07] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns5003 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:19] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns4004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:19] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns4003 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:19] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns3004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:53:19] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns3003 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7dbcd24333e5f745d96a631867e856a4651b937e, dns.git is dcaf3660a8b45cb72baa861825bf75a5404e7c5b) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:54:05] (03PS6) 10Ayounsi: Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 [11:54:05] (03PS1) 10Ayounsi: Add suport for short hostname in query [software/homer] - 10https://gerrit.wikimedia.org/r/1337571 [11:54:50] !log jmm@dns1004 END - running authdns-update [11:55:37] FIRING: [4x] ProbeDown: Service aqs2002-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:57:08] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2002.codfw.wmnet [11:57:10] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs2002.codfw.wmnet [11:58:03] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns2006 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:03] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns2004 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:03] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns1006 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:03] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns1005 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:05] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns6002 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:05] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns6001 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:05] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns7002 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:05] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns7001 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:07] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns5003 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:07] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns5004 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:19] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns4003 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:19] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns4004 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:19] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns3004 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:58:19] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns3003 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [12:00:22] (03CR) 10Gmodena: [C:03+1] WDQS: Bump wdqs-proxy version to 0.8.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337553 (https://phabricator.wikimedia.org/T436916) (owner: 10Trueg) [12:00:37] RESOLVED: [4x] ProbeDown: Service aqs2002-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:01:23] !log mvernon@cumin2003 START - Cookbook sre.hosts.provision for host aqs2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:02:59] !log jmm@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-replica2007.wikimedia.org with reason: host reimage [12:03:09] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:03:16] (03CR) 10Gmodena: [C:03+1] WDQS: Improved resource limits for Qlever on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335771 (https://phabricator.wikimedia.org/T436295) (owner: 10Trueg) [12:03:38] (03CR) 10Samwilson: [C:03+1] "Yes that looks better, thanks. But there are three sets of tests I think, not two, because of integration being split into online and offl" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333252 (owner: 10Ladsgroup-claude) [12:04:09] jouncebot: nowandnext [12:04:09] No deployments scheduled for the next 0 hour(s) and 55 minute(s) [12:04:09] In 0 hour(s) and 55 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1300) [12:04:20] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host aqs2002.codfw.wmnet with OS bookworm [12:04:42] (03CR) 10Elukey: [C:03+2] docker_registry: move away from Swift completely [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) (owner: 10Elukey) [12:04:50] (03CR) 10Hnowlan: [C:03+1] Review access change [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337307 (owner: 10Ladsgroup) [12:05:22] (03CR) 10Ladsgroup: [V:03+2 C:03+2] Review access change [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337307 (owner: 10Ladsgroup) [12:06:27] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (NOOP 3): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9373/console" [puppet] - 10https://gerrit.wikimedia.org/r/1335928 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [12:06:36] (03CR) 10Ladsgroup: [V:03+1 C:03+1] "can't merge it 😞" [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337307 (owner: 10Ladsgroup) [12:07:10] (03CR) 10Jelto: [V:03+1 C:03+1] "lgtm, thank you" [puppet] - 10https://gerrit.wikimedia.org/r/1335928 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [12:07:25] FIRING: [3x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:07:38] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-replica2007.wikimedia.org with reason: host reimage [12:08:07] FIRING: [2x] ProbeDown: Service aqs2002-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:08:19] (03CR) 10Ladsgroup: [C:03+2] "I'd say let's keep it this way and then if needs arises, we can change later. Having a unit test at all is a major improvement in itself." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333252 (owner: 10Ladsgroup-claude) [12:09:46] (03CR) 10Hnowlan: "sgtm, thanks" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333155 (owner: 10Hnowlan) [12:12:25] FIRING: [3x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:12:44] (03PS1) 10Novem Linguae: nlwiki: enable SecurePoll local elections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) [12:13:05] PROBLEM - Host wikikube-worker1053 is DOWN: PING CRITICAL - Packet loss = 60%, RTA = 4313.40 ms [12:13:07] FIRING: [4x] ProbeDown: Service aqs2002-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:13:45] RECOVERY - Host wikikube-worker1053 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [12:14:36] !log moved most of the Docker Registry's prefixes to a new internal S3 backend. For any docker pull failure that worked in the past, please ping me or drop a note in T435499 or contact the oncall SREs [12:14:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:14:40] T435499: Move the majority of the Registry's docker image prefixes to a new s3 bucket - https://phabricator.wikimedia.org/T435499 [12:15:19] (03Merged) 10jenkins-bot: Add a unit test layer that runs without the container [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333252 (owner: 10Ladsgroup-claude) [12:17:25] FIRING: [3x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:18:07] RESOLVED: [4x] ProbeDown: Service aqs2002-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:18:53] (03CR) 10Btullis: [C:03+2] Tone down the error messages for the ceph clusters [alerts] - 10https://gerrit.wikimedia.org/r/1337564 (https://phabricator.wikimedia.org/T437206) (owner: 10Btullis) [12:19:58] (03PS1) 10Hashar: Review access change [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337581 [12:20:40] (03Merged) 10jenkins-bot: Tone down the error messages for the ceph clusters [alerts] - 10https://gerrit.wikimedia.org/r/1337564 (https://phabricator.wikimedia.org/T437206) (owner: 10Btullis) [12:22:11] (03PS2) 10Hashar: Change ownership to 'mediawiki' [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337581 [12:22:35] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-replica2007.wikimedia.org with OS trixie [12:22:35] !log jmm@cumin1004 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ldap-replica2007.wikimedia.org [12:22:47] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2002.codfw.wmnet with reason: host reimage [12:22:51] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12294567 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host ldap-replica2007.wikimedia.org with OS t... [12:23:14] (03CR) 10Hashar: [V:03+2 C:03+2] Change ownership to 'mediawiki' [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337581 (owner: 10Hashar) [12:23:32] (03CR) 10Ladsgroup: [C:03+1] Change ownership to 'mediawiki' [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337581 (owner: 10Hashar) [12:27:37] (03CR) 10Cathal Mooney: "LGTM! Nice one will be a big help" [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 (owner: 10Ayounsi) [12:28:18] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2002.codfw.wmnet with reason: host reimage [12:28:31] (03CR) 10Cathal Mooney: [C:03+1] Add suport for short hostname in query [software/homer] - 10https://gerrit.wikimedia.org/r/1337571 (owner: 10Ayounsi) [12:29:54] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1181: Pool db1181.eqiad.wmnet in after cloning [12:31:31] (03Abandoned) 10Ladsgroup: Review access change [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337307 (owner: 10Ladsgroup) [12:34:31] (03CR) 10Arnaudb: [C:03+2] "thanks for the review: fwiw the "expire-only" is deliberate." [cookbooks] - 10https://gerrit.wikimedia.org/r/1337547 (https://phabricator.wikimedia.org/T437189) (owner: 10Arnaudb) [12:34:44] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (NOOP 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9374/console" [puppet] - 10https://gerrit.wikimedia.org/r/1335927 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [12:35:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [12:35:26] (03CR) 10Samwilson: [C:03+1] "Yeah that sounds good. The tests are currently failing for me locally, but I think I'm doing something wrong." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 (owner: 10Ladsgroup-claude) [12:36:38] (03CR) 10Jelto: [V:03+1 C:03+1] "lgtm thank you" [puppet] - 10https://gerrit.wikimedia.org/r/1335927 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [12:38:10] (03Merged) 10jenkins-bot: gitlab: silence ATSBackendErrorsHigh during primary upgrades [cookbooks] - 10https://gerrit.wikimedia.org/r/1337547 (https://phabricator.wikimedia.org/T437189) (owner: 10Arnaudb) [12:39:51] (03CR) 10Filippo Giunchedi: [C:03+1] "Thank you David for the comments !" [puppet] - 10https://gerrit.wikimedia.org/r/1331487 (https://phabricator.wikimedia.org/T436252) (owner: 10Filippo Giunchedi) [12:39:55] (03CR) 10Filippo Giunchedi: [C:03+2] cloudnfs: add textfile exporter for extra nfsd metrics [puppet] - 10https://gerrit.wikimedia.org/r/1331487 (https://phabricator.wikimedia.org/T436252) (owner: 10Filippo Giunchedi) [12:40:10] (03CR) 10Samwilson: [C:03+2] Replace flake8 with ruff and apply its fixes [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 (owner: 10Ladsgroup-claude) [12:40:44] (03Merged) 10jenkins-bot: Replace flake8 with ruff and apply its fixes [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 (owner: 10Ladsgroup-claude) [12:46:14] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2002.codfw.wmnet with OS bookworm [12:46:54] (03PS1) 10Filippo Giunchedi: dumps: include nfsd_exporter [puppet] - 10https://gerrit.wikimedia.org/r/1337587 (https://phabricator.wikimedia.org/T436252) [12:47:52] !log mvernon@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs2003.codfw.wmnet [12:48:14] (03CR) 10Elukey: "Looks good, but I have a couple of questions since we are working on this:" [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 (owner: 10Ayounsi) [12:48:58] (03CR) 10CI reject: [V:04-1] dumps: include nfsd_exporter [puppet] - 10https://gerrit.wikimedia.org/r/1337587 (https://phabricator.wikimedia.org/T436252) (owner: 10Filippo Giunchedi) [12:49:59] (03PS2) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-analytics* in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334803 (https://phabricator.wikimedia.org/T420436) [12:50:12] (03CR) 10Volans: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1337456 (owner: 10Filippo Giunchedi) [12:51:52] (03CR) 10Dreamy Jazz: [C:03+1] SI: Use new InfoChip text class in case status updater [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337450 (https://phabricator.wikimedia.org/T437020) (owner: 10STran) [12:53:41] (03PS2) 10Filippo Giunchedi: dumps: include nfsd_exporter [puppet] - 10https://gerrit.wikimedia.org/r/1337587 (https://phabricator.wikimedia.org/T436252) [12:54:36] !log jmm@cumin1004 START - Cookbook sre.ganeti.makevm for new host ldap-replica2008.wikimedia.org [12:54:38] !log jmm@cumin1004 START - Cookbook sre.dns.netbox [12:54:48] (03CR) 10Filippo Giunchedi: [C:03+2] Revert "site: temp decom for cloudvirt10(6[567]|7[234]) ahead of rack move" [puppet] - 10https://gerrit.wikimedia.org/r/1337456 (owner: 10Filippo Giunchedi) [12:55:15] (03PS4) 10Ayounsi: Add Python 3.14 support [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 [12:55:15] (03PS7) 10Ayounsi: Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 [12:55:15] (03PS2) 10Ayounsi: Add suport for short hostname in query [software/homer] - 10https://gerrit.wikimedia.org/r/1337571 [12:56:56] (03CR) 10Ayounsi: "Sure, updated!" [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 (owner: 10Ayounsi) [12:57:24] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for eventgate-analytics* in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334803 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [12:58:40] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw [13:00:02] jmm@cumin1004 makevm (PID 902804) is awaiting input [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: I, the Bot under the Fountain, call upon thee, The Deployer, to do UTC afternoon backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1300). [13:00:05] Tran: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:10] o/ [13:00:18] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host aqs2003.codfw.wmnet [13:00:53] (03PS3) 10Filippo Giunchedi: dumps: include nfsd_exporter [puppet] - 10https://gerrit.wikimedia.org/r/1337587 (https://phabricator.wikimedia.org/T436252) [13:01:36] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica2008.wikimedia.org - jmm@cumin1004" [13:01:44] Tran: do you need a deployer? :) [13:02:03] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [13:02:11] No I can self-deploy and I'll also be doing a private deploy afterwards [13:02:31] ack [13:02:38] If there's no one else in the window, I would also like to run a related maintenance script if there's time and no objections [13:02:48] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM ldap-replica2008.wikimedia.org - jmm@cumin1004" [13:02:48] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:02:48] !log jmm@cumin1004 START - Cookbook sre.dns.wipe-cache ldap-replica2008.wikimedia.org on all recursors [13:02:51] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ldap-replica2008.wikimedia.org on all recursors [13:02:51] Sounds good [13:02:54] (03CR) 10Filippo Giunchedi: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9377/co" [puppet] - 10https://gerrit.wikimedia.org/r/1337587 (https://phabricator.wikimedia.org/T436252) (owner: 10Filippo Giunchedi) [13:02:59] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [13:02:59] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw [13:03:23] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica2008.wikimedia.org - jmm@cumin1004" [13:03:26] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM ldap-replica2008.wikimedia.org - jmm@cumin1004" [13:03:31] going to get started [13:03:31] !log jmm@cumin1004 START - Cookbook sre.hosts.reimage for host ldap-replica2008.wikimedia.org with OS trixie [13:03:40] FIRING: [3x] ProbeDown: Service aqs2003-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:03:47] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12294646 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host ldap-replica2008.wikimedia.org with... [13:03:55] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy1003 using scap backport" [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337450 (https://phabricator.wikimedia.org/T437020) (owner: 10STran) [13:06:15] (03PS2) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-analytics* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334804 (https://phabricator.wikimedia.org/T420436) [13:07:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:08:40] FIRING: [4x] ProbeDown: Service aqs2003-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:09:23] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 07 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/Math] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335752 (https://phabricator.wikimedia.org/T436876) (owner: 10Krinkle) [13:09:40] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 07 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/Math] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1336611 (https://phabricator.wikimedia.org/T379359) (owner: 10Krinkle) [13:09:54] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 07 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1336609 (https://phabricator.wikimedia.org/T297300) (owner: 10Krinkle) [13:10:07] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 07 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1336613 (https://phabricator.wikimedia.org/T428517) (owner: 10Krinkle) [13:10:58] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2003.codfw.wmnet [13:11:01] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs2003.codfw.wmnet [13:11:15] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for eventgate-analytics* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334804 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [13:11:30] (03CR) 10Filippo Giunchedi: [V:03+1 C:03+2] "Trivial, self-merging" [puppet] - 10https://gerrit.wikimedia.org/r/1337587 (https://phabricator.wikimedia.org/T436252) (owner: 10Filippo Giunchedi) [13:12:02] (03Merged) 10jenkins-bot: SI: Use new InfoChip text class in case status updater [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337450 (https://phabricator.wikimedia.org/T437020) (owner: 10STran) [13:12:20] !log stran@deploy1003 Started scap sync-world: Backport for [[gerrit:1337450|SI: Use new InfoChip text class in case status updater (T437020)]] [13:12:24] T437020: SI case status InfoChips are not updating their text properly - https://phabricator.wikimedia.org/T437020 [13:13:09] (03CR) 10Trueg: [C:03+2] WDQS: Improved resource limits for Qlever on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335771 (https://phabricator.wikimedia.org/T436295) (owner: 10Trueg) [13:13:18] (03CR) 10Trueg: [C:03+2] WDQS: Bump wdqs-proxy version to 0.8.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337553 (https://phabricator.wikimedia.org/T436916) (owner: 10Trueg) [13:13:40] RESOLVED: [4x] ProbeDown: Service aqs2003-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:15:03] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1181: Pool db1181.eqiad.wmnet in after cloning [13:15:13] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1181.eqiad.wmnet onto db1190.eqiad.wmnet [13:15:29] !log mvernon@cumin2003 START - Cookbook sre.hosts.provision for host aqs2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:15:49] (03CR) 10Ladsgroup: "A lot of them get fixed in the patch above it. After this, I'll make sure CI runs it 😄" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 (owner: 10Ladsgroup-claude) [13:16:07] (03Merged) 10jenkins-bot: WDQS: Improved resource limits for Qlever on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335771 (https://phabricator.wikimedia.org/T436295) (owner: 10Trueg) [13:16:12] (03Merged) 10jenkins-bot: WDQS: Bump wdqs-proxy version to 0.8.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337553 (https://phabricator.wikimedia.org/T436916) (owner: 10Trueg) [13:16:21] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad [13:16:27] !log stran@deploy1003 stran: Backport for [[gerrit:1337450|SI: Use new InfoChip text class in case status updater (T437020)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:17:15] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:17:43] testing looks good, continuing [13:17:47] !log stran@deploy1003 stran: Continuing with deployment [13:17:59] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host aqs2003.codfw.wmnet with OS bookworm [13:18:23] (03PS1) 10Btullis: ceph: Allow CephX key material to be unmanaged [puppet] - 10https://gerrit.wikimedia.org/r/1337592 (https://phabricator.wikimedia.org/T428445) [13:18:25] (03PS1) 10Btullis: ceph: Stop managing CephX key material in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1337593 (https://phabricator.wikimedia.org/T428445) [13:18:28] (03PS1) 10Btullis: ceph: Use the squid package repository on cephosd2001 [puppet] - 10https://gerrit.wikimedia.org/r/1337594 (https://phabricator.wikimedia.org/T428445) [13:18:30] (03PS1) 10Btullis: ceph: Use the squid package repository on cephosd2002 [puppet] - 10https://gerrit.wikimedia.org/r/1337595 (https://phabricator.wikimedia.org/T428445) [13:18:33] (03PS1) 10Btullis: ceph: Use the squid package repository on cephosd2003 [puppet] - 10https://gerrit.wikimedia.org/r/1337596 (https://phabricator.wikimedia.org/T428445) [13:18:40] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:19:09] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1337592 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [13:21:42] FIRING: [2x] ProbeDown: Service aqs2003-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:22:26] !log stran@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337450|SI: Use new InfoChip text class in case status updater (T437020)]] (duration: 10m 06s) [13:22:29] T437020: SI case status InfoChips are not updating their text properly - https://phabricator.wikimedia.org/T437020 [13:22:39] starting my private deploy [13:22:50] (03CR) 10Samwilson: [C:03+2] Apply ruff format [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333254 (owner: 10Ladsgroup-claude) [13:23:05] !log jmm@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-replica2008.wikimedia.org with reason: host reimage [13:23:10] jelto@cumin1003 migrate-service-ipip (PID 3197661) is awaiting input [13:23:57] (03Merged) 10jenkins-bot: Apply ruff format [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333254 (owner: 10Ladsgroup-claude) [13:24:26] (03PS2) 10Btullis: ceph: Allow CephX key material to be unmanaged [puppet] - 10https://gerrit.wikimedia.org/r/1337592 (https://phabricator.wikimedia.org/T428445) [13:24:26] (03PS2) 10Btullis: ceph: Stop managing CephX key material in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1337593 (https://phabricator.wikimedia.org/T428445) [13:24:26] (03PS2) 10Btullis: ceph: Use the squid package repository on cephosd2001 [puppet] - 10https://gerrit.wikimedia.org/r/1337594 (https://phabricator.wikimedia.org/T428445) [13:24:27] (03PS2) 10Btullis: ceph: Use the squid package repository on cephosd2002 [puppet] - 10https://gerrit.wikimedia.org/r/1337595 (https://phabricator.wikimedia.org/T428445) [13:24:28] (03PS2) 10Btullis: ceph: Use the squid package repository on cephosd2003 [puppet] - 10https://gerrit.wikimedia.org/r/1337596 (https://phabricator.wikimedia.org/T428445) [13:24:40] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1340.eqiad.wmnet [13:24:42] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1340.eqiad.wmnet [13:24:45] !log cgoubert@cumin2003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1340.eqiad.wmnet [13:24:49] Tran: ack, let me know when you're done ) [13:24:52] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [13:25:06] !log installing openssh security updates [13:25:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:26:08] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [13:26:08] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad [13:26:42] FIRING: [4x] ProbeDown: Service aqs2003-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:27:00] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudvirt1065.eqiad.wmnet [13:27:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:28:18] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-replica2008.wikimedia.org with reason: host reimage [13:32:25] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:33:07] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1065.eqiad.wmnet [13:33:10] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudvirt1066.eqiad.wmnet [13:33:42] (03PS1) 10Dpogorzelski: deployment: add liftwing-studio to dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1337598 [13:34:54] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2003.codfw.wmnet with reason: host reimage [13:36:04] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1190: Repooling after cloning [13:36:34] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1337592 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [13:37:03] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1228: Repooling db1228 into s4 [13:37:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:37:18] (03PS2) 10Dpogorzelski: deployment: add liftwing-studio to dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1337598 [13:37:30] (03PS3) 10Dpogorzelski: deployment: add liftwing-studio to dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1337598 [13:38:06] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2003.codfw.wmnet with reason: host reimage [13:38:32] (03CR) 10Dpogorzelski: "I forgot this. Should have done this as step 1." [puppet] - 10https://gerrit.wikimedia.org/r/1337598 (owner: 10Dpogorzelski) [13:38:36] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Alert on packet loss to core sites from POPs - https://phabricator.wikimedia.org/T435592#12294746 (10cmooney) >>! In T435592#12274367, @ayounsi wrote: > Should that be abandoned in favor of RPM probes alerting ? As the RPM probes are more a... [13:38:39] (03CR) 10Clément Goubert: "Additional cleanup of `profile::memcached::{ssl_cert,ssl_key,localcacert}` hiera keys in:" [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:39:22] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [13:39:27] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [13:39:40] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1066.eqiad.wmnet [13:39:43] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudvirt1067.eqiad.wmnet [13:39:50] (03CR) 10JMeybohm: [C:03+1] memcached: Clean up pki migration switch. [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:40:19] (03CR) 10Btullis: [C:03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/1337598 (owner: 10Dpogorzelski) [13:40:42] (03CR) 10Dpogorzelski: [C:03+2] deployment: add liftwing-studio to dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1337598 (owner: 10Dpogorzelski) [13:41:08] Krinkle: I'm done. Could you ping me when you're done? I'd like to run a maintenance script after if nothing's happening. [13:41:36] (03CR) 10Elukey: [C:03+1] "Left a nit, good to go afterwards!" [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 (owner: 10Ayounsi) [13:42:55] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335754 (https://phabricator.wikimedia.org/T436971) (owner: 10Krinkle) [13:42:55] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1336613 (https://phabricator.wikimedia.org/T428517) (owner: 10Krinkle) [13:42:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [extensions/Math] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335752 (https://phabricator.wikimedia.org/T436876) (owner: 10Krinkle) [13:43:01] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [extensions/Math] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1336611 (https://phabricator.wikimedia.org/T379359) (owner: 10Krinkle) [13:43:06] Tran: ack will do [13:43:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1336609 (https://phabricator.wikimedia.org/T297300) (owner: 10Krinkle) [13:43:55] (03CR) 10Brouberol: [C:03+1] ceph: Allow CephX key material to be unmanaged [puppet] - 10https://gerrit.wikimedia.org/r/1337592 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [13:44:10] (03CR) 10Brouberol: [C:03+1] ceph: Stop managing CephX key material in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1337593 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [13:44:19] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-replica2008.wikimedia.org with OS trixie [13:44:19] !log jmm@cumin1004 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ldap-replica2008.wikimedia.org [13:44:31] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12294761 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host ldap-replica2008.wikimedia.org with OS t... [13:44:36] (03CR) 10Brouberol: [C:03+1] ceph: Use the squid package repository on cephosd2001 [puppet] - 10https://gerrit.wikimedia.org/r/1337594 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [13:44:42] (03CR) 10Brouberol: [C:03+1] ceph: Use the squid package repository on cephosd2002 [puppet] - 10https://gerrit.wikimedia.org/r/1337595 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [13:44:50] (03CR) 10Brouberol: [C:03+1] ceph: Use the squid package repository on cephosd2003 [puppet] - 10https://gerrit.wikimedia.org/r/1337596 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [13:45:35] (03CR) 10Btullis: [C:03+2] ceph: Allow CephX key material to be unmanaged [puppet] - 10https://gerrit.wikimedia.org/r/1337592 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [13:46:37] (03CR) 10Clément Goubert: "Again it got auto marked as resolved..." [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:50:02] !log cgoubert@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1341.eqiad.wmnet [13:50:07] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1341.eqiad.wmnet [13:51:18] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1341.eqiad.wmnet [13:52:26] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1341.eqiad.wmnet with OS trixie [13:52:37] (03CR) 10Ayounsi: Add Python 3.14 support (031 comment) [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 (owner: 10Ayounsi) [13:52:45] !log cgoubert@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1341 [13:52:46] (03PS5) 10Ayounsi: Add Python 3.14 support [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 [13:52:46] (03PS8) 10Ayounsi: Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 [13:52:46] (03PS3) 10Ayounsi: Add suport for short hostname in query [software/homer] - 10https://gerrit.wikimedia.org/r/1337571 [13:53:02] (03Merged) 10jenkins-bot: tests: Add user_id and actor in testLogDatabaseRowsForHiddenUser [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335754 (https://phabricator.wikimedia.org/T436971) (owner: 10Krinkle) [13:53:08] (03Merged) 10jenkins-bot: User: Use ActorStore on User::load [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1336613 (https://phabricator.wikimedia.org/T428517) (owner: 10Krinkle) [13:54:33] (03CR) 10Btullis: [C:03+2] ceph: Stop managing CephX key material in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1337593 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [13:55:06] (03CR) 10Ladsgroup: [C:03+2] thumbor: use service mesh to talk to swift [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333155 (owner: 10Hnowlan) [13:55:28] !log filippo@cumin1003 END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host cloudvirt1067.eqiad.wmnet [13:55:32] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudvirt1072.eqiad.wmnet [13:55:41] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [13:57:02] (03Merged) 10jenkins-bot: Fix empty bases in m(under|over) [extensions/Math] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335752 (https://phabricator.wikimedia.org/T436876) (owner: 10Krinkle) [13:57:03] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2003.codfw.wmnet with OS bookworm [13:57:49] (03Merged) 10jenkins-bot: thumbor: use service mesh to talk to swift [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333155 (owner: 10Hnowlan) [13:57:53] (03Merged) 10jenkins-bot: Use Core-compatible output for cancellation [extensions/Math] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1336611 (https://phabricator.wikimedia.org/T379359) (owner: 10Krinkle) [13:59:10] (03PS1) 10Ladsgroup: [DNM] Try to run test on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337608 [13:59:12] !log mvernon@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs2004.codfw.wmnet [13:59:30] (03Merged) 10jenkins-bot: Page: Fix absence caching when combined with multiple properties [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1336609 (https://phabricator.wikimedia.org/T297300) (owner: 10Krinkle) [13:59:48] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1335754|tests: Add user_id and actor in testLogDatabaseRowsForHiddenUser (T436971)]], [[gerrit:1336613|User: Use ActorStore on User::load (T428517)]], [[gerrit:1335752|Fix empty bases in m(under|over) (T436876)]], [[gerrit:1336611|Use Core-compatible output for cancellation (T379359)]], [[gerrit:1336609|Page: Fix absence caching when combined with mult [13:59:48] iple properties (T297300)]] [13:59:53] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1341 - cgoubert@cumin2003" [13:59:54] (03CR) 10Ayounsi: [C:03+2] Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 (owner: 10Ayounsi) [13:59:56] T436971: CheckUser: Add user_id and log_actor in testLogDatabaseRowsForHiddenUser - https://phabricator.wikimedia.org/T436971 [13:59:57] T428517: The user's row is being loaded three times on every page view - https://phabricator.wikimedia.org/T428517 [13:59:57] T436876: \underline{} on lone space in \align block yield SVG error "munder" in client-side MathJax - https://phabricator.wikimedia.org/T436876 [13:59:57] T379359: Fix \cancelto{y} in native MathML mode Chrome - https://phabricator.wikimedia.org/T379359 [13:59:58] T297300: PageProps doesn't cache result when it is false - https://phabricator.wikimedia.org/T297300 [14:00:04] (03PS1) 10STran: Split out edit and block-based filters from activity filters [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337610 (https://phabricator.wikimedia.org/T436508) [14:00:13] (03PS2) 10STran: Split out edit and block-based filters from activity filters [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337610 (https://phabricator.wikimedia.org/T436508) [14:00:22] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1341 - cgoubert@cumin2003" [14:00:22] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:00:23] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1341.eqiad.wmnet 159.32.64.10.in-addr.arpa 9.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [14:00:26] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1341.eqiad.wmnet 159.32.64.10.in-addr.arpa 9.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [14:00:27] !log cgoubert@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1341 [14:00:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:00:39] (03PS5) 10Fabfur: profile:haproxy: kapow score setup [puppet] - 10https://gerrit.wikimedia.org/r/1328543 (https://phabricator.wikimedia.org/T435799) [14:00:43] (03CR) 10Clément Goubert: [C:03+1] "Done" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333856 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [14:01:02] (03CR) 10CI reject: [V:04-1] profile:haproxy: kapow score setup [puppet] - 10https://gerrit.wikimedia.org/r/1328543 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [14:01:03] (03CR) 10CI reject: [V:04-1] [DNM] Try to run test on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337608 (owner: 10Ladsgroup) [14:01:12] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [14:01:12] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1341 [14:01:13] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1341 [14:01:21] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [14:01:22] (03CR) 10Clément Goubert: [C:03+1] httpd: Add version 1.0.3 (copy) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334012 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [14:01:42] RESOLVED: [4x] ProbeDown: Service aqs2003-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:02:00] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1072.eqiad.wmnet [14:02:04] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudvirt1073.eqiad.wmnet [14:02:06] (03PS6) 10Fabfur: profile:haproxy: kapow score setup [puppet] - 10https://gerrit.wikimedia.org/r/1328543 (https://phabricator.wikimedia.org/T435799) [14:02:30] (03PS2) 10Ladsgroup: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337608 [14:02:49] 10ops-eqiad, 10Cloud-VPS, 06DC-Ops, 06tools-infrastructure-team: cloudvirt1067 missing hwraid BBU - https://phabricator.wikimedia.org/T437221 (10fgiunchedi) 03NEW [14:03:04] (03CR) 10Clément Goubert: [C:03+1] httpd: Add switch to enable Kubernetes native sidecar behaviour. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334013 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [14:03:11] 10ops-eqiad, 10Cloud-VPS, 06DC-Ops, 06tools-infrastructure-team: cloudvirt1067 missing hwraid BBU - https://phabricator.wikimedia.org/T437221#12294886 (10fgiunchedi) Host is not in service and can be taken down at any time (with silences applied) [14:03:13] (03CR) 10Clément Goubert: [C:03+1] php-fpm: Add version 1.0.2 (copy) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334015 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [14:03:43] (03CR) 10Clément Goubert: [C:03+1] php-fpm: Make php-fpm-exporter a sidecar. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334016 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [14:04:06] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1335754|tests: Add user_id and actor in testLogDatabaseRowsForHiddenUser (T436971)]], [[gerrit:1336613|User: Use ActorStore on User::load (T428517)]], [[gerrit:1335752|Fix empty bases in m(under|over) (T436876)]], [[gerrit:1336611|Use Core-compatible output for cancellation (T379359)]], [[gerrit:1336609|Page: Fix absence caching when combined with multiple properties [14:04:06] (T297300)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:05:04] (03CR) 10Clément Goubert: [C:03+1] statsd: Upgrade to 1.0.5 (copy) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334738 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [14:05:10] (03CR) 10Ayounsi: [C:03+2] Add Python 3.14 support [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 (owner: 10Ayounsi) [14:05:14] (03CR) 10Btullis: [C:03+2] ceph: Use the squid package repository on cephosd2001 [puppet] - 10https://gerrit.wikimedia.org/r/1337594 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [14:05:17] (03CR) 10Clément Goubert: [C:03+1] statsd: Enable a native k8s sidecar option. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334739 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [14:06:23] Krinkle: o/ not sure if you saw the announce but I did an invasive maintenance on the docker registry, that nows is completely on S3 (internal cluster etc..), lemme know if you see anything weird [14:06:40] weird == docker pull errors etc.. [14:07:05] (03CR) 10Fabfur: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1328543 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [14:07:09] !log krinkle@deploy1003 krinkle: Continuing with deployment [14:07:27] elukey: ok [14:08:11] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1073.eqiad.wmnet [14:08:14] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudvirt1074.eqiad.wmnet [14:09:40] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host aqs2004.codfw.wmnet [14:10:02] (03CR) 10CI reject: [V:04-1] Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337608 (owner: 10Ladsgroup) [14:11:29] (03CR) 10Clément Goubert: [C:03+1] localmemcached: Add restart policy if we should be a sidecar. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334735 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [14:11:43] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1335754|tests: Add user_id and actor in testLogDatabaseRowsForHiddenUser (T436971)]], [[gerrit:1336613|User: Use ActorStore on User::load (T428517)]], [[gerrit:1335752|Fix empty bases in m(under|over) (T436876)]], [[gerrit:1336611|Use Core-compatible output for cancellation (T379359)]], [[gerrit:1336609|Page: Fix absence caching when combined with mul [14:11:43] tiple properties (T297300)]] (duration: 11m 55s) [14:11:51] T436971: CheckUser: Add user_id and log_actor in testLogDatabaseRowsForHiddenUser - https://phabricator.wikimedia.org/T436971 [14:11:51] T428517: The user's row is being loaded three times on every page view - https://phabricator.wikimedia.org/T428517 [14:11:51] T436876: \underline{} on lone space in \align block yield SVG error "munder" in client-side MathJax - https://phabricator.wikimedia.org/T436876 [14:11:52] T379359: Fix \cancelto{y} in native MathML mode Chrome - https://phabricator.wikimedia.org/T379359 [14:11:52] T297300: PageProps doesn't cache result when it is false - https://phabricator.wikimedia.org/T297300 [14:13:04] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [14:13:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:13:39] !log cgoubert@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1341.eqiad.wmnet with reason: host reimage [14:14:19] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [14:14:36] (03Merged) 10jenkins-bot: Add Python 3.14 support [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 (owner: 10Ayounsi) [14:14:37] (03Merged) 10jenkins-bot: Add support for multiple host queries comma separated [software/homer] - 10https://gerrit.wikimedia.org/r/1337453 (owner: 10Ayounsi) [14:14:46] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1074.eqiad.wmnet [14:15:25] (03PS2) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-main in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334805 (https://phabricator.wikimedia.org/T420436) [14:16:48] PROBLEM - Host wikikube-worker1341 is DOWN: PING CRITICAL - Packet loss = 100% [14:16:55] FIRING: [7x] ProbeDown: Service aqs2003-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:18:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:18:36] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1341.eqiad.wmnet with reason: host reimage [14:18:37] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync [14:18:39] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync [14:18:46] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:18:48] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2004.codfw.wmnet [14:18:50] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs2004.codfw.wmnet [14:19:07] !log mvernon@cumin2003 START - Cookbook sre.hosts.provision for host aqs2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:20:52] (03CR) 10Clément Goubert: [C:03+1] mesh: upgrade to 1.3.3 (copy) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334741 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [14:20:53] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:21:11] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1190: Repooling after cloning [14:21:13] (03CR) 10Clément Goubert: [C:03+1] mesh: Add switch to enable k8s native sidecar behaviour. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334742 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [14:21:28] (03PS2) 10Krinkle: Page: Clean up tests of PageProps service class [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327671 [14:21:38] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host aqs2004.codfw.wmnet with OS bookworm [14:21:50] RECOVERY - Host wikikube-worker1341 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [14:21:55] RESOLVED: [4x] ProbeDown: Service aqs2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:22:09] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling db1228 into s4 [14:24:22] (03PS7) 10Blake: memcached: Clean up pki migration switch. [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) [14:24:35] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [14:24:42] (03CR) 10Btullis: [C:03+2] ceph: Use the squid package repository on cephosd2002 [puppet] - 10https://gerrit.wikimedia.org/r/1337595 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [14:24:54] (03CR) 10Btullis: [C:03+2] ceph: Use the squid package repository on cephosd2003 [puppet] - 10https://gerrit.wikimedia.org/r/1337596 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [14:25:22] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for eventgate-main in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1334805 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [14:25:25] FIRING: [3x] ProbeDown: Service aqs2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:25:52] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw [14:26:57] FIRING: [4x] ProbeDown: Service aqs2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:27:29] (03CR) 10Blake: "Done, thanks for the catch!" [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [14:29:04] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1430) [14:30:08] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [14:30:08] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw [14:32:37] !log jgiannelos@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [14:33:14] !log jgiannelos@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [14:33:15] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [14:33:27] (03PS2) 10Jelto: service::catalog: Set ipip_encapsulation for eventgate-main in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334806 (https://phabricator.wikimedia.org/T420436) [14:33:56] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [14:34:31] (03PS1) 10Elukey: profile::docker_registry: remove swift when generating image lists [puppet] - 10https://gerrit.wikimedia.org/r/1337614 (https://phabricator.wikimedia.org/T435499) [14:35:05] !log jgiannelos@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [14:35:07] !log jgiannelos@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [14:35:08] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply [14:35:08] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [14:35:10] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply [14:35:12] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [14:36:50] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for eventgate-main in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1334806 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [14:37:15] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad [14:37:37] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1341.eqiad.wmnet with OS trixie [14:37:43] (03PS3) 10Federico Ceratto: core-mysql.my.cnf.erb: support templating expire_logs_days for core_test [puppet] - 10https://gerrit.wikimedia.org/r/1337601 (https://phabricator.wikimedia.org/T435059) [14:37:43] (03CR) 10Federico Ceratto: "(I'm intentionally not making changes to the prod configuration here)" [puppet] - 10https://gerrit.wikimedia.org/r/1337601 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [14:39:37] (03PS1) 10Dpogorzelski: cloudnative-pg: add liftwing-studio ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337617 [14:40:02] (03CR) 10Btullis: [C:03+1] cloudnative-pg: add liftwing-studio ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337617 (owner: 10Dpogorzelski) [14:40:22] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [14:40:30] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2004.codfw.wmnet with reason: host reimage [14:40:33] (03CR) 10Dpogorzelski: [C:03+2] cloudnative-pg: add liftwing-studio ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337617 (owner: 10Dpogorzelski) [14:40:38] nemo-yiannis: o/ not sure if you've read but I've done an invasive maintenance to the docker registry, if you see docker pull errors etc.. lemme know! [14:40:46] ok [14:41:16] 06SRE, 10SRE-Access-Requests: Requesting access to production for derenrich - https://phabricator.wikimedia.org/T436965#12295036 (10Volans) [14:41:31] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [14:41:31] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad [14:42:06] (03PS1) 10Volans: admin: add derenrich to the deployment group [puppet] - 10https://gerrit.wikimedia.org/r/1337618 (https://phabricator.wikimedia.org/T436965) [14:44:44] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2004.codfw.wmnet with reason: host reimage [14:50:00] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1337618 (https://phabricator.wikimedia.org/T436965) (owner: 10Volans) [14:51:10] (03Merged) 10jenkins-bot: cloudnative-pg: add liftwing-studio ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337617 (owner: 10Dpogorzelski) [14:51:34] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.14 point update - https://phabricator.wikimedia.org/T426759#12295106 (10MoritzMuehlenhoff) [14:54:51] (03CR) 10Elukey: [C:03+2] profile::docker_registry: remove swift when generating image lists [puppet] - 10https://gerrit.wikimedia.org/r/1337614 (https://phabricator.wikimedia.org/T435499) (owner: 10Elukey) [14:55:23] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team (Radar): apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12295132 (10hashar) [14:57:34] cgoubert@cumin2003 renumber-node (PID 4163525) is awaiting input [14:58:43] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [15:00:06] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [15:02:08] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2004.codfw.wmnet with OS bookworm [15:03:55] !log mvernon@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs2005.codfw.wmnet [15:03:57] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 07 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326400 (https://phabricator.wikimedia.org/T429510) (owner: 10Daimona Eaytoy) [15:04:11] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1341.eqiad.wmnet [15:04:13] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1341.eqiad.wmnet [15:04:16] !log cgoubert@cumin2003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1341.eqiad.wmnet [15:05:25] RESOLVED: [4x] ProbeDown: Service aqs2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:11:33] !log installing rsync security updates [15:11:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:13:32] (03PS1) 10Ladsgroup: thumbor: Enable service when mesh is enabled [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337626 [15:14:06] !log mvernon@cumin2003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts aqs2005.codfw.wmnet [15:14:40] !log mvernon@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs2005.codfw.wmnet [15:15:07] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host aqs2005.codfw.wmnet [15:18:18] (03PS2) 10Ladsgroup: thumbor: Enable service when mesh is enabled [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337626 [15:19:51] (03CR) 10Hnowlan: [C:03+1] thumbor: Enable service when mesh is enabled [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337626 (owner: 10Ladsgroup) [15:21:10] FIRING: [8x] ProbeDown: Service aqs2004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:22:13] (03PS1) 10Btullis: Add two new dse-k8s-worker nodes to conftool-data in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1337629 (https://phabricator.wikimedia.org/T432206) [15:22:16] (03PS1) 10Btullis: Add the new dse-k8s-worker nodes to the cluster in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1337630 (https://phabricator.wikimedia.org/T432206) [15:22:19] (03PS1) 10Btullis: Promote dse-k8s-worker200[4-5] to their correct role [puppet] - 10https://gerrit.wikimedia.org/r/1337631 (https://phabricator.wikimedia.org/T432206) [15:22:30] (03CR) 10Clément Goubert: [C:03+1] thumbor: Enable service when mesh is enabled [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337626 (owner: 10Ladsgroup) [15:22:45] (03CR) 10Ladsgroup: [C:03+2] thumbor: Enable service when mesh is enabled [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337626 (owner: 10Ladsgroup) [15:24:04] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs2005.codfw.wmnet [15:24:06] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts aqs2005.codfw.wmnet [15:24:28] !log mvernon@cumin2003 START - Cookbook sre.hosts.provision for host aqs2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:25:38] (03Merged) 10jenkins-bot: thumbor: Enable service when mesh is enabled [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337626 (owner: 10Ladsgroup) [15:25:44] (03CR) 10Brouberol: [C:03+1] Add two new dse-k8s-worker nodes to conftool-data in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1337629 (https://phabricator.wikimedia.org/T432206) (owner: 10Btullis) [15:26:10] RESOLVED: [4x] ProbeDown: Service aqs2005-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:26:15] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:26:18] (03CR) 10Brouberol: [C:03+1] Add the new dse-k8s-worker nodes to the cluster in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1337630 (https://phabricator.wikimedia.org/T432206) (owner: 10Btullis) [15:26:28] (03CR) 10Brouberol: [C:03+1] Promote dse-k8s-worker200[4-5] to their correct role [puppet] - 10https://gerrit.wikimedia.org/r/1337631 (https://phabricator.wikimedia.org/T432206) (owner: 10Btullis) [15:26:56] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host aqs2005.codfw.wmnet with OS bookworm [15:27:12] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [15:28:36] (03PS1) 10Hnowlan: cdn: create non-paging, ticketing ATS backend error rate alert [alerts] - 10https://gerrit.wikimedia.org/r/1337633 (https://phabricator.wikimedia.org/T436049) [15:29:48] (03PS4) 10Ayounsi: Add support for short hostname in query [software/homer] - 10https://gerrit.wikimedia.org/r/1337571 [15:30:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2005 has a BGP session which is not in the 'established' state. [15:30:04] jan_drewniak: #bothumor My software never has bugs. It just develops random features. Rise for Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1530). [15:31:55] FIRING: [4x] ProbeDown: Service aqs2005-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:36:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.56% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:37:22] (03CR) 10Elukey: [C:03+1] "I am not an expert on the codebase but given what I understood, it seems worth to try!" [software/homer] - 10https://gerrit.wikimedia.org/r/1337571 (owner: 10Ayounsi) [15:37:42] (03CR) 10Volans: [C:03+2] admin: add derenrich to the deployment group [puppet] - 10https://gerrit.wikimedia.org/r/1337618 (https://phabricator.wikimedia.org/T436965) (owner: 10Volans) [15:40:49] (03CR) 10Hnowlan: sre.opensearch.roll-restart-reboot: include checklist items (037 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (https://phabricator.wikimedia.org/T435265) (owner: 10Herron) [15:40:59] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to production for derenrich - https://phabricator.wikimedia.org/T436965#12295325 (10Volans) @derenrich All done, within ~30m the patch will be propagated. Once tested that you have access and it works fine feel free to resolve this task. Tha... [15:42:09] jouncebot: nowandnext [15:42:09] For the next 0 hour(s) and 17 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1530) [15:42:09] In 1 hour(s) and 17 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1700) [15:42:09] In 1 hour(s) and 17 minute(s): Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1700) [15:44:19] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [15:44:29] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [15:45:37] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2005.codfw.wmnet with reason: host reimage [15:46:00] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [15:46:07] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.14 point update - https://phabricator.wikimedia.org/T426759#12295336 (10MoritzMuehlenhoff) [15:46:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:47:24] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [15:48:32] (03PS1) 10Zabe: Move globalimagelinks queries to x4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337635 (https://phabricator.wikimedia.org/T437108) [15:49:34] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2005.codfw.wmnet with reason: host reimage [15:49:53] (03CR) 10Zabe: [C:03+2] Disable query pages not yet compatible with commons split [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336829 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [15:50:52] (03Merged) 10jenkins-bot: Disable query pages not yet compatible with commons split [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336829 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [15:51:11] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1336829|Disable query pages not yet compatible with commons split (T437108)]] [15:51:14] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [15:51:55] RESOLVED: [4x] ProbeDown: Service aqs2005-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:55:05] (03Abandoned) 10Zabe: Set remote virtual domain for globalusage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1331617 (https://phabricator.wikimedia.org/T422940) (owner: 10Zabe) [15:55:13] !log zabe@deploy1003 zabe: Backport for [[gerrit:1336829|Disable query pages not yet compatible with commons split (T437108)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:56:43] !log zabe@deploy1003 zabe: Continuing with deployment [15:57:29] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [15:59:32] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [16:01:33] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1336829|Disable query pages not yet compatible with commons split (T437108)]] (duration: 10m 22s) [16:01:36] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [16:01:53] (03CR) 10Federico Ceratto: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1337601 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [16:06:59] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2005.codfw.wmnet with OS bookworm [16:07:37] (03PS2) 10Zabe: Set RemoteVirtualDomainsMapping for test-commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328365 [16:08:35] (03CR) 10Zabe: "has no effect anywhere I believe, but for the sake of completnes" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328365 (owner: 10Zabe) [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:32] (03CR) 10Clément Goubert: memcached: Clean up pki migration switch. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [16:14:59] (03PS5) 10Zabe: Move production reads for commons link tables to x4 for all of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) [16:15:37] (03PS6) 10Zabe: Move production reads for commons link tables to x4 for all requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:40] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:33:08] 06SRE, 10Phabricator: Querying Phabricator DB via phab1005 fails with TLS errors - https://phabricator.wikimedia.org/T437242 (10Urbanecm) 03NEW [16:35:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [16:54:57] (03PS8) 10Blake: memcached: Clean up pki migration switch. [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) [16:56:58] (03CR) 10Blake: memcached: Clean up pki migration switch. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1337558 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1700) [17:00:05] ryankemper: #bothumor My software never has bugs. It just develops random features. Rise for Wikidata Query Service weekly deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T1700). [17:03:52] (03CR) 10Bellevarunchayap: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [17:03:55] (03CR) 10Bellevarunchayap: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [17:18:40] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:32:08] (03CR) 10CWilliams: core-mysql.my.cnf.erb: support templating expire_logs_days for core_test (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1337601 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [17:32:40] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:47:14] (03PS1) 10Ladsgroup-claude: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 [17:49:27] (03CR) 10Ladsgroup: [C:03+1] "Will switch both of them to x4 after setting it up." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328365 (owner: 10Zabe) [17:51:55] (03PS2) 10Ladsgroup: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (owner: 10Ladsgroup-claude) [17:52:19] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (owner: 10Ladsgroup-claude) [17:55:19] (03PS1) 10Zabe: SpecialMostGloballyLinkedFiles: Recache from the globalusage domain [extensions/GlobalUsage] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337646 (https://phabricator.wikimedia.org/T400824) [17:55:20] (03CR) 10CI reject: [V:04-1] Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337644 (owner: 10Ladsgroup-claude) [18:00:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:02:25] RESOLVED: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:20:16] jouncebot: nowandnext [18:20:16] No deployments scheduled for the next 1 hour(s) and 39 minute(s) [18:20:16] In 1 hour(s) and 39 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T2000) [18:20:52] (03CR) 10Zabe: [C:03+2] SpecialMostGloballyLinkedFiles: Recache from the globalusage domain [extensions/GlobalUsage] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337646 (https://phabricator.wikimedia.org/T400824) (owner: 10Zabe) [18:22:08] (03Merged) 10jenkins-bot: SpecialMostGloballyLinkedFiles: Recache from the globalusage domain [extensions/GlobalUsage] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337646 (https://phabricator.wikimedia.org/T400824) (owner: 10Zabe) [18:22:29] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1337646|SpecialMostGloballyLinkedFiles: Recache from the globalusage domain (T400824)]] [18:22:33] T400824: Prepare globalimagelinks for s4 split - https://phabricator.wikimedia.org/T400824 [18:26:37] !log zabe@deploy1003 zabe: Backport for [[gerrit:1337646|SpecialMostGloballyLinkedFiles: Recache from the globalusage domain (T400824)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:27:04] !log zabe@deploy1003 zabe: Continuing with deployment [18:31:41] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337646|SpecialMostGloballyLinkedFiles: Recache from the globalusage domain (T400824)]] (duration: 09m 12s) [18:31:45] T400824: Prepare globalimagelinks for s4 split - https://phabricator.wikimedia.org/T400824 [18:32:41] !log zabe@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [18:33:13] !log zabe@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply [18:52:00] (03CR) 10Zabe: [C:03+2] Set RemoteVirtualDomainsMapping for test-commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328365 (owner: 10Zabe) [18:52:50] (03Merged) 10jenkins-bot: Set RemoteVirtualDomainsMapping for test-commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328365 (owner: 10Zabe) [18:53:11] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1328365|Set RemoteVirtualDomainsMapping for test-commons]] [18:57:16] !log zabe@deploy1003 zabe: Backport for [[gerrit:1328365|Set RemoteVirtualDomainsMapping for test-commons]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:00:49] !log zabe@deploy1003 zabe: Continuing with deployment [19:07:28] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1328365|Set RemoteVirtualDomainsMapping for test-commons]] (duration: 14m 17s) [19:07:44] (03CR) 10Bellevarunchayap: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [19:10:10] (03CR) 10Bellevarunchayap: "Move production reads for commons link tables to x4 for all requests" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [19:18:02] (03CR) 10Zabe: [C:03+2] "scary scary" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [19:18:58] (03Merged) 10jenkins-bot: Move production reads for commons link tables to x4 for 1% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [19:19:37] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1336120|Move production reads for commons link tables to x4 for 1% of requests (T437108)]] [19:19:40] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [19:22:45] (03CR) 10Bellevarunchayap: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [19:23:54] !log zabe@deploy1003 zabe: Backport for [[gerrit:1336120|Move production reads for commons link tables to x4 for 1% of requests (T437108)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:26:36] !log zabe@deploy1003 zabe: Continuing with deployment [19:30:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2005 has a BGP session which is not in the 'established' state. [19:31:17] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1336120|Move production reads for commons link tables to x4 for 1% of requests (T437108)]] (duration: 11m 40s) [19:31:21] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [19:41:56] (03PS1) 10Zabe: Move production reads for commons link tables to x4 for 10% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337650 (https://phabricator.wikimedia.org/T437108) [19:45:11] (03PS7) 10Zabe: Move production reads for commons link tables to x4 for all requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) [19:45:59] (03CR) 10Zabe: [C:03+2] Move production reads for commons link tables to x4 for 10% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337650 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [19:46:48] (03Merged) 10jenkins-bot: Move production reads for commons link tables to x4 for 10% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337650 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [19:48:16] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1337650|Move production reads for commons link tables to x4 for 10% of requests (T437108)]] [19:48:19] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [19:52:26] !log zabe@deploy1003 zabe: Backport for [[gerrit:1337650|Move production reads for commons link tables to x4 for 10% of requests (T437108)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:55:06] !log zabe@deploy1003 zabe: Continuing with deployment [19:59:43] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337650|Move production reads for commons link tables to x4 for 10% of requests (T437108)]] (duration: 11m 27s) [19:59:47] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [20:00:04] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: #bothumor I � Unicode. All rise for UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T2000). [20:00:05] Daimona: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:17] o/ [20:00:42] I can deploy [20:00:45] (03CR) 10Zabe: [C:03+2] Stop setting wgCampaignEventsEnableWorklists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326400 (https://phabricator.wikimedia.org/T429510) (owner: 10Daimona Eaytoy) [20:00:58] Thank you! [20:01:44] (03Merged) 10jenkins-bot: Stop setting wgCampaignEventsEnableWorklists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326400 (https://phabricator.wikimedia.org/T429510) (owner: 10Daimona Eaytoy) [20:02:30] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1326400|Stop setting wgCampaignEventsEnableWorklists (T429510)]] [20:02:33] T429510: Drop feature flag for event worklists - https://phabricator.wikimedia.org/T429510 [20:06:38] !log zabe@deploy1003 zabe, daimona: Backport for [[gerrit:1326400|Stop setting wgCampaignEventsEnableWorklists (T429510)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:07:16] Daimona: I guess it is not really testable? [20:07:49] I verified that the change is a no-op, so that's good enough :) [20:07:59] !log zabe@deploy1003 zabe, daimona: Continuing with deployment [20:08:02] sounds good:) [20:08:56] Great, thank you! [20:09:28] (03PS1) 10Zabe: Move production reads for commons link tables to x4 for 25% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337655 (https://phabricator.wikimedia.org/T437108) [20:12:36] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1326400|Stop setting wgCampaignEventsEnableWorklists (T429510)]] (duration: 10m 06s) [20:12:40] T429510: Drop feature flag for event worklists - https://phabricator.wikimedia.org/T429510 [20:12:54] Daimona: done [20:13:05] (03CR) 10Zabe: [C:03+2] Move production reads for commons link tables to x4 for 25% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337655 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [20:13:55] (03Merged) 10jenkins-bot: Move production reads for commons link tables to x4 for 25% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337655 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [20:14:18] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1337655|Move production reads for commons link tables to x4 for 25% of requests (T437108)]] [20:14:22] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [20:18:35] !log zabe@deploy1003 zabe: Backport for [[gerrit:1337655|Move production reads for commons link tables to x4 for 25% of requests (T437108)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:19:14] !log zabe@deploy1003 zabe: Continuing with deployment [20:23:46] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337655|Move production reads for commons link tables to x4 for 25% of requests (T437108)]] (duration: 09m 28s) [20:23:50] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [20:25:58] (03PS1) 10Zabe: Move production reads for commons link tables to x4 for 50% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337656 (https://phabricator.wikimedia.org/T437108) [20:35:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [20:38:05] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Set weight of db1261 to zero in s4 (T437108)', diff saved to https://phabricator.wikimedia.org/P96385 and previous config saved to /var/cache/conftool/dbconfig/20260907-203804-ladsgroup.json [20:38:08] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [20:43:31] (03PS1) 10AOkoth: phabricator: add skip-ssl option to client config [puppet] - 10https://gerrit.wikimedia.org/r/1337657 (https://phabricator.wikimedia.org/T437242) [20:46:07] (03CR) 10AOkoth: "https://puppet-compiler.wmflabs.org/output/1337657/9378/" [puppet] - 10https://gerrit.wikimedia.org/r/1337657 (https://phabricator.wikimedia.org/T437242) (owner: 10AOkoth) [20:50:52] (03CR) 10Zabe: [C:03+2] Move production reads for commons link tables to x4 for 50% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337656 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [20:51:47] (03Merged) 10jenkins-bot: Move production reads for commons link tables to x4 for 50% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337656 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [20:52:35] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1337656|Move production reads for commons link tables to x4 for 50% of requests (T437108)]] [20:52:39] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [20:57:05] !log zabe@deploy1003 zabe: Backport for [[gerrit:1337656|Move production reads for commons link tables to x4 for 50% of requests (T437108)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:58:04] !log zabe@deploy1003 zabe: Continuing with deployment [21:00:04] alexsanford, Reedy, sbassett, Maryum, and manfredi: gettimeofday() says it's time for Weekly Security deployment window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T2100) [21:03:03] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337656|Move production reads for commons link tables to x4 for 50% of requests (T437108)]] (duration: 10m 27s) [21:03:06] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [21:13:25] RESOLVED: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:26:27] (03PS8) 10Zabe: Move production reads for commons link tables to x4 for all requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) [21:26:56] (03CR) 10Zabe: [C:03+2] Move production reads for commons link tables to x4 for all requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [21:27:51] (03Merged) 10jenkins-bot: Move production reads for commons link tables to x4 for all requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [21:28:07] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1337319|Move production reads for commons link tables to x4 for all requests (T437108)]] [21:28:11] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [21:32:28] !log zabe@deploy1003 zabe: Backport for [[gerrit:1337319|Move production reads for commons link tables to x4 for all requests (T437108)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:32:40] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:33:03] !log zabe@deploy1003 zabe: Continuing with deployment [21:37:41] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337319|Move production reads for commons link tables to x4 for all requests (T437108)]] (duration: 09m 34s) [21:37:44] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [21:37:47] (03CR) 10Zabe: [C:03+2] Move globalimagelinks queries to x4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337635 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [21:39:35] (03Merged) 10jenkins-bot: Move globalimagelinks queries to x4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337635 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [21:40:58] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1337635|Move globalimagelinks queries to x4 (T437108)]] [21:42:22] (03CR) 10Urbanecm: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1337657 (https://phabricator.wikimedia.org/T437242) (owner: 10AOkoth) [21:45:07] !log zabe@deploy1003 zabe: Backport for [[gerrit:1337635|Move globalimagelinks queries to x4 (T437108)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:45:11] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [21:47:23] !log zabe@deploy1003 zabe: Continuing with deployment [21:51:58] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337635|Move globalimagelinks queries to x4 (T437108)]] (duration: 11m 00s) [21:52:02] T437108: Move production reads for commons link tables to x4 - https://phabricator.wikimedia.org/T437108 [22:00:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [22:26:26] (03CR) 10DLynch: "Great, thanks! Are those changes something I should be making patches for myself, or is that something serviceops would be doing?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [22:44:11] PROBLEM - Auth DNS on dns5004 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [22:45:09] RECOVERY - Auth DNS on dns5004 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [22:46:37] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 116049432 and 6 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [22:48:37] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 1634320 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [23:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260907T2300) [23:16:47] PROBLEM - ganeti-noded running on ganeti2044 is CRITICAL: PROCS CRITICAL: 3 processes with UID = 0 (root), command name ganeti-noded https://wikitech.wikimedia.org/wiki/Ganeti [23:17:47] RECOVERY - ganeti-noded running on ganeti2044 is OK: PROCS OK: 2 processes with UID = 0 (root), command name ganeti-noded https://wikitech.wikimedia.org/wiki/Ganeti [23:30:15] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2005 has a BGP session which is not in the 'established' state. [23:41:16] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1337663 [23:41:16] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1337663 (owner: 10TrainBranchBot) [23:49:37] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1337663 (owner: 10TrainBranchBot)