[00:01:24] !log pt1979@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [00:01:47] !log pt1979@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [00:01:49] !log pt1979@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl2006.codfw.wmnet with OS trixie [00:02:01] 06SRE, 06ServiceOps new, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12027390 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host wikikube-ctrl2006.codf... [00:13:23] RESOLVED: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [00:14:23] FIRING: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [00:30:35] (03PS2) 10RLazarus: tox: Bump flake8 to 7.3.0 [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302989 [00:30:35] (03PS2) 10RLazarus: tox: Test up to Python 3.14 [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302990 [00:30:35] (03PS2) 10RLazarus: Release 4.0.5 [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302991 [00:39:23] RESOLVED: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [00:40:23] FIRING: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [00:53:54] (03PS1) 10Jasmine: Add Kubernetes POD IP reverse range delegations for wikikube-ctrl1005 [dns] - 10https://gerrit.wikimedia.org/r/1302996 (https://phabricator.wikimedia.org/T418920) [01:00:23] RESOLVED: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [01:12:15] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1302997 [01:12:15] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1302997 (owner: 10TrainBranchBot) [01:17:41] (03CR) 10ArielGlenn: rest-gateway: emit 401 if rate limit is 0 (033 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1298031 (https://phabricator.wikimedia.org/T428184) (owner: 10Daniel Kinzler) [01:19:23] FIRING: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [01:20:36] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1302997 (owner: 10TrainBranchBot) [01:30:06] (03PS1) 10DDesouza: Add English Wikipedia Mobile App Survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1302998 (https://phabricator.wikimedia.org/T428876) [01:31:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, June 17 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1302998 (https://phabricator.wikimedia.org/T428876) (owner: 10DDesouza) [01:47:35] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:52:35] FIRING: [2x] SystemdUnitFailed: database-backups-snapshots.service on cumin2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:01:26] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:35] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:08:01] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) [02:10:41] (03CR) 10Scott French: [C:03+1] fundraising_data_import maintenance script wrapper & timer (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1271028 (https://phabricator.wikimedia.org/T416948) (owner: 10CDanis) [02:12:35] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:40] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:15:03] (03PS3) 10RLazarus: Release 4.0.5 [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302991 [02:15:03] (03PS1) 10RLazarus: builder: Fix type error and unpin mypy version [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302999 [02:16:11] FIRING: [5x] BFDdown: BFD session down between cr2-eqsin and 103.102.166.8 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [02:21:11] FIRING: [5x] BFDdown: BFD session down between cr2-eqsin and 103.102.166.8 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [02:39:49] (03PS1) 10Clare Ming: Add phabricator api token for Test Kitchen [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303003 (https://phabricator.wikimedia.org/T428986) [02:41:37] (03CR) 10CI reject: [V:04-1] Add phabricator api token for Test Kitchen [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303003 (https://phabricator.wikimedia.org/T428986) (owner: 10Clare Ming) [02:41:58] (03PS2) 10Clare Ming: Add phabricator api token for Test Kitchen [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303003 (https://phabricator.wikimedia.org/T428986) [02:42:28] (03PS1) 10BPirkle: REST: Adjust key of Reading Lists OpenAPI spec in RestSandboxSpecs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303004 (https://phabricator.wikimedia.org/T422771) [02:43:47] (03CR) 10CI reject: [V:04-1] Add phabricator api token for Test Kitchen [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303003 (https://phabricator.wikimedia.org/T428986) (owner: 10Clare Ming) [02:50:07] (03PS3) 10Clare Ming: Add phabricator api token for Test Kitchen [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303003 (https://phabricator.wikimedia.org/T428986) [02:50:57] (03PS4) 10Clare Ming: Add phabricator api token for Test Kitchen [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303003 (https://phabricator.wikimedia.org/T428986) [02:52:45] (03PS5) 10Clare Ming: Add Phabricator specific configuration for Test Kitchen [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303003 (https://phabricator.wikimedia.org/T428986) [02:52:50] (03CR) 10CI reject: [V:04-1] Add Phabricator specific configuration for Test Kitchen [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303003 (https://phabricator.wikimedia.org/T428986) (owner: 10Clare Ming) [02:54:33] (03CR) 10CI reject: [V:04-1] Add Phabricator specific configuration for Test Kitchen [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303003 (https://phabricator.wikimedia.org/T428986) (owner: 10Clare Ming) [02:55:47] 10SRE-Access-Requests: Change SSH key for denisse after new laptop provissioning - https://phabricator.wikimedia.org/T429429 (10andrea.denisse) 03NEW [02:56:43] (03PS1) 10Krinkle: Disable ShortUrl on mrwiki, newiki, pawiki, tawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303006 (https://phabricator.wikimedia.org/T107188) [03:05:50] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2013.codfw.wmnet, wdqs2008.codfw.wmnet, wdqs2011.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:06:12] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2021.codfw.wmnet, wdqs2007.codfw.wmnet, wdqs2008.codfw.wmnet, wdqs2010.codfw.wmnet, wdqs2012.codfw.wmnet, wdqs2013.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:06:34] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:07:34] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:11:12] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:11:50] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:21:20] 10SRE-Access-Requests, 13Patch-For-Review: Change SSH key for denisse after new laptop provissioning - https://phabricator.wikimedia.org/T429429#12027519 (10andrea.denisse) [03:27:37] (03PS3) 10Abijeet Patro: ULS rewrite: Lock body scroll when open on mobile [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1302739 [03:28:26] (03PS1) 10Abijeet Patro: ULS rewrite: Capture trigger element before async module load [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303009 (https://phabricator.wikimedia.org/T429145) [03:28:59] (03PS1) 10Abijeet Patro: ULS rewrite: Show variants even when no languages are available [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303010 (https://phabricator.wikimedia.org/T426532) [03:45:06] (03PS2) 10Abijeet Patro: Enable ULS v2 on group1 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303012 [03:45:17] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, June 17 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303012 (owner: 10Abijeet Patro) [03:46:51] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, June 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303010 (https://phabricator.wikimedia.org/T426532) (owner: 10Abijeet Patro) [03:47:13] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, June 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303009 (https://phabricator.wikimedia.org/T429145) (owner: 10Abijeet Patro) [03:51:24] (03CR) 10Scott French: [C:03+1] test_cli: Update assertEquals to assertEqual [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302988 (owner: 10RLazarus) [03:51:26] (03CR) 10Scott French: [C:03+1] tox: Bump flake8 to 7.3.0 [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302989 (owner: 10RLazarus) [03:51:28] (03CR) 10Scott French: [C:03+1] tox: Test up to Python 3.14 [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302990 (owner: 10RLazarus) [03:51:31] (03CR) 10Scott French: [C:03+1] builder: Fix type error and unpin mypy version [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302999 (owner: 10RLazarus) [03:51:40] (03CR) 10Scott French: [C:03+1] Release 4.0.5 [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302991 (owner: 10RLazarus) [05:02:43] (03CR) 10WMDE-Fisch: "Thanks a lot 🙏" [extensions/VisualEditor] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1302872 (https://phabricator.wikimedia.org/T428764) (owner: 10WMDE-Fisch) [05:03:08] (03Abandoned) 10WMDE-Fisch: Fix VE core submodule update to 3e79e9934 [extensions/VisualEditor] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1302872 (https://phabricator.wikimedia.org/T428764) (owner: 10WMDE-Fisch) [05:07:41] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for laurabarluzzi - https://phabricator.wikimedia.org/T429431 (10Laurabarluzzi) 03NEW [05:19:38] FIRING: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [05:19:57] (03CR) 10Marostegui: [C:03+2] mysql-gtid.yaml: Add pint [alerts] - 10https://gerrit.wikimedia.org/r/1302724 (https://phabricator.wikimedia.org/T427469) (owner: 10Marostegui) [05:23:09] 06SRE, 10observability, 06SRE Observability, 13Patch-For-Review: Alerts showing "AlertLintProblem" - MySQLReplicaNotUsingGTID - https://phabricator.wikimedia.org/T427469#12027616 (10Marostegui) Merged - let's give it sometime to see if this fixes the problem. [05:32:06] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1224 is unreachable - https://phabricator.wikimedia.org/T427535#12027626 (10Marostegui) @VRiley-WMF did you have some time to check if any of the above hosts could work to get some pieces to replace the ones failing on db1224? Thanks! [05:37:10] !log marostegui@cumin1003 START - Cookbook sre.mysql.major-upgrade [05:37:30] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool es1037: Upgrading es1037.eqiad.wmnet [05:37:40] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1037: Upgrading es1037.eqiad.wmnet [05:38:38] !log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host es1037.eqiad.wmnet with OS trixie [05:47:35] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [05:52:35] FIRING: [2x] SystemdUnitFailed: database-backups-snapshots.service on cumin2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:54:17] !log marostegui@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on es1037.eqiad.wmnet with reason: host reimage [05:59:12] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1037.eqiad.wmnet with reason: host reimage [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260617T0600) [06:04:44] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: Degraded RAID on db2247 - https://phabricator.wikimedia.org/T429348#12027675 (10Marostegui) p:05Triage→03Medium [06:07:35] FIRING: [4x] SystemdUnitFailed: database-backups-snapshots.service on cumin2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:12:41] PROBLEM - OSPF status on cr1-drmrs is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:14:41] RECOVERY - OSPF status on cr1-drmrs is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:15:39] FIRING: [4x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [06:16:11] FIRING: [8x] BFDdown: BFD session down between cr1-drmrs and 185.15.58.138 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [06:16:20] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1037.eqiad.wmnet with OS trixie [06:20:05] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T428832#12027720 (10Marostegui) @Jclark-ctr what do you think of the above? worth investigating or should I repool? [06:20:39] RESOLVED: [4x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr2-eqiad (185.15.58.138) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [06:28:34] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool es1037: Migration of es1037.eqiad.wmnet completed [06:29:23] RESOLVED: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [06:31:43] 06SRE, 06Infrastructure-Foundations: Upgrade Cumin hosts to Trixie - https://phabricator.wikimedia.org/T427897#12027726 (10MoritzMuehlenhoff) @jcrespo Cumin is now working on cumin2003, you can test backups. [06:32:32] (03PS1) 10Gerrit maintenance bot: mariadb: Promote es1037 to es6 master [puppet] - 10https://gerrit.wikimedia.org/r/1303286 (https://phabricator.wikimedia.org/T429436) [06:32:38] (03PS1) 10Gerrit maintenance bot: wmnet: Update es6-master alias [dns] - 10https://gerrit.wikimedia.org/r/1303287 (https://phabricator.wikimedia.org/T429436) [06:35:46] (03PS1) 10Marostegui: db-production.php: Disable writes on es6 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303288 (https://phabricator.wikimedia.org/T429118) [06:43:59] PROBLEM - Host lsw1-b7-codfw.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [06:44:08] (03PS1) 10Giuseppe Lavagetto: Re-release changes introduced previously, plus use a trie to aggregate ipblocks [software/hiddenparma/deploy] - 10https://gerrit.wikimedia.org/r/1303289 [06:44:26] (03CR) 10Giuseppe Lavagetto: [V:03+2 C:03+2] Re-release changes introduced previously, plus use a trie to aggregate ipblocks [software/hiddenparma/deploy] - 10https://gerrit.wikimedia.org/r/1303289 (owner: 10Giuseppe Lavagetto) [06:44:36] (03CR) 10Elukey: [C:03+2] preseed: fix partman config for the new conf2* hosts [puppet] - 10https://gerrit.wikimedia.org/r/1302921 (https://phabricator.wikimedia.org/T418914) (owner: 10Elukey) [06:44:43] PROBLEM - Host ps1-b7-codfw is DOWN: PING CRITICAL - Packet loss = 100% [06:46:01] !log oblivian@cumin1003 START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Haproxy provenance maps in HP; UX changes - oblivian@cumin1003" [06:46:03] !log oblivian@cumin1003 START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Haproxy provenance maps in HP; UX changes - oblivian@cumin1003 [06:46:54] !log oblivian@cumin1003 END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Haproxy provenance maps in HP; UX changes - oblivian@cumin1003 [06:46:55] !log oblivian@cumin1003 END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Haproxy provenance maps in HP; UX changes - oblivian@cumin1003" [06:48:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-b7-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:50:19] (03PS1) 10Marostegui: Revert "es2045: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1303290 [06:51:28] (03PS1) 10Giuseppe Lavagetto: Revert "Re-release changes introduced previously, plus use a trie to aggregate ipblocks" [software/hiddenparma/deploy] - 10https://gerrit.wikimedia.org/r/1303291 [06:51:51] (03CR) 10Giuseppe Lavagetto: [V:03+2 C:03+2] Revert "Re-release changes introduced previously, plus use a trie to aggregate ipblocks" [software/hiddenparma/deploy] - 10https://gerrit.wikimedia.org/r/1303291 (owner: 10Giuseppe Lavagetto) [06:52:18] !log oblivian@cumin1003 START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "revert deployment - oblivian@cumin1003" [06:52:19] !log oblivian@cumin1003 START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: revert deployment - oblivian@cumin1003 [06:53:12] !log oblivian@cumin1003 END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: revert deployment - oblivian@cumin1003 [06:53:13] !log oblivian@cumin1003 END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "revert deployment - oblivian@cumin1003" [06:53:40] (03CR) 10Muehlenhoff: [C:03+2] Remove the black box check for mirrors.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1302858 (https://phabricator.wikimedia.org/T416707) (owner: 10Muehlenhoff) [06:55:43] RECOVERY - Host ps1-b7-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.66 ms [06:56:01] RECOVERY - Host lsw1-b7-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 31.88 ms [06:58:22] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-b7-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:59:20] (03CR) 10Elukey: sre.hosts.provision: introduce the wmfroot user (034 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1291994 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [07:00:01] (03CR) 10Nikerabbit: [C:03+1] ULS rewrite: Capture trigger element before async module load [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303009 (https://phabricator.wikimedia.org/T429145) (owner: 10Abijeet Patro) [07:00:05] Amir1, urbanecm, and awight: UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260617T0700). Please do the needful. [07:00:05] abijeet: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:33] (03CR) 10Nikerabbit: [C:03+1] ULS rewrite: Show variants even when no languages are available [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303010 (https://phabricator.wikimedia.org/T426532) (owner: 10Abijeet Patro) [07:00:47] (03CR) 10Nikerabbit: [C:03+1] Enable ULS v2 on group1 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303012 (owner: 10Abijeet Patro) [07:02:00] o/ [07:04:36] (03PS7) 10Elukey: sre.hosts.provision: introduce the wmfroot user [cookbooks] - 10https://gerrit.wikimedia.org/r/1291994 (https://phabricator.wikimedia.org/T426180) [07:04:59] (03CR) 10Elukey: sre.hosts.provision: introduce the wmfroot user (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1291994 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [07:12:15] (03CR) 10Nikerabbit: [C:03+1] ULS rewrite: Lock body scroll when open on mobile [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1302739 (owner: 10Abijeet Patro) [07:12:34] (03CR) 10Nikerabbit: [C:03+1] ULS rewrite: Fix settings dialog width and field sizing [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1302743 (https://phabricator.wikimedia.org/T416512) (owner: 10Abijeet Patro) [07:14:05] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1037: Migration of es1037.eqiad.wmnet completed [07:14:06] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0) [07:15:25] !log marostegui@cumin1003 START - Cookbook sre.mysql.major-upgrade [07:15:42] (03CR) 10JMeybohm: [C:03+1] "Nice!" [debs/helm3] - 10https://gerrit.wikimedia.org/r/1300145 (https://phabricator.wikimedia.org/T427403) (owner: 10Jelto) [07:15:45] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool es1044: Upgrading es1044.eqiad.wmnet [07:16:16] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1044: Upgrading es1044.eqiad.wmnet [07:17:26] !log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host es1044.eqiad.wmnet with OS trixie [07:20:11] (03PS1) 10Giuseppe Lavagetto: Code changes [software/hiddenparma/deploy] - 10https://gerrit.wikimedia.org/r/1303294 [07:20:30] (03CR) 10Giuseppe Lavagetto: [V:03+2 C:03+2] Code changes [software/hiddenparma/deploy] - 10https://gerrit.wikimedia.org/r/1303294 (owner: 10Giuseppe Lavagetto) [07:21:41] !log oblivian@cumin1003 START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Haproxy provenance maps in HP; UX changes (attempt 3) - oblivian@cumin1003" [07:21:43] !log oblivian@cumin1003 START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Haproxy provenance maps in HP; UX changes (attempt 3) - oblivian@cumin1003 [07:21:50] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [07:22:31] !log oblivian@cumin1003 END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Haproxy provenance maps in HP; UX changes (attempt 3) - oblivian@cumin1003 [07:22:32] !log oblivian@cumin1003 END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Haproxy provenance maps in HP; UX changes (attempt 3) - oblivian@cumin1003" [07:22:57] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [07:23:23] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host conf2007.codfw.wmnet with OS trixie [07:23:51] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [07:25:45] (03PS1) 10Matthias Mullie: Squashed diff to master [extensions/MultimediaViewer] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303295 [07:26:10] (03PS1) 10Matthias Mullie: Squashed diff to master [extensions/MultimediaViewer] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303296 [07:27:29] (03CR) 10CI reject: [V:04-1] Squashed diff to master [extensions/MultimediaViewer] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303295 (owner: 10Matthias Mullie) [07:30:44] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host conf2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [07:32:29] !log marostegui@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on es1044.eqiad.wmnet with reason: host reimage [07:33:35] (03CR) 10Giuseppe Lavagetto: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/8747/co" [puppet] - 10https://gerrit.wikimedia.org/r/1299939 (https://phabricator.wikimedia.org/T422249) (owner: 10Giuseppe Lavagetto) [07:34:46] (03CR) 10Giuseppe Lavagetto: [V:03+1 C:03+2] haproxy: get ipblock map directly from HP [puppet] - 10https://gerrit.wikimedia.org/r/1299939 (https://phabricator.wikimedia.org/T422249) (owner: 10Giuseppe Lavagetto) [07:35:07] (03CR) 10Matthias Mullie: "recheck" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303295 (owner: 10Matthias Mullie) [07:39:40] (03PS2) 10Matthias Mullie: Squashed diff to master [extensions/MultimediaViewer] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303295 [07:39:41] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on conf2007.codfw.wmnet with reason: host reimage [07:40:08] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1044.eqiad.wmnet with reason: host reimage [07:40:48] (03CR) 10CI reject: [V:04-1] Squashed diff to master [extensions/MultimediaViewer] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303295 (owner: 10Matthias Mullie) [07:41:34] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host conf2008.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [07:42:13] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host conf2008.codfw.wmnet with OS trixie [07:43:19] (03PS3) 10Matthias Mullie: Squashed diff to master [extensions/MultimediaViewer] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303295 [07:43:25] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [07:43:33] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [07:43:55] (03CR) 10Arnaudb: [C:03+2] ci: monitor a wider variety of network errors [puppet] - 10https://gerrit.wikimedia.org/r/1302829 (https://phabricator.wikimedia.org/T420865) (owner: 10Arnaudb) [07:44:38] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2007.codfw.wmnet with reason: host reimage [07:46:27] noone deploying atm? I'll use the remainder of this slot if that's alright :) [07:46:35] (03CR) 10Filippo Giunchedi: [C:03+1] ceph: allow to set client transport encryption [puppet] - 10https://gerrit.wikimedia.org/r/1302904 (https://phabricator.wikimedia.org/T294432) (owner: 10Volans) [07:47:12] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303296 (owner: 10Matthias Mullie) [07:47:12] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303295 (owner: 10Matthias Mullie) [07:47:33] (03PS1) 10Giuseppe Lavagetto: cache::haproxy: fix reference in template [puppet] - 10https://gerrit.wikimedia.org/r/1303300 [07:47:45] (03CR) 10Filippo Giunchedi: [C:03+1] Cinder backups: enable transport encryption part 1 [puppet] - 10https://gerrit.wikimedia.org/r/1302905 (https://phabricator.wikimedia.org/T294432) (owner: 10Volans) [07:47:50] (03CR) 10Giuseppe Lavagetto: [V:03+2 C:03+2] cache::haproxy: fix reference in template [puppet] - 10https://gerrit.wikimedia.org/r/1303300 (owner: 10Giuseppe Lavagetto) [07:48:21] (03Merged) 10jenkins-bot: Squashed diff to master [extensions/MultimediaViewer] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303296 (owner: 10Matthias Mullie) [07:48:22] (03Merged) 10jenkins-bot: Squashed diff to master [extensions/MultimediaViewer] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303295 (owner: 10Matthias Mullie) [07:49:51] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1303296|Squashed diff to master]], [[gerrit:1303295|Squashed diff to master]] [07:50:13] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [07:51:31] (03PS1) 10Slyngshede: C:apereo_cas: Script for resetting webauthn device registration [puppet] - 10https://gerrit.wikimedia.org/r/1303304 [07:52:04] (03CR) 10CI reject: [V:04-1] C:apereo_cas: Script for resetting webauthn device registration [puppet] - 10https://gerrit.wikimedia.org/r/1303304 (owner: 10Slyngshede) [07:52:40] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/8748/co" [puppet] - 10https://gerrit.wikimedia.org/r/1303304 (owner: 10Slyngshede) [07:53:13] (03PS4) 10Abijeet Patro: ULS rewrite: Lock body scroll when open on mobile [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1302739 [07:53:52] (03PS1) 10Abijeet Patro: ULS rewrite: Lock scroll too, not just [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303323 [07:53:55] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [07:54:11] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, June 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303323 (owner: 10Abijeet Patro) [07:54:29] (03PS2) 10Abijeet Patro: ULS rewrite: Sync the fullscreen mobile selector with a URL route [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303297 (https://phabricator.wikimedia.org/T428778) [07:54:39] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, June 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [extensions/UniversalLanguageSelector] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303297 (https://phabricator.wikimedia.org/T428778) (owner: 10Abijeet Patro) [07:57:53] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1044.eqiad.wmnet with OS trixie [07:58:06] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [07:58:07] (03CR) 10Muehlenhoff: [C:03+2] Apply urldownloader role to urldownloader1005/1006/2006 [puppet] - 10https://gerrit.wikimedia.org/r/1302805 (https://phabricator.wikimedia.org/T427282) (owner: 10Muehlenhoff) [07:59:57] !log marostegui@cumin1003 END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) [08:00:06] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on conf2008.codfw.wmnet with reason: host reimage [08:00:12] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool es1044: repool after upgrade [08:01:31] !log btullis@cumin1003 END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003" [08:01:31] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-wdqs1001.eqiad.wmnet with OS bookworm [08:03:06] !log elukey@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" [08:03:32] PROBLEM - Improperly owned -0:0- files in /srv/mediawiki-staging on deploy1003 is CRITICAL: Improperly owned (0:0) files in /srv/mediawiki-staging https://wikitech.wikimedia.org/wiki/Monitoring/bad_directory_owner [08:04:10] !log elukey@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" [08:04:10] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2007.codfw.wmnet with OS trixie [08:04:23] PROBLEM - statsv Varnishkafka log producer on cp7005 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [08:04:34] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2008.codfw.wmnet with reason: host reimage [08:04:51] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [08:05:23] RECOVERY - statsv Varnishkafka log producer on cp7005 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [08:06:07] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [08:06:51] btullis@cumin1003 reimage (PID 348375) is awaiting input [08:07:29] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm [08:07:43] (03CR) 10Jelto: [V:03+1 C:03+2] Build helm3.19 with new upstream version [debs/helm3] - 10https://gerrit.wikimedia.org/r/1300145 (https://phabricator.wikimedia.org/T427403) (owner: 10Jelto) [08:08:32] (03CR) 10Marostegui: [C:03+2] Revert "es2045: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1303290 (owner: 10Marostegui) [08:09:27] !log mlitn@deploy1003 mlitn: Backport for [[gerrit:1303296|Squashed diff to master]], [[gerrit:1303295|Squashed diff to master]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:12:02] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host conf2009.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [08:12:20] (03CR) 10Mvolz: [C:03+2] citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1300750 (owner: 10PipelineBot) [08:12:21] !log mlitn@deploy1003 mlitn: Continuing with deployment [08:13:08] (03PS2) 10Slyngshede: C:apereo_cas: Script for resetting webauthn device registration [puppet] - 10https://gerrit.wikimedia.org/r/1303304 [08:13:37] 06SRE, 06Data-Engineering, 06Data-Platform-SRE, 06serviceops-radar, 10Event-Platform: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12028023 (10RKemper) We should be able to get the PoC working again by bumping confluent-kafka to 2.14.2 (latest) and swapping out... [08:14:00] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host conf2009.codfw.wmnet with OS trixie [08:14:26] (03Merged) 10jenkins-bot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1300750 (owner: 10PipelineBot) [08:17:48] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm [08:19:15] (03PS1) 10Muehlenhoff: admin-ng: Allow the new URL downloaders [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303341 (https://phabricator.wikimedia.org/T427282) [08:21:40] (03PS24) 10Trueg: dse-k8s-services: WDQS deployment helmfile values [deployment-charts] - 10https://gerrit.wikimedia.org/r/1297067 (https://phabricator.wikimedia.org/T424338) [08:22:24] (03PS4) 10Trueg: dse-k8s-services: Enable ingress on WDQS namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1302784 (https://phabricator.wikimedia.org/T429313) [08:22:56] !log elukey@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" [08:23:13] !log elukey@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" [08:23:13] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2008.codfw.wmnet with OS trixie [08:23:28] (03PS1) 10Giuseppe Lavagetto: requestctl_cli: add update-provenance-map command [puppet] - 10https://gerrit.wikimedia.org/r/1303342 [08:23:47] (03PS1) 10Kevin Bazira: ml-services: deploy cope-b-a4b isvc that trims response to violation, p_violation, p_safe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303343 (https://phabricator.wikimedia.org/T427497) [08:25:25] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1303296|Squashed diff to master]], [[gerrit:1303295|Squashed diff to master]] (duration: 35m 34s) [08:25:58] (03CR) 10Bartosz Wójtowicz: [C:03+1] ml-services: deploy cope-b-a4b isvc that trims response to violation, p_violation, p_safe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303343 (https://phabricator.wikimedia.org/T427497) (owner: 10Kevin Bazira) [08:27:13] (03CR) 10Giuseppe Lavagetto: [C:03+2] requestctl_cli: add update-provenance-map command [puppet] - 10https://gerrit.wikimedia.org/r/1303342 (owner: 10Giuseppe Lavagetto) [08:29:10] (03PS1) 10Arnaudb: ci: update jenkins build monitor [puppet] - 10https://gerrit.wikimedia.org/r/1303344 (https://phabricator.wikimedia.org/T420865) [08:29:44] (03CR) 10Arnaudb: [C:03+2] ci: update jenkins build monitor [puppet] - 10https://gerrit.wikimedia.org/r/1303344 (https://phabricator.wikimedia.org/T420865) (owner: 10Arnaudb) [08:30:35] (03PS4) 10Arnaudb: gerrit: add 5xx and 4xx alert thresholds [alerts] - 10https://gerrit.wikimedia.org/r/1301233 (https://phabricator.wikimedia.org/T428979) [08:30:59] (03CR) 10Gmodena: EventStreamConfig: add stream for WDQS V2 external queries. (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1302923 (https://phabricator.wikimedia.org/T429380) (owner: 10Lerickson) [08:31:11] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on conf2009.codfw.wmnet with reason: host reimage [08:31:40] (03CR) 10Gmodena: "+ Andrew and Thomas for an ACK." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1302923 (https://phabricator.wikimedia.org/T429380) (owner: 10Lerickson) [08:32:36] (03CR) 10Brouberol: dse-k8s-services: WDQS deployment helmfile values (033 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1297067 (https://phabricator.wikimedia.org/T424338) (owner: 10Trueg) [08:32:48] 06SRE, 06Data-Engineering, 06Data-Platform-SRE, 06serviceops-radar, 10Event-Platform: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12028089 (10elukey) We have a Kafka upgrade Cloud project that we could use with Pontoon, we could use it to create the PoC in ther... [08:33:25] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy cope-b-a4b isvc that trims response to violation, p_violation, p_safe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303343 (https://phabricator.wikimedia.org/T427497) (owner: 10Kevin Bazira) [08:34:22] (03PS1) 10Joal: Update turnilo config for banner_activity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303346 (https://phabricator.wikimedia.org/T414478) [08:35:09] (03CR) 10Btullis: [C:03+1] Update turnilo config for banner_activity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303346 (https://phabricator.wikimedia.org/T414478) (owner: 10Joal) [08:35:18] (03PS2) 10Joal: Update turnilo config for banner_activity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303346 (https://phabricator.wikimedia.org/T414478) [08:35:20] (03CR) 10Trueg: dse-k8s-services: WDQS deployment helmfile values (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1297067 (https://phabricator.wikimedia.org/T424338) (owner: 10Trueg) [08:35:33] (03Merged) 10jenkins-bot: ml-services: deploy cope-b-a4b isvc that trims response to violation, p_violation, p_safe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303343 (https://phabricator.wikimedia.org/T427497) (owner: 10Kevin Bazira) [08:35:47] (03CR) 10Fabfur: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1301883 (https://phabricator.wikimedia.org/T422249) (owner: 10Giuseppe Lavagetto) [08:35:58] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2009.codfw.wmnet with reason: host reimage [08:36:37] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [08:37:29] (03CR) 10Brouberol: dse-k8s-services: Enable ingress on WDQS namespaces (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1302784 (https://phabricator.wikimedia.org/T429313) (owner: 10Trueg) [08:37:34] (03CR) 10Btullis: [C:03+2] Update turnilo config for banner_activity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303346 (https://phabricator.wikimedia.org/T414478) (owner: 10Joal) [08:38:08] !log installing apache2 security updates [08:38:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:38:36] !log "Imported helm3 3.19.5-1 to bullseye-wikimedia, bookworm-wikimedia and trixie-wikimedia - T427403" [08:38:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:38:41] T427403: Update helm to 3.19 - https://phabricator.wikimedia.org/T427403 [08:39:34] (03PS2) 10Giuseppe Lavagetto: fetch_external_clouds_vendors_nets: commit changes to provenance map [puppet] - 10https://gerrit.wikimedia.org/r/1301883 (https://phabricator.wikimedia.org/T422249) [08:39:44] (03Merged) 10jenkins-bot: Update turnilo config for banner_activity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303346 (https://phabricator.wikimedia.org/T414478) (owner: 10Joal) [08:41:38] (03CR) 10Fabfur: [C:03+1] haproxy: use ipblocks map created by hiddenparma [puppet] - 10https://gerrit.wikimedia.org/r/1299940 (https://phabricator.wikimedia.org/T422249) (owner: 10Giuseppe Lavagetto) [08:41:46] (03PS25) 10Trueg: dse-k8s-services: WDQS deployment helmfile values [deployment-charts] - 10https://gerrit.wikimedia.org/r/1297067 (https://phabricator.wikimedia.org/T424338) [08:43:03] (03PS5) 10Trueg: dse-k8s-services: Enable ingress on WDQS namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1302784 (https://phabricator.wikimedia.org/T429313) [08:43:44] 10ops-codfw, 06DC-Ops: Power Supply - Status - issue on cirrussearch2080:9290 - https://phabricator.wikimedia.org/T429448 (10phaultfinder) 03NEW [08:44:22] (03CR) 10Giuseppe Lavagetto: [C:03+2] fetch_external_clouds_vendors_nets: commit changes to provenance map [puppet] - 10https://gerrit.wikimedia.org/r/1301883 (https://phabricator.wikimedia.org/T422249) (owner: 10Giuseppe Lavagetto) [08:45:07] (03CR) 10Fabfur: [C:03+1] "lgtm, [nit] missing Bug on commit message" [puppet] - 10https://gerrit.wikimedia.org/r/1299941 (owner: 10Giuseppe Lavagetto) [08:45:36] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1044: repool after upgrade [08:46:04] (03CR) 10Clément Goubert: rest-gateway: emit 401 if rate limit is 0 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1298031 (https://phabricator.wikimedia.org/T428184) (owner: 10Daniel Kinzler) [08:46:20] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s1 T429190 [08:46:23] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm [08:46:24] T429190: Switchover s1 master (db2203 -> db2212) - https://phabricator.wikimedia.org/T429190 [08:46:44] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Set db2212 with weight 0 T429190', diff saved to https://phabricator.wikimedia.org/P94215 and previous config saved to /var/cache/conftool/dbconfig/20260617-084642-cwilliams.json [08:48:31] (03PS1) 10Marostegui: wmnet: Update es5-master CNAME [dns] - 10https://gerrit.wikimedia.org/r/1303347 (https://phabricator.wikimedia.org/T428572) [08:48:46] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [08:49:33] (03PS2) 10Gerrit maintenance bot: mariadb: Promote db2212 to s1 master [puppet] - 10https://gerrit.wikimedia.org/r/1302167 (https://phabricator.wikimedia.org/T429190) [08:49:34] (03CR) 10Marostegui: [C:03+2] wmnet: Update es5-master CNAME [dns] - 10https://gerrit.wikimedia.org/r/1303347 (https://phabricator.wikimedia.org/T428572) (owner: 10Marostegui) [08:49:38] !log marostegui@dns1004 START - running authdns-update [08:50:00] (03CR) 10JMeybohm: [C:03+1] admin-ng: Allow the new URL downloaders [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303341 (https://phabricator.wikimedia.org/T427282) (owner: 10Muehlenhoff) [08:51:21] !log marostegui@dns1004 END - running authdns-update [08:51:30] (03CR) 10CWilliams: [C:03+2] mariadb: Promote db2212 to s1 master [puppet] - 10https://gerrit.wikimedia.org/r/1302167 (https://phabricator.wikimedia.org/T429190) (owner: 10Gerrit maintenance bot) [08:51:54] !log Starting s1 codfw failover from db2203 to db2212 - T429190 [08:51:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:51:58] T429190: Switchover s1 master (db2203 -> db2212) - https://phabricator.wikimedia.org/T429190 [08:52:59] (03CR) 10Elukey: [C:03+1] admin-ng: Allow the new URL downloaders [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303341 (https://phabricator.wikimedia.org/T427282) (owner: 10Muehlenhoff) [08:53:11] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Promote db2212 to s1 primary T429190', diff saved to https://phabricator.wikimedia.org/P94217 and previous config saved to /var/cache/conftool/dbconfig/20260617-085310-cwilliams.json [08:55:03] !log elukey@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" [08:55:24] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps new, 10ServiceOps-Upgrades-Hardware: Q3:rack/setup/install conf200[7-9] - https://phabricator.wikimedia.org/T418914#12028189 (10elukey) All hosts provisioned and reimaged. Please keep in mind that I used test-cookbook to test new changes, they are still in code... [08:55:28] !log elukey@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1003" [08:55:28] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2009.codfw.wmnet with OS trixie [08:56:16] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depool db2203 T429190', diff saved to https://phabricator.wikimedia.org/P94218 and previous config saved to /var/cache/conftool/dbconfig/20260617-085615-cwilliams.json [08:57:02] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs2001.codfw.wmnet with OS bookworm [08:57:14] (03CR) 10Brouberol: dse-k8s-services: WDQS deployment helmfile values (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1297067 (https://phabricator.wikimedia.org/T424338) (owner: 10Trueg) [08:58:21] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 2 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [08:58:38] (03PS3) 10Giuseppe Lavagetto: haproxy: use ipblocks map created by hiddenparma [puppet] - 10https://gerrit.wikimedia.org/r/1299940 (https://phabricator.wikimedia.org/T422249) [08:59:06] (03PS1) 10JavierMonton: stream: mw-page-html-content-change-enrich [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303349 (https://phabricator.wikimedia.org/T425336) [08:59:44] !log joal@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply [08:59:59] !log joal@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply [09:01:50] !log joal@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply [09:02:03] !log joal@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply [09:02:22] !log marostegui@cumin1003 dbctl commit (dc=all): 'Promote es2046 to es5 codfw primary T428572', diff saved to https://phabricator.wikimedia.org/P94219 and previous config saved to /var/cache/conftool/dbconfig/20260617-090221-marostegui.json [09:02:26] T428572: Migrate es5 section to Debian Trixie - https://phabricator.wikimedia.org/T428572 [09:02:28] (03CR) 10Trueg: dse-k8s-services: Enable ingress on WDQS namespaces (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1302784 (https://phabricator.wikimedia.org/T429313) (owner: 10Trueg) [09:03:41] (03CR) 10Trueg: dse-k8s-services: WDQS deployment helmfile values (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1297067 (https://phabricator.wikimedia.org/T424338) (owner: 10Trueg) [09:03:44] (03PS1) 10Blake: mcrouter_wancache: Remove 2 gutterpool servers for maintenance. [puppet] - 10https://gerrit.wikimedia.org/r/1303350 (https://phabricator.wikimedia.org/T426044) [09:04:58] !log marostegui@cumin1003 START - Cookbook sre.mysql.major-upgrade [09:05:09] (03PS12) 10Aleksandar Mastilovic: Presto memory tuning, resource groups [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) [09:05:16] (03CR) 10Aleksandar Mastilovic: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) (owner: 10Aleksandar Mastilovic) [09:05:19] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool es2044: Upgrading es2044.codfw.wmnet [09:05:24] (03PS6) 10Trueg: dse-k8s-services: Enable ingress on WDQS namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1302784 (https://phabricator.wikimedia.org/T429313) [09:05:39] (03CR) 10Aleksandar Mastilovic: Presto memory tuning, resource groups (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) (owner: 10Aleksandar Mastilovic) [09:05:57] (03CR) 10Aleksandar Mastilovic: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) (owner: 10Aleksandar Mastilovic) [09:05:58] !log marostegui@cumin1003 END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es2044: Upgrading es2044.codfw.wmnet [09:06:08] !log marostegui@cumin1003 END (FAIL) - Cookbook sre.mysql.major-upgrade (exit_code=99) [09:06:24] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool es2045: repool after maintenance es2045 [09:07:08] 06SRE: Rework ACLs on Kafka 3.x clusters - https://phabricator.wikimedia.org/T425528#12028243 (10elukey) Added ACls in logging-codfw: ` elukey@kafka-logging2002:~$ bash /etc/kafka/acls.sh kafka-acls --bootstrap-server kafka-logging2001.codfw.wmnet:9092,kafka-logging2002.codfw.wmnet:9092,kafka-logging2003.codfw.... [09:07:45] !log add basic Kafka ACLs for anonymous to logging-codfw - T425528 (I'll add rollback steps in the task if needed) [09:07:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:07:49] T425528: Rework ACLs on Kafka 3.x clusters - https://phabricator.wikimedia.org/T425528 [09:08:23] (03CR) 10Aleksandar Mastilovic: Presto memory tuning, resource groups (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) (owner: 10Aleksandar Mastilovic) [09:09:26] !log cwilliams@cumin1003 START - Cookbook sre.mysql.major-upgrade [09:09:37] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db2203: Upgrading db2203.codfw.wmnet [09:09:48] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2203: Upgrading db2203.codfw.wmnet [09:10:47] 06SRE: Rework ACLs on Kafka 3.x clusters - https://phabricator.wikimedia.org/T425528#12028264 (10elukey) To rollback the above if needed: * ssh to any kafka-logging2* * cat /etc/kafka/acls.sh * replace "--add" with "--remove" * Execute the commands using `sudo -E` [09:11:33] !log cwilliams@cumin1003 START - Cookbook sre.hosts.reimage for host db2203.codfw.wmnet with OS trixie [09:12:54] (03PS1) 10Jcrespo: Revert^4 "dbbackups: Testing x1 backups on new cumin2003 trixie host" [puppet] - 10https://gerrit.wikimedia.org/r/1303352 [09:14:33] (03PS2) 10Jcrespo: Revert^4 "dbbackups: Testing x1 backups on new cumin2003 trixie host" [puppet] - 10https://gerrit.wikimedia.org/r/1303352 (https://phabricator.wikimedia.org/T427897) [09:14:41] (03CR) 10Jcrespo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1303352 (https://phabricator.wikimedia.org/T427897) (owner: 10Jcrespo) [09:15:06] (03CR) 10Santiago Faci: Add Phabricator specific configuration for Test Kitchen (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303003 (https://phabricator.wikimedia.org/T428986) (owner: 10Clare Ming) [09:16:46] (03CR) 10Jcrespo: [C:03+2] Revert^4 "dbbackups: Testing x1 backups on new cumin2003 trixie host" [puppet] - 10https://gerrit.wikimedia.org/r/1303352 (https://phabricator.wikimedia.org/T427897) (owner: 10Jcrespo) [09:17:34] Should I merge mariadb: Promote db2212 to s1 master (27708f85b7) ? [09:17:57] cezmunsta: ^ [09:18:07] (03PS1) 10Dreamy Jazz: Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits [extensions/DiscussionTools] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303354 (https://phabricator.wikimedia.org/T428883) [09:18:23] jouncebot: nowandnext [09:18:23] No deployments scheduled for the next 0 hour(s) and 41 minute(s) [09:18:23] In 0 hour(s) and 41 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260617T1000) [09:18:30] Any problem with me using scap? [09:18:30] (03PS1) 10Blake: test_main: Remove redundant wildcards. [puppet] - 10https://gerrit.wikimedia.org/r/1303353 (https://phabricator.wikimedia.org/T428772) [09:21:24] 10ops-eqiad, 06SRE, 06DC-Ops: Q3 :rack/setup/install cloudvirt refresh - https://phabricator.wikimedia.org/T425088#12028314 (10cmooney) >>! In T425088#12023306, @Jclark-ctr wrote: > Looks like this is failing with the provision script. @cmooney Said he can take a look later to resolve it > > > > ` > clo... [09:21:48] (03CR) 10Gmodena: dse-k8s-services: WDQS deployment helmfile values (037 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1297067 (https://phabricator.wikimedia.org/T424338) (owner: 10Trueg) [09:22:15] Going to use scap shortly [09:23:00] 10ops-eqiad, 06SRE, 06DC-Ops: Q3 :rack/setup/install cloudvirt refresh - https://phabricator.wikimedia.org/T425088#12028315 (10Jclark-ctr) Thank you! [09:23:36] (03PS1) 10Cathal Mooney: Provision script: improve error message if two prefixes found for vlan [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1303355 (https://phabricator.wikimedia.org/T425088) [09:23:57] (03CR) 10Gmodena: [C:03+1] "LGTM! LGTM! If helmfile works locally, let's merge and iterate on any issues in follow up patches." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1297067 (https://phabricator.wikimedia.org/T424338) (owner: 10Trueg) [09:25:07] (03CR) 10Trueg: dse-k8s-services: WDQS deployment helmfile values (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1297067 (https://phabricator.wikimedia.org/T424338) (owner: 10Trueg) [09:26:23] (03CR) 10Filippo Giunchedi: [C:03+1] Provision script: improve error message if two prefixes found for vlan [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1303355 (https://phabricator.wikimedia.org/T425088) (owner: 10Cathal Mooney) [09:26:31] !log testing x1 backups @ cumin2003 T427897 [09:26:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:26:35] T427897: Upgrade Cumin hosts to Trixie - https://phabricator.wikimedia.org/T427897 [09:26:42] (03Abandoned) 10Marostegui: db-production.php: Disable writes on es6 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303288 (https://phabricator.wikimedia.org/T429118) (owner: 10Marostegui) [09:27:43] !log cwilliams@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db2203.codfw.wmnet with reason: host reimage [09:28:00] (03PS1) 10Dreamy Jazz: hCaptcha: Remove config for VE and DT enable [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303356 (https://phabricator.wikimedia.org/T428883) [09:28:02] (03PS1) 10Gkyziridis: ml-services: Deploy Qwen3.6-27B-FP8 model in experimental ns. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303357 (https://phabricator.wikimedia.org/T425680) [09:29:14] !log marostegui@cumin1003 dbctl commit (dc=all): 'Set es6 eqiad as read-only for maintenance - T429436', diff saved to https://phabricator.wikimedia.org/P94222 and previous config saved to /var/cache/conftool/dbconfig/20260617-092913-marostegui.json [09:29:19] T429436: Switchover es6 master (es1038 -> es1037) - https://phabricator.wikimedia.org/T429436 [09:29:29] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm [09:29:31] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: Primary switchover es6 T429436 [09:29:35] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303356 (https://phabricator.wikimedia.org/T428883) (owner: 10Dreamy Jazz) [09:29:36] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/DiscussionTools] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303354 (https://phabricator.wikimedia.org/T428883) (owner: 10Dreamy Jazz) [09:29:41] !log marostegui@cumin1003 dbctl commit (dc=all): 'Set es1037 with weight 0 T429436', diff saved to https://phabricator.wikimedia.org/P94223 and previous config saved to /var/cache/conftool/dbconfig/20260617-092940-marostegui.json [09:29:43] 10ops-eqiad, 06SRE, 06DC-Ops, 06Wikidata Platform Team, and 2 others: Q4:rack/setup/install dse-k8s-wdqs100[1-3] (formerly wdqs103[6-8]) - https://phabricator.wikimedia.org/T423314#12028337 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host dse-k8s-wdqs100... [09:31:04] (03Merged) 10jenkins-bot: hCaptcha: Remove config for VE and DT enable [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303356 (https://phabricator.wikimedia.org/T428883) (owner: 10Dreamy Jazz) [09:31:31] (03Merged) 10jenkins-bot: Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits [extensions/DiscussionTools] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303354 (https://phabricator.wikimedia.org/T428883) (owner: 10Dreamy Jazz) [09:31:59] (03CR) 10Marostegui: [C:03+2] mariadb: Promote es1037 to es6 master [puppet] - 10https://gerrit.wikimedia.org/r/1303286 (https://phabricator.wikimedia.org/T429436) (owner: 10Gerrit maintenance bot) [09:32:04] !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1303356|hCaptcha: Remove config for VE and DT enable (T428883)]], [[gerrit:1303354|Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits (T428883)]] [09:32:10] T428883: hCaptcha: Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits - https://phabricator.wikimedia.org/T428883 [09:32:17] !log Starting es6 eqiad failover from es1038 to es1037 - T429436 [09:32:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:32:46] (03CR) 10Gmodena: [C:03+1] Added DNS entries for the new WDQS 2 deployments in DSE K8s. (032 comments) [dns] - 10https://gerrit.wikimedia.org/r/1301301 (https://phabricator.wikimedia.org/T428925) (owner: 10Trueg) [09:33:11] !log marostegui@cumin1003 dbctl commit (dc=all): 'Promote es1037 to es6 primary T429436', diff saved to https://phabricator.wikimedia.org/P94224 and previous config saved to /var/cache/conftool/dbconfig/20260617-093310-marostegui.json [09:33:49] (03Abandoned) 10Marostegui: wmnet: Update es6-master alias [dns] - 10https://gerrit.wikimedia.org/r/1303287 (https://phabricator.wikimedia.org/T429436) (owner: 10Gerrit maintenance bot) [09:34:29] (03PS1) 10Marostegui: wmnet: Update es6-master alias [dns] - 10https://gerrit.wikimedia.org/r/1303359 (https://phabricator.wikimedia.org/T429436) [09:34:31] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2203.codfw.wmnet with reason: host reimage [09:34:32] (03PS2) 10Gkyziridis: ml-services: Deploy Qwen3.6-27B-FP8 model in experimental ns. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303357 (https://phabricator.wikimedia.org/T425680) [09:34:48] (03PS1) 10FNegri: aptrepo: move wikireplicas-utils to trixie [puppet] - 10https://gerrit.wikimedia.org/r/1303360 (https://phabricator.wikimedia.org/T351637) [09:35:13] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T428832#12028369 (10Jclark-ctr) I will look again in an hour when I get on site [09:35:15] !log marostegui@cumin1003 dbctl commit (dc=all): 'Depool es1038 T429436', diff saved to https://phabricator.wikimedia.org/P94225 and previous config saved to /var/cache/conftool/dbconfig/20260617-093513-marostegui.json [09:35:19] T429436: Switchover es6 master (es1038 -> es1037) - https://phabricator.wikimedia.org/T429436 [09:35:30] (03CR) 10Marostegui: [C:03+2] wmnet: Update es6-master alias [dns] - 10https://gerrit.wikimedia.org/r/1303359 (https://phabricator.wikimedia.org/T429436) (owner: 10Marostegui) [09:35:31] (03PS2) 10FNegri: aptrepo: move wikireplicas-utils to trixie [puppet] - 10https://gerrit.wikimedia.org/r/1303360 (https://phabricator.wikimedia.org/T351637) [09:35:34] !log marostegui@dns1004 START - running authdns-update [09:36:00] !log marostegui@cumin1003 dbctl commit (dc=all): 'Set es6 eqiad back to read-write - T429436', diff saved to https://phabricator.wikimedia.org/P94226 and previous config saved to /var/cache/conftool/dbconfig/20260617-093559-marostegui.json [09:36:22] (03CR) 10Filippo Giunchedi: [C:03+1] aptrepo: move wikireplicas-utils to trixie [puppet] - 10https://gerrit.wikimedia.org/r/1303360 (https://phabricator.wikimedia.org/T351637) (owner: 10FNegri) [09:36:55] (03CR) 10FNegri: [C:03+2] aptrepo: move wikireplicas-utils to trixie [puppet] - 10https://gerrit.wikimedia.org/r/1303360 (https://phabricator.wikimedia.org/T351637) (owner: 10FNegri) [09:37:19] !log marostegui@dns1004 END - running authdns-update [09:37:43] !log marostegui@cumin1003 START - Cookbook sre.mysql.major-upgrade [09:37:52] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool es1038: Upgrading es1038.eqiad.wmnet [09:38:02] !log dreamyjazz@deploy1003 dreamyjazz: Backport for [[gerrit:1303356|hCaptcha: Remove config for VE and DT enable (T428883)]], [[gerrit:1303354|Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits (T428883)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:38:02] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1038: Upgrading es1038.eqiad.wmnet [09:38:05] T428883: hCaptcha: Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits - https://phabricator.wikimedia.org/T428883 [09:38:53] !log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host es1038.eqiad.wmnet with OS trixie [09:39:17] (03PS1) 10FNegri: aptrepo: move wikireplicas-utils to trixie, part 2 [puppet] - 10https://gerrit.wikimedia.org/r/1303361 (https://phabricator.wikimedia.org/T351637) [09:39:32] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm [09:39:45] 10ops-eqiad, 06SRE, 06DC-Ops, 06Wikidata Platform Team, and 2 others: Q4:rack/setup/install dse-k8s-wdqs100[1-3] (formerly wdqs103[6-8]) - https://phabricator.wikimedia.org/T423314#12028413 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host dse-k8s-wdqs1002.eq... [09:41:04] !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment [09:45:59] (03CR) 10Clément Goubert: [C:03+1] test_main: Remove redundant wildcards. [puppet] - 10https://gerrit.wikimedia.org/r/1303353 (https://phabricator.wikimedia.org/T428772) (owner: 10Blake) [09:47:35] (03PS1) 10Sergio Gimeno: migrateMentorStatusAway: Return SIMULATED for all dry-run executions [extensions/GrowthExperiments] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303364 (https://phabricator.wikimedia.org/T409170) [09:47:35] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:47:37] !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1303356|hCaptcha: Remove config for VE and DT enable (T428883)]], [[gerrit:1303354|Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits (T428883)]] (duration: 15m 32s) [09:47:42] T428883: hCaptcha: Drop $wgDiscussionToolsHCaptchaRequiredForAllEdits - https://phabricator.wikimedia.org/T428883 [09:47:52] (03PS1) 10Sergio Gimeno: migrateMentorStatusAway: Return SIMULATED for all dry-run executions [extensions/GrowthExperiments] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303365 (https://phabricator.wikimedia.org/T409170) [09:48:15] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, June 17 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.6) - 10https://gerrit.wikimedia.org/r/1303365 (https://phabricator.wikimedia.org/T409170) (owner: 10Sergio Gimeno) [09:48:24] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm [09:48:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, June 17 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.7) - 10https://gerrit.wikimedia.org/r/1303364 (https://phabricator.wikimedia.org/T409170) (owner: 10Sergio Gimeno) [09:48:41] 10ops-eqiad, 06SRE, 06DC-Ops, 06Wikidata Platform Team, and 2 others: Q4:rack/setup/install dse-k8s-wdqs100[1-3] (formerly wdqs103[6-8]) - https://phabricator.wikimedia.org/T423314#12028451 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host dse-k8s-wdqs100... [09:50:48] (03PS1) 10Jelto: helm: remove helm311 package and make helm317 default [puppet] - 10https://gerrit.wikimedia.org/r/1303367 (https://phabricator.wikimedia.org/T341984) [09:50:53] (03PS1) 10Jelto: helm: install helm317 and helm319 in parallel [puppet] - 10https://gerrit.wikimedia.org/r/1303368 (https://phabricator.wikimedia.org/T341984) [09:51:50] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2045: repool after maintenance es2045 [09:51:55] (03PS1) 10JavierMonton: stream: mw-page-html-content-change-enrich-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1303369 (https://phabricator.wikimedia.org/T429456) [09:52:08] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2203.codfw.wmnet with OS trixie [09:52:15] (03CR) 10Jelto: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1303367 (https://phabricator.wikimedia.org/T341984) (owner: 10Jelto) [09:52:20] (03CR) 10Jelto: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1303368 (https://phabricator.wikimedia.org/T341984) (owner: 10Jelto) [09:52:21]