[00:00:25] The good old days when I didn't have +2 in that repo, so it was impossible to ever make that mistake. [00:00:30] Ha. [00:00:47] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2011.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [00:00:47] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2008.codfw.wmnet, wdqs2010.codfw.wmnet, wdqs2011.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [00:06:47] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:07:47] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:10:47] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2013.codfw.wmnet, wdqs2015.codfw.wmnet, wdqs2007.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [00:11:47] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:19:16] (03PS1) 10Jforrester: Provide abstractwiki-rust, using Trixie-backports [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1289012 (https://phabricator.wikimedia.org/T425340) [00:20:17] (03PS2) 10Jforrester: Provide abstractwiki-rust, using Trixie-backports [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1289012 (https://phabricator.wikimedia.org/T425340) [00:20:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289005 (owner: 10Jforrester) [00:20:43] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289006 (owner: 10Jforrester) [00:20:43] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289007 (owner: 10Jforrester) [00:22:29] (03CR) 10Jdlrobson: [C:03+1] ThumbLimits: Harmonize svwiki large size with the rest of wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289008 (https://phabricator.wikimedia.org/T376152) (owner: 10Ladsgroup) [00:23:06] (03Merged) 10jenkins-bot: IS: Drop wgGraphDefaultVegaVer, never used any more [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289005 (owner: 10Jforrester) [00:23:08] (03Merged) 10jenkins-bot: IS: Drop wgEnableSpecialMute, ignored since MW 1.46 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289006 (owner: 10Jforrester) [00:23:11] (03Merged) 10jenkins-bot: IS: Drop wgDiscussionTools_visualenhancements_*, ignored since 2025 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289007 (owner: 10Jforrester) [00:23:28] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1289005|IS: Drop wgGraphDefaultVegaVer, never used any more]], [[gerrit:1289006|IS: Drop wgEnableSpecialMute, ignored since MW 1.46]], [[gerrit:1289007|IS: Drop wgDiscussionTools_visualenhancements_*, ignored since 2025]] [00:24:10] !log dzahn@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on gitlab2002.wikimedia.org with reason: T426563 [00:25:12] !log ladsgroup@deploy1003 ladsgroup, jforrester: Backport for [[gerrit:1289005|IS: Drop wgGraphDefaultVegaVer, never used any more]], [[gerrit:1289006|IS: Drop wgEnableSpecialMute, ignored since MW 1.46]], [[gerrit:1289007|IS: Drop wgDiscussionTools_visualenhancements_*, ignored since 2025]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [00:25:39] !log dzahn@cumin2002 START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org [00:26:24] !log ladsgroup@deploy1003 ladsgroup, jforrester: Continuing with deployment [00:30:28] Whee. [00:30:35] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1289005|IS: Drop wgGraphDefaultVegaVer, never used any more]], [[gerrit:1289006|IS: Drop wgEnableSpecialMute, ignored since MW 1.46]], [[gerrit:1289007|IS: Drop wgDiscussionTools_visualenhancements_*, ignored since 2025]] (duration: 07m 08s) [00:32:05] !log dzahn@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org [00:40:09] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289008 (https://phabricator.wikimedia.org/T376152) (owner: 10Ladsgroup) [00:40:36] (03PS1) 10Dzahn: tcpircbot (logmsgbot): replace deploy2002 with deploy2003 [puppet] - 10https://gerrit.wikimedia.org/r/1289019 (https://phabricator.wikimedia.org/T426222) [00:42:37] (03CR) 10CI reject: [V:04-1] ThumbLimits: Harmonize svwiki large size with the rest of wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289008 (https://phabricator.wikimedia.org/T376152) (owner: 10Ladsgroup) [00:49:13] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289008 (https://phabricator.wikimedia.org/T376152) (owner: 10Ladsgroup) [00:51:56] (03Merged) 10jenkins-bot: ThumbLimits: Harmonize svwiki large size with the rest of wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289008 (https://phabricator.wikimedia.org/T376152) (owner: 10Ladsgroup) [00:52:11] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1289008|ThumbLimits: Harmonize svwiki large size with the rest of wikis (T376152)]] [00:52:15] T376152: Evaluate feasibility of deprecating (or limiting) user media size preferences - https://phabricator.wikimedia.org/T376152 [00:54:12] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1289008|ThumbLimits: Harmonize svwiki large size with the rest of wikis (T376152)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [00:54:35] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [00:55:24] (03PS5) 10Aleksandar Mastilovic: Presto memory tuning, resource groups [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) [00:56:23] (03CR) 10Aleksandar Mastilovic: Presto memory tuning, resource groups (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) (owner: 10Aleksandar Mastilovic) [00:56:30] (03CR) 10Aleksandar Mastilovic: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) (owner: 10Aleksandar Mastilovic) [00:57:20] (03CR) 10CI reject: [V:04-1] Presto memory tuning, resource groups [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) (owner: 10Aleksandar Mastilovic) [00:58:47] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1289008|ThumbLimits: Harmonize svwiki large size with the rest of wikis (T376152)]] (duration: 06m 36s) [00:58:51] T376152: Evaluate feasibility of deprecating (or limiting) user media size preferences - https://phabricator.wikimedia.org/T376152 [00:59:11] (03PS6) 10Aleksandar Mastilovic: Presto memory tuning, resource groups [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) [01:00:19] (03CR) 10Aleksandar Mastilovic: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1285926 (https://phabricator.wikimedia.org/T424112) (owner: 10Aleksandar Mastilovic) [01:05:08] (03PS1) 10Jasmine: k8s: add wikikube-worker2331 [puppet] - 10https://gerrit.wikimedia.org/r/1289022 (https://phabricator.wikimedia.org/T426688) [01:09:12] (03PS1) 10TrainBranchBot: Branch commit for wmf/1.47.0-wmf.3 [core] (wmf/1.47.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1289023 (https://phabricator.wikimedia.org/T423912) [01:09:14] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/1.47.0-wmf.3 [core] (wmf/1.47.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1289023 (https://phabricator.wikimedia.org/T423912) (owner: 10TrainBranchBot) [01:09:29] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1289024 [01:09:29] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1289024 (owner: 10TrainBranchBot) [01:19:12] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:21:30] (03Merged) 10jenkins-bot: Branch commit for wmf/1.47.0-wmf.3 [core] (wmf/1.47.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1289023 (https://phabricator.wikimedia.org/T423912) (owner: 10TrainBranchBot) [01:21:37] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1289024 (owner: 10TrainBranchBot) [01:24:12] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:31:18] (03PS1) 10DDesouza: miscweb(design-landing-page): bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1289031 (https://phabricator.wikimedia.org/T344471) [01:34:54] (03CR) 10DDesouza: [C:03+2] miscweb(design-landing-page): bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1289031 (https://phabricator.wikimedia.org/T344471) (owner: 10DDesouza) [01:37:23] (03Merged) 10jenkins-bot: miscweb(design-landing-page): bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1289031 (https://phabricator.wikimedia.org/T344471) (owner: 10DDesouza) [02:00:04] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260519T0200) [02:00:09] !log dani@deploy1003 helmfile [staging] START helmfile.d/services/miscweb: apply [02:00:22] !log dani@deploy1003 helmfile [staging] DONE helmfile.d/services/miscweb: apply [02:00:24] !log dani@deploy1003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [02:00:36] !log dani@deploy1003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [02:00:37] !log dani@deploy1003 helmfile [codfw] START helmfile.d/services/miscweb: apply [02:00:52] !log dani@deploy1003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [02:01:23] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:08:02] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 39s) [02:09:12] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:30:10] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:34:12] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:36:32] (03CR) 10RLazarus: "Building and testing locally, this doesn't have the right version:" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1289012 (https://phabricator.wikimedia.org/T425340) (owner: 10Jforrester) [02:46:25] FIRING: [42x] SystemdUnitFailed: cfssl-ocsprefresh-Wikimedia_Internal_Root_CA.service on pki1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:49:12] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:50:09] (03CR) 10Scott French: [C:03+1] "Thanks, Jasmine!" [puppet] - 10https://gerrit.wikimedia.org/r/1289022 (https://phabricator.wikimedia.org/T426688) (owner: 10Jasmine) [02:50:27] RESOLVED: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:00:04] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260519T0300) [03:01:52] (03PS1) 10TrainBranchBot: testwikis to 1.47.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289054 (https://phabricator.wikimedia.org/T423912) [03:01:55] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by mwpresync@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289054 (https://phabricator.wikimedia.org/T423912) (owner: 10TrainBranchBot) [03:02:51] (03Merged) 10jenkins-bot: testwikis to 1.47.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1289054 (https://phabricator.wikimedia.org/T423912) (owner: 10TrainBranchBot) [03:03:18] !log mwpresync@deploy1003 Started scap sync-world: testwikis to 1.47.0-wmf.3 refs T423912 [03:03:22] T423912: 1.47.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T423912 [03:09:07] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, May 19 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1287433 (https://phabricator.wikimedia.org/T355445) (owner: 10Codename Noreste) [03:10:38] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, May 19 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1281901 (https://phabricator.wikimedia.org/T424413) (owner: 10Codename Noreste) [03:17:47] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2013.codfw.wmnet, wdqs2015.codfw.wmnet, wdqs2022.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:19:49] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2013.codfw.wmnet, wdqs2021.codfw.wmnet, wdqs2015.codfw.wmnet, wdqs2014.codfw.wmnet, wdqs2007.codfw.wmnet, wdqs2008.codfw.wmnet, wdqs2010.codfw.wmnet, wdqs2012.codfw.wmnet, wdqs2011.codfw.wmnet, wdqs2022.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:20:47] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:21:47] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:25:47] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:25:47] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:25:47] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:25:47] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:26:47] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:26:47] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:28:47] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2021.codfw.wmnet, wdqs2007.codfw.wmnet, wdqs2014.codfw.wmnet, wdqs2008.codfw.wmnet, wdqs2011.codfw.wmnet, wdqs2022.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:28:47] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2015.codfw.wmnet, wdqs2007.codfw.wmnet, wdqs2022.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:29:49] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:29:49] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:33:41] PROBLEM - MariaDB Replica Lag: m2 on db2160 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 639.14 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [03:34:41] RECOVERY - MariaDB Replica Lag: m2 on db2160 is OK: OK slave_sql_lag Replication lag: 0.33 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [03:34:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx errors from swift.discovery.wmnet in eqsin #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqsin&var-cluster=upload&var-origin=swift.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [03:39:48] looking [03:40:06] FIRING: [6x] SLOBudgetBurn: Search update lag is below 95% target in codfw - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [03:41:40] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service on wdqs1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:41:42] !log mwpresync@deploy1003 Finished scap sync-world: testwikis to 1.47.0-wmf.3 refs T423912 (duration: 38m 23s) [03:41:46] T423912: 1.47.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T423912 [03:43:47] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs2015.codfw.wmnet, wdqs2022.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:46:49] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:54:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx errors from swift.discovery.wmnet in eqsin #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqsin&var-cluster=upload&var-origin=swift.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [04:00:05] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260519T0400)