[00:09:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [00:10:16] (03PS1) 10Ryan Kemper: move-vlan: only query target host for inplace [cookbooks] - 10https://gerrit.wikimedia.org/r/1324839 (https://phabricator.wikimedia.org/T434737) [00:14:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [00:14:44] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:16:07] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host an-worker1178 [00:16:07] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1178 [00:17:48] (03CR) 10Ryan Kemper: "tested the no-op path on an-worker1178 and it works. we can test the other codepath tomorrow" [cookbooks] - 10https://gerrit.wikimedia.org/r/1324839 (https://phabricator.wikimedia.org/T434737) (owner: 10Ryan Kemper) [00:19:32] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host an-worker1142.eqiad.wmnet with OS bookworm [00:34:32] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1142.eqiad.wmnet with reason: host reimage [00:38:40] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1142.eqiad.wmnet with reason: host reimage [00:48:52] 06SRE, 06serviceops-deprecated, 10TimedMediaHandler: Upgrade Wikimedia production's ffmpeg to 4.4 or later so we can use the fpsmax flag - https://phabricator.wikimedia.org/T318419#12210300 (10Ladsgroup) The version of ffmpeg we now run in production is `5.1.9-0+deb12u1`. I pulled the container based on depl... [00:57:34] (03PS2) 10Lerickson: Enable egress via urldownloader for WDQS V2. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324809 (https://phabricator.wikimedia.org/T434321) [01:03:02] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1142.eqiad.wmnet with OS bookworm [01:12:21] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1324854 [01:12:21] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1324854 (owner: 10TrainBranchBot) [01:14:45] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [01:23:41] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1324854 (owner: 10TrainBranchBot) [01:46:18] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host an-worker1219.eqiad.wmnet with OS bookworm [02:00:53] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1219.eqiad.wmnet with reason: host reimage [02:00:56] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:02:55] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:03:54] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1219.eqiad.wmnet with reason: host reimage [02:04:45] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:07:43] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 46s) [02:14:33] FIRING: KubernetesAPINotScrapable: k8s-dse@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [02:15:25] FIRING: [2x] GanetiBGPDown: BGP session down between ganeti3005 and asw1-by27-esams - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPDown - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPDown [02:18:34] !log T434494 `ryankemper@deploy1003:~$ echo 'https://stats.wikimedia.org/' | mwscript-k8s --attach -- purgeList.php` (default page got cached during yesterday's `an-web1001` reimage) [02:18:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:18:38] T434494: Migrate production hadoop cluster to bookworm - https://phabricator.wikimedia.org/T434494 [02:19:33] RESOLVED: KubernetesAPINotScrapable: k8s-dse@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [02:23:46] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1219.eqiad.wmnet with OS bookworm [02:29:34] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host an-worker1151 [02:29:34] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1151 [02:39:13] FIRING: KubernetesCalicoDown: kubestagemaster2005.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s-staging&var-instance=kubestagemaster2005.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:42:16] (03CR) 10NguoiDungKhongDinhDanh: "I don't use IRC. Is being on IRC an absolute requirement?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311141 (https://phabricator.wikimedia.org/T432311) (owner: 10NguoiDungKhongDinhDanh) [02:54:06] FIRING: [6x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:49:02] (03PS1) 10Tim Starling: Add Produnto to extension-list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324963 (https://phabricator.wikimedia.org/T421436) [03:49:02] (03CR) 10Tim Starling: "Safe to go whenever per T411516, no need to wait for https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/258 and train." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324963 (https://phabricator.wikimedia.org/T421436) (owner: 10Tim Starling) [03:50:36] (03PS1) 10Tim Starling: Enable Produnto on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324964 (https://phabricator.wikimedia.org/T421436) [04:06:46] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [04:06:59] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12210529 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS t... [04:46:40] FIRING: KubernetesAPINotScrapable: k8s-aux@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [04:51:40] RESOLVED: KubernetesAPINotScrapable: k8s-aux@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [05:14:45] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [05:27:02] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie [05:27:16] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12210572 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS trixi... [05:37:06] (03PS2) 10Ryan Kemper: move-vlan: only query target host for inplace [cookbooks] - 10https://gerrit.wikimedia.org/r/1324839 (https://phabricator.wikimedia.org/T434737) [05:40:08] (03CR) 10Arnaudb: [C:03+2] gitlab: point gitlab-replica-a at the CDN [dns] - 10https://gerrit.wikimedia.org/r/1324280 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [05:40:24] !log arnaudb@dns1006 START - running authdns-update [05:42:23] !log arnaudb@dns1006 END - running authdns-update [05:49:32] (03CR) 10Trueg: [C:03+1] "nice" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324809 (https://phabricator.wikimedia.org/T434321) (owner: 10Lerickson) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T0600) [06:00:05] marostegui, Amir1, and federico3: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Primary database switchover deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T0600). [06:04:45] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:08:33] FIRING: KubernetesAPINotScrapable: k8s-dse@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [06:09:45] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:13:33] RESOLVED: KubernetesAPINotScrapable: k8s-dse@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [06:15:25] FIRING: [2x] GanetiBGPDown: BGP session down between ganeti3005 and asw1-by27-esams - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPDown - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPDown [06:36:02] (03PS1) 10Trueg: WDQS: Set default query timeout to 300s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325235 (https://phabricator.wikimedia.org/T433391) [06:39:03] 06SRE, 06Data-Engineering: WE 5.4 FY 25/26: Improve automata detection at the edge and pass it to the refinery pipeline - https://phabricator.wikimedia.org/T396562#12210640 (10Joe) 05Open→03Resolved [06:39:13] FIRING: KubernetesCalicoDown: kubestagemaster2005.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s-staging&var-instance=kubestagemaster2005.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:41:45] (03CR) 10Gehel: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1324763 (https://phabricator.wikimedia.org/T434685) (owner: 10Bking) [06:51:02] 06SRE, 06Data-Engineering, 06Product-Analytics, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Add cloudflare_radar API connection to Airflow analytics main instance - https://phabricator.wikimedia.org/T433982#12210657 (10Gehel) [06:51:34] 06SRE, 06Data-Engineering, 06Product-Analytics, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Add cloudflare_radar API connection to Airflow analytics main instance - https://phabricator.wikimedia.org/T433982#12210660 (10Gehel) Tagging #data-engineering for visibility. [06:54:06] FIRING: [6x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [06:55:12] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [07:00:05] Amir1, urbanecm, and awight: OwO what's this, a deployment window?? UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T0700). nyaa~ [07:00:05] No Gerrit patches in the queue for this window AFAICS. [07:01:26] 10ops-codfw, 06SRE, 06DC-Ops: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681#12210681 (10MoritzMuehlenhoff) >>! In T434681#12210194, @Jhancock.wm wrote: > What was the reboot for? If there was a firmware upgrade, it might have a conflict somewhere. a quick internet sea... [07:12:00] 10SRE-swift-storage, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): RdfStreamingUpdaterSpaceUsageTooHigh - https://phabricator.wikimedia.org/T431506#12210713 (10Gehel) [07:13:27] 10SRE-swift-storage, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): RdfStreamingUpdaterSpaceUsageTooHigh - https://phabricator.wikimedia.org/T431506#12210715 (10Gehel) [07:16:03] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Create Kerberos identity for Randall Scout - https://phabricator.wikimedia.org/T430598#12210719 (10Gehel) [07:19:25] !log jayme@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on kubestagemaster2005.codfw.wmnet with reason: downtime because of hardware failure and no DRBD [07:19:30] 10ops-codfw, 06SRE, 06DC-Ops: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681#12210728 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=43af85a6-59b3-4dce-93f8-886c8bd5c61d) set by jayme@cumin1003 for 7 days, 0:00:00 on 1 host(s) and their services wit... [07:24:51] !log marostegui@cumin1003 START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis testwiki in section s3 [07:25:04] (03PS1) 10Ammarpad: admin: Add new SSH public key for Ammarpad [puppet] - 10https://gerrit.wikimedia.org/r/1325272 [07:28:01] (03CR) 10Ammarpad: "I have uploaded a new public key at I3fd8c46792c29531fb749806ac0b67e8c7b66dac" [puppet] - 10https://gerrit.wikimedia.org/r/1318214 (owner: 10Ammarpad) [07:28:56] (03CR) 10Filippo Giunchedi: "I would personally be ok without the "in ...seconds" and only timestamp, though just a matter of preference. Thank you for taking this on!" [software/pywmflib] - 10https://gerrit.wikimedia.org/r/1320151 (https://phabricator.wikimedia.org/T433698) (owner: 10Mahveotm) [07:30:58] (03CR) 10Klausman: [C:03+1] move-vlan: only query target host for inplace [cookbooks] - 10https://gerrit.wikimedia.org/r/1324839 (https://phabricator.wikimedia.org/T434737) (owner: 10Ryan Kemper) [07:31:23] !log marostegui@cumin1003 END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis testwiki in section s3 [07:31:25] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681#12210770 (10JMeybohm) [07:32:23] (03PS1) 10Fabfur: cache::proxy: remove avalanche parameter from sdbm function [puppet] - 10https://gerrit.wikimedia.org/r/1325279 (https://phabricator.wikimedia.org/T426379) [07:32:34] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-private-users for aramirezwmf - https://phabricator.wikimedia.org/T433999#12210779 (10jcrespo) @ARamirez_WMF I've added you to the wmf ldap group. Could you try again logging in into superset.wikimedia.org ? It should work now. [07:33:22] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): eqiad row A&B host migration details request for Data Platform - https://phabricator.wikimedia.org/T432645#12210785 (10Gehel) [07:34:26] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): eqiad row A&B host migration details request for Data Platform - https://phabricator.wikimedia.org/T432645#12210789 (10Gehel) @RobH : this seems to be completed on our side. I'll keep this task open and let you close it when it ma... [07:34:44] (03CR) 10JavierMonton: [C:03+2] topic: webrequest-page-view [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321976 (https://phabricator.wikimedia.org/T433962) (owner: 10JavierMonton) [07:35:06] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 06Machine-Learning-Team (Q1 FY2026-27): eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12210790 (10Gehel) [07:35:22] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 06Machine-Learning-Team (Q1 FY2026-27): eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12210792 (10Gehel) @RobH : this seems to be completed on our side. I'll keep t... [07:38:00] (03CR) 10Fabfur: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1325279 (https://phabricator.wikimedia.org/T426379) (owner: 10Fabfur) [07:38:21] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1273.eqiad.wmnet with reason: Reboot [07:39:53] 06SRE, 13Patch-For-Review: sre.hosts.move-vlan crashes during init for hosts absent from PuppetDB - https://phabricator.wikimedia.org/T434737#12210818 (10Aklapper) [07:45:33] (03Merged) 10jenkins-bot: topic: webrequest-page-view [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321976 (https://phabricator.wikimedia.org/T433962) (owner: 10JavierMonton) [07:46:01] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Create Kerberos identity for Randall Scout - https://phabricator.wikimedia.org/T430598#12210852 (10Rsilvola) Approved as Randall’s manager [07:49:06] (03PS1) 10Marostegui: db1273: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1325285 [07:51:32] (03CR) 10Marostegui: [C:03+2] db1273: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1325285 (owner: 10Marostegui) [07:52:46] (03PS1) 10Marostegui: instances.yaml: Add db1273 [puppet] - 10https://gerrit.wikimedia.org/r/1325286 (https://phabricator.wikimedia.org/T407942) [07:53:36] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12210883 (10fgiunchedi) >>! In T431682#12209331, @VRiley-WMF wrote: > Hey @fgiunchedi Yes, that would work for me! Would monday next week work for you? Monday so... [07:54:33] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1273 [puppet] - 10https://gerrit.wikimedia.org/r/1325286 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [07:55:12] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [07:56:12] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1273 to dbctl T434288', diff saved to https://phabricator.wikimedia.org/P96035 and previous config saved to /var/cache/conftool/dbconfig/20260813-075611-marostegui.json [07:56:16] T434288: Switchover ms1, ms2 and ms3 master - https://phabricator.wikimedia.org/T434288 [07:56:34] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1273: Pool in s7 [07:59:00] (03PS1) 10Marostegui: mariadb: Productionize db1275 [puppet] - 10https://gerrit.wikimedia.org/r/1325293 (https://phabricator.wikimedia.org/T407942) [07:59:21] (03PS1) 10Santiago Faci: Deploy GrowthBook 5.0.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325294 (https://phabricator.wikimedia.org/T434098) [08:00:04] brennen and jnuche: gettimeofday() says it's time for MediaWiki train - Utc-7+Utc-0 Version (secondary timeslot). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T0800) [08:01:05] (03PS2) 10Marostegui: mariadb: Productionize db1275 [puppet] - 10https://gerrit.wikimedia.org/r/1325293 (https://phabricator.wikimedia.org/T407942) [08:01:43] (03PS3) 10Marostegui: mariadb: Productionize db1275 [puppet] - 10https://gerrit.wikimedia.org/r/1325293 (https://phabricator.wikimedia.org/T407942) [08:02:59] (03PS4) 10Marostegui: mariadb: Productionize db1275 [puppet] - 10https://gerrit.wikimedia.org/r/1325293 (https://phabricator.wikimedia.org/T407942) [08:05:11] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1275 [puppet] - 10https://gerrit.wikimedia.org/r/1325293 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [08:07:23] (03PS1) 10Dreamy Jazz: Improve performance of DB query in BackfillAbuseReview.php [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325407 (https://phabricator.wikimedia.org/T434688) [08:07:55] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:08:13] (03PS1) 10Dreamy Jazz: Improve performance of DB query in BackfillAbuseReview.php [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1325408 (https://phabricator.wikimedia.org/T434688) [08:09:41] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Create Kerberos identity for Randall Scout - https://phabricator.wikimedia.org/T430598#12210917 (10jcrespo) @Rscout we ask kindly to use [[ https://wikitech.wikimedia.org/wiki/SRE/Pr... [08:09:52] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:09:55] (03CR) 10Ammarpad: "I have the public key on my home dir at ~/key_verify.txt" [puppet] - 10https://gerrit.wikimedia.org/r/1325272 (owner: 10Ammarpad) [08:10:47] (03PS1) 10Marostegui: site.pp: Fix host [puppet] - 10https://gerrit.wikimedia.org/r/1325414 [08:11:03] (03CR) 10Marostegui: [V:03+2 C:03+2] site.pp: Fix host [puppet] - 10https://gerrit.wikimedia.org/r/1325414 (owner: 10Marostegui) [08:15:01] !log roll-restart apus frontends in eqiad [08:15:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:15:30] !log mvernon@cumin1003 START - Cookbook sre.hosts.reboot-cluster [08:20:15] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1161.eqiad.wmnet onto db1275.eqiad.wmnet [08:20:19] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1161: Depool db1161.eqiad.wmnet to then clone it to db1275.eqiad.wmnet - marostegui@cumin1003 [08:20:34] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1161: Depool db1161.eqiad.wmnet to then clone it to db1275.eqiad.wmnet - marostegui@cumin1003 [08:22:13] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host lists2001.wikimedia.org [08:22:50] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1151.eqiad.wmnet with OS bookworm [08:23:18] !log btullis@cumin1003 START - Cookbook sre.hosts.move-vlan for host an-worker1151 [08:23:32] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host aphlict2001.codfw.wmnet [08:23:33] FIRING: KubernetesAPINotScrapable: k8s-dse@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [08:23:36] PROBLEM - MariaDB Replica IO: s5 on db1154 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db1161.eqiad.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db1161.eqiad.wmnet (111 Connection refused) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [08:23:58] (03PS1) 10Hashar: Revert "REST: Enable `GET /lexemes/{lexeme_id}` by default" [extensions/WikibaseLexeme] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325416 (https://phabricator.wikimedia.org/T434712) [08:26:02] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org [08:26:21] btullis@cumin1003 reimage (PID 3315000) is awaiting input [08:27:31] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict2001.codfw.wmnet [08:27:53] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host aphlict1002.eqiad.wmnet [08:28:22] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gitlab2003.wikimedia.org [08:28:33] RESOLVED: KubernetesAPINotScrapable: k8s-dse@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [08:28:42] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists2001.wikimedia.org [08:29:06] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org [08:29:57] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host planet1003.eqiad.wmnet [08:30:11] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host planet2003.codfw.wmnet [08:31:26] PROBLEM - MariaDB Replica Lag: s5 on clouddb1029 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 600.95 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [08:31:36] PROBLEM - MariaDB Replica Lag: s5 on db1154 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 611.95 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [08:31:51] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict1002.eqiad.wmnet [08:31:56] PROBLEM - MariaDB Replica Lag: s5 on clouddb1033 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 630.98 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [08:32:10] expected ^ [08:32:22] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org [08:32:24] (03PS5) 10Dreamy Jazz: Register the mediawiki.wikimedia_antiabuse.content_policy_score stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1321579 (https://phabricator.wikimedia.org/T432848) (owner: 10Mpostoronca) [08:32:36] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 22 hosts with reason: Cloning [08:33:03] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org [08:33:55] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet1003.eqiad.wmnet [08:34:09] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2003.wikimedia.org [08:34:13] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet2003.codfw.wmnet [08:34:22] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org [08:34:57] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) [08:35:12] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host doc1004.eqiad.wmnet [08:35:37] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host etherpad1004.eqiad.wmnet [08:36:53] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org [08:37:19] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host phab2003.codfw.wmnet [08:38:00] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host etherpad2002.codfw.wmnet [08:38:48] (03PS1) 10Gerrit maintenance bot: mariadb: Promote db2229 to s6 master [puppet] - 10https://gerrit.wikimedia.org/r/1325417 (https://phabricator.wikimedia.org/T434754) [08:39:04] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2002.wikimedia.org [08:39:09] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc1004.eqiad.wmnet [08:39:35] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad1004.eqiad.wmnet [08:39:47] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-private-users for aramirezwmf - https://phabricator.wikimedia.org/T433999#12211025 (10jcrespo) a:05jcrespo→03None [08:40:07] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gerrit1003.wikimedia.org [08:40:17] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host doc2003.codfw.wmnet [08:40:36] jouncebot: nowandnext [08:40:36] For the next 1 hour(s) and 19 minute(s): MediaWiki train - Utc-7+Utc-0 Version (secondary timeslot) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T0800) [08:40:37] In 1 hour(s) and 19 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1000) [08:40:50] (03CR) 10Marostegui: [C:03+1] mariadb: Productionize db1277 [puppet] - 10https://gerrit.wikimedia.org/r/1324768 (https://phabricator.wikimedia.org/T407942) (owner: 10CWilliams) [08:40:55] (03PS2) 10CWilliams: mariadb: Productionize db1277 [puppet] - 10https://gerrit.wikimedia.org/r/1324768 (https://phabricator.wikimedia.org/T407942) [08:41:06] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org [08:41:19] I am backporting a patch for Wikibase and will proceed with the train (it is brennen this week but in interest of time we agreed to push it this morning) [poke andre/jnuche for info] [08:41:32] aight [08:41:40] thanks! [08:41:43] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1273: Pool in s7 [08:41:57] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host stewards1001.eqiad.wmnet [08:42:00] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad2002.codfw.wmnet [08:42:24] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host stewards2001.codfw.wmnet [08:42:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by hashar@deploy1003 using scap backport" [extensions/WikibaseLexeme] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325416 (https://phabricator.wikimedia.org/T434712) (owner: 10Hashar) [08:42:55] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:43:05] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab2003.codfw.wmnet [08:43:39] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host vrts1004.eqiad.wmnet [08:44:07] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc2003.codfw.wmnet [08:44:45] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host vrts2002.codfw.wmnet [08:45:20] !log mvernon@cumin1003 START - Cookbook sre.hosts.reboot-cluster [08:45:23] !log roll-restart apus frontends in codfw [08:45:26] (03Merged) 10jenkins-bot: Revert "REST: Enable `GET /lexemes/{lexeme_id}` by default" [extensions/WikibaseLexeme] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325416 (https://phabricator.wikimedia.org/T434712) (owner: 10Hashar) [08:45:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:45:36] RECOVERY - MariaDB Replica IO: s5 on db1154 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [08:45:42] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stewards1001.eqiad.wmnet [08:45:54] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit1003.wikimedia.org [08:46:12] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stewards2001.codfw.wmnet [08:46:26] RECOVERY - MariaDB Replica Lag: s5 on clouddb1029 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [08:46:36] RECOVERY - MariaDB Replica Lag: s5 on db1154 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [08:46:38] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1161: Pool db1161.eqiad.wmnet in after cloning [08:46:56] RECOVERY - MariaDB Replica Lag: s5 on clouddb1033 is OK: OK slave_sql_lag Replication lag: 0.03 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [08:47:02] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gitlab-runner1002.eqiad.wmnet [08:47:12] !log hashar@deploy1003 Started scap sync-world: Backport for [[gerrit:1325416|Revert "REST: Enable `GET /lexemes/{lexeme_id}` by default" (T434712)]] [08:47:16] T434712: Error: Class "Wikibase\Repo\WikibaseRepo" not found - https://phabricator.wikimedia.org/T434712 [08:47:21] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1004.eqiad.wmnet [08:49:27] !log hashar@deploy1003 hashar: Backport for [[gerrit:1325416|Revert "REST: Enable `GET /lexemes/{lexeme_id}` by default" (T434712)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:50:28] !log hashar@deploy1003 hashar: Continuing with deployment [08:51:12] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts2002.codfw.wmnet [08:51:38] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s6 T434754 [08:51:43] T434754: Switchover s6 master (db2214 -> db2229) - https://phabricator.wikimedia.org/T434754 [08:51:52] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Set db2229 with weight 0 T434754', diff saved to https://phabricator.wikimedia.org/P96043 and previous config saved to /var/cache/conftool/dbconfig/20260813-085151-cwilliams.json [08:53:11] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1002.eqiad.wmnet [08:53:33] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gitlab-runner1003.eqiad.wmnet [08:54:34] !log hashar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325416|Revert "REST: Enable `GET /lexemes/{lexeme_id}` by default" (T434712)]] (duration: 07m 22s) [08:54:38] T434712: Error: Class "Wikibase\Repo\WikibaseRepo" not found - https://phabricator.wikimedia.org/T434712 [08:55:18] I'll let some time pass to ensure the cluster is stable then move on and proceed with the train [08:55:46] (03CR) 10CWilliams: [C:03+2] mariadb: Promote db2229 to s6 master [puppet] - 10https://gerrit.wikimedia.org/r/1325417 (https://phabricator.wikimedia.org/T434754) (owner: 10Gerrit maintenance bot) [08:56:24] (03PS6) 10Cparle: Baseline user-facing metrics for thumbnails, with an experiment mechanism [puppet] - 10https://gerrit.wikimedia.org/r/1324778 (https://phabricator.wikimedia.org/T431597) [08:56:39] (03PS1) 10JMeybohm: wikikube: Update codfw to calico 3.30 [puppet] - 10https://gerrit.wikimedia.org/r/1325420 (https://phabricator.wikimedia.org/T427400) [08:57:12] !log Starting s6 codfw failover from db2214 to db2229 - T434754 [08:57:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:57:17] T434754: Switchover s6 master (db2214 -> db2229) - https://phabricator.wikimedia.org/T434754 [08:57:53] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Promote db2229 to s6 primary T434754', diff saved to https://phabricator.wikimedia.org/P96044 and previous config saved to /var/cache/conftool/dbconfig/20260813-085752-cwilliams.json [08:58:23] (03PS1) 10JMeybohm: wikikube-codfw: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325421 (https://phabricator.wikimedia.org/T427400) [08:58:51] (03PS2) 10JMeybohm: wikikube-codfw: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325421 (https://phabricator.wikimedia.org/T427400) [08:59:04] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1325420 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [08:59:41] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1003.eqiad.wmnet [08:59:45] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gitlab-runner1004.eqiad.wmnet [09:00:02] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depool db2214 T434754', diff saved to https://phabricator.wikimedia.org/P96045 and previous config saved to /var/cache/conftool/dbconfig/20260813-090001-cwilliams.json [09:01:24] !log btullis@cumin1003 START - Cookbook sre.dns.netbox [09:01:40] (03PS3) 10CWilliams: mariadb: Productionize db1277 [puppet] - 10https://gerrit.wikimedia.org/r/1324768 (https://phabricator.wikimedia.org/T407942) [09:01:42] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 2 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [09:03:28] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 7 hosts with reason: reboot [09:05:01] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) [09:05:45] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, August 13 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324731 (owner: 10Jgiannelos) [09:05:55] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1004.eqiad.wmnet [09:06:04] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gitlab-runner2002.codfw.wmnet [09:07:11] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host apus-be2004.codfw.wmnet [09:07:38] btullis@cumin1003 reimage (PID 3315000) is awaiting input [09:09:06] !log cwilliams@cumin1003 START - Cookbook sre.mysql.upgrade for 1 hosts [09:09:16] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db2214: Upgrading db2214.codfw.wmnet [09:09:20] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2214: Upgrading db2214.codfw.wmnet [09:11:30] jouncebot: nowandnext [09:11:30] For the next 0 hour(s) and 48 minute(s): MediaWiki train - Utc-7+Utc-0 Version (secondary timeslot) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T0800) [09:11:30] In 0 hour(s) and 48 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1000) [09:11:56] (03CR) 10CWilliams: [C:03+2] mariadb: Productionize db1277 [puppet] - 10https://gerrit.wikimedia.org/r/1324768 (https://phabricator.wikimedia.org/T407942) (owner: 10CWilliams) [09:12:25] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2002.codfw.wmnet [09:12:29] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gitlab-runner2003.codfw.wmnet [09:12:55] FIRING: [2x] JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:13:08] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be2004.codfw.wmnet [09:14:12] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host apus-be2005.codfw.wmnet [09:14:45] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [09:15:33] (03PS1) 10TrainBranchBot: group1 to 1.47.0-wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325422 (https://phabricator.wikimedia.org/T430834) [09:15:36] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by hashar@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325422 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [09:16:07] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2214: Upgrade of db2214.codfw.wmnet completed [09:16:44] (03Merged) 10jenkins-bot: group1 to 1.47.0-wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325422 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [09:18:16] !log jynus@cumin1003 START - Cookbook sre.hosts.remove-downtime for 7 hosts [09:18:20] !log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 7 hosts [09:18:35] (03PS1) 10Bartosz Wójtowicz: ml-services: Update revise-tone-task-generator. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325423 (https://phabricator.wikimedia.org/T433319) [09:18:48] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 7 hosts with reason: reboot [09:19:07] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2003.codfw.wmnet [09:19:12] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gitlab-runner2004.codfw.wmnet [09:19:44] (03CR) 10Majavah: openstack: export compute service info to prometheus (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1313958 (https://phabricator.wikimedia.org/T284747) (owner: 10Filippo Giunchedi) [09:19:48] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be2005.codfw.wmnet [09:19:49] (03CR) 10AikoChou: [C:03+1] ml-services: Update revise-tone-task-generator. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325423 (https://phabricator.wikimedia.org/T433319) (owner: 10Bartosz Wójtowicz) [09:21:16] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host apus-be2006.codfw.wmnet [09:21:41] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: Update revise-tone-task-generator. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325423 (https://phabricator.wikimedia.org/T433319) (owner: 10Bartosz Wójtowicz) [09:22:49] !log hashar@deploy1003 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.15 refs T430834 [09:22:53] T430834: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834 [09:24:14] (03Merged) 10jenkins-bot: ml-services: Update revise-tone-task-generator. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325423 (https://phabricator.wikimedia.org/T433319) (owner: 10Bartosz Wójtowicz) [09:24:57] hashar: Could I use scap to backport changes to a maintenance script? [09:25:11] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2004.codfw.wmnet [09:25:31] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be2006.codfw.wmnet [09:25:45] (03PS1) 10Marostegui: installserver: Do not format db1273 [puppet] - 10https://gerrit.wikimedia.org/r/1325424 [09:25:55] Dreamy_Jazz: yeah sure, looks like there is no errors as wmf.15 got promoted to group 1 [09:25:58] well, so far [09:26:00] Thanks [09:26:31] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1325408 (https://phabricator.wikimedia.org/T434688) (owner: 10Dreamy Jazz) [09:26:32] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325407 (https://phabricator.wikimedia.org/T434688) (owner: 10Dreamy Jazz) [09:27:03] :) [09:27:45] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host moss-be2003.codfw.wmnet [09:27:47] (03Merged) 10jenkins-bot: Improve performance of DB query in BackfillAbuseReview.php [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1325408 (https://phabricator.wikimedia.org/T434688) (owner: 10Dreamy Jazz) [09:27:48] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . [09:28:00] Dreamy_Jazz: for the record, I really like the commit message of that patch. It is pristine clear and I feel I could have +2ed it myself without even looking at the code 8) [09:28:10] :D [09:28:58] (03CR) 10Marostegui: [C:03+2] installserver: Do not format db1273 [puppet] - 10https://gerrit.wikimedia.org/r/1325424 (owner: 10Marostegui) [09:29:45] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . [09:30:03] (03Merged) 10jenkins-bot: Improve performance of DB query in BackfillAbuseReview.php [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325407 (https://phabricator.wikimedia.org/T434688) (owner: 10Dreamy Jazz) [09:30:26] !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1325408|Improve performance of DB query in BackfillAbuseReview.php (T434688)]], [[gerrit:1325407|Improve performance of DB query in BackfillAbuseReview.php (T434688)]] [09:30:31] T434688: Create maintenance script to evaluate revisions performed between timestamps against content policies - https://phabricator.wikimedia.org/T434688 [09:31:51] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1161: Pool db1161.eqiad.wmnet in after cloning [09:31:59] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . [09:32:06] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1161.eqiad.wmnet onto db1275.eqiad.wmnet [09:32:29] !log dreamyjazz@deploy1003 dreamyjazz: Backport for [[gerrit:1325408|Improve performance of DB query in BackfillAbuseReview.php (T434688)]], [[gerrit:1325407|Improve performance of DB query in BackfillAbuseReview.php (T434688)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:33:18] !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment [09:33:55] !log jynus@cumin1003 START - Cookbook sre.hosts.remove-downtime for 7 hosts [09:33:59] !log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 7 hosts [09:35:02] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2003.codfw.wmnet [09:36:37] !log btullis@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1151 - btullis@cumin1003" [09:36:42] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: reboot [09:36:43] !log btullis@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1151 - btullis@cumin1003" [09:36:44] !log btullis@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:36:44] !log btullis@cumin1003 START - Cookbook sre.dns.wipe-cache an-worker1151.eqiad.wmnet 13.36.64.10.in-addr.arpa 3.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [09:36:48] !log btullis@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1151.eqiad.wmnet 13.36.64.10.in-addr.arpa 3.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [09:36:48] !log btullis@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host an-worker1151 [09:37:23] !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325408|Improve performance of DB query in BackfillAbuseReview.php (T434688)]], [[gerrit:1325407|Improve performance of DB query in BackfillAbuseReview.php (T434688)]] (duration: 06m 57s) [09:37:29] T434688: Create maintenance script to evaluate revisions performed between timestamps against content policies - https://phabricator.wikimedia.org/T434688 [09:38:37] (03PS1) 10Marostegui: mariadb: Productionize db1279 [puppet] - 10https://gerrit.wikimedia.org/r/1325425 (https://phabricator.wikimedia.org/T407942) [09:39:35] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1279 [puppet] - 10https://gerrit.wikimedia.org/r/1325425 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [09:40:07] !log btullis@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1151 [09:40:07] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1151 [09:40:56] !log jynus@cumin1003 START - Cookbook sre.hosts.remove-downtime for backupmon1001.eqiad.wmnet [09:40:57] !log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for backupmon1001.eqiad.wmnet [09:41:12] !log Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=testwiki --start-timestamp="20260311000000" --end-timestamp="20260805000000" --sleep="5" --batch-size="2"` for T434688 [09:41:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:42:51] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1165.eqiad.wmnet onto db1279.eqiad.wmnet [09:42:54] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1165: Depool db1165.eqiad.wmnet to then clone it to db1279.eqiad.wmnet - marostegui@cumin1003 [09:43:29] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1165: Depool db1165.eqiad.wmnet to then clone it to db1279.eqiad.wmnet - marostegui@cumin1003 [09:43:51] (03CR) 10JMeybohm: "Nice one. This even makes sense. 😊" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324344 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [09:44:15] PROBLEM - Recursive DNS on 2001:df2:e500:1:103:102:166:10 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [09:44:25] PROBLEM - Router interfaces on mr1-eqsin is CRITICAL: CRITICAL: No response from remote host 103.102.166.128 for 1.3.6.1.2.1.2.2.1.7 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [09:45:17] RECOVERY - Recursive DNS on 2001:df2:e500:1:103:102:166:10 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [09:45:25] RECOVERY - Router interfaces on mr1-eqsin is OK: OK: host 103.102.166.128, interfaces up: 38, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [09:45:42] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 22 hosts with reason: Cloning [09:45:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx errors from gitlab-replica-a.discovery.wmnet in magru #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=magru&var-cluster=text&var-origin=gitlab-replica-a.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [09:45:54] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 8 hosts with reason: reboot [09:46:32] checking [09:46:33] !log Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260801000000" --end-timestamp="20260802000000" --sleep="3" --batch-size="5"` for T434688 [09:46:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:46:40] T434688: Create maintenance script to evaluate revisions performed between timestamps against content policies - https://phabricator.wikimedia.org/T434688 [09:47:21] !incidents [09:47:21] 8245 (ACKED) ATSBackendErrorsHigh cache_text sre (gitlab-replica-a.discovery.wmnet magru) [09:47:37] arnaudb: any chance gitlab-replica-a 5xx are related to your reboots ? [09:47:55] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:48:41] gitlab.w.o is up for me FWIW [09:49:16] yeah, me too [09:50:02] investigating the 5xx, though my money is on https://gerrit.wikimedia.org/r/c/operations/dns/+/1324280 [09:50:16] or at least sth related [09:50:43] yeah [09:50:45] godog: it can be related yes [09:50:56] I can check in a few if needed [09:51:00] this is ongoing since 8:40 [09:51:39] "upstream connect error or disconnect/reset before headers. reset reason: connection timeout" [09:51:59] <_joe_> that sounds likee envoy not reaching its backend [09:52:20] yes very much needed arnaudb, thank you [09:52:21] replica-b is fine [09:53:46] also yes what jayme said, has been going on for a while [09:54:17] Side note, GitLab replicas shouldn't be a paging alert. I know, granularity of ATS monitoring, etc. but I want to clarify the lack of impact. [09:54:40] ack thank you sobanski that's useful [09:55:36] (03CR) 10Hnowlan: [V:03+2 C:03+2] debian: guard against missing systemd package [debs/pint] - 10https://gerrit.wikimedia.org/r/1319039 (owner: 10Hnowlan) [09:55:36] means we can wait for arnaudb to investigate? [09:55:52] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1151.eqiad.wmnet with reason: host reimage [09:56:17] PROBLEM - Recursive DNS on 2001:df2:e500:1:103:102:166:10 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [09:56:33] FIRING: KubernetesAPINotScrapable: k8s-dse@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [09:57:17] RECOVERY - Recursive DNS on 2001:df2:e500:1:103:102:166:10 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [09:57:55] jayme: yes [09:58:06] \i/ [09:58:11] PROBLEM - Recursive DNS on 2001:df2:e500:2:103:102:166:36 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [09:58:42] (03PS9) 10Majavah: P:wmcs::novaproxy: Add HAProxy in front of all requests [puppet] - 10https://gerrit.wikimedia.org/r/1308623 (https://phabricator.wikimedia.org/T429930) [09:58:42] (03PS1) 10Majavah: P:wmcs::novaproxy: Add timeouts and limits to HTTP frontend [puppet] - 10https://gerrit.wikimedia.org/r/1325427 (https://phabricator.wikimedia.org/T429930) [09:58:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145503 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:58:45] (03PS1) 10Majavah: P:wmcs::novaproxy: Move rate limiting to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325428 (https://phabricator.wikimedia.org/T429930) [09:58:47] (03PS1) 10CWilliams: mariadb: Add db1274 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1325429 (https://phabricator.wikimedia.org/T407942) [09:59:10] RECOVERY - Recursive DNS on 2001:df2:e500:2:103:102:166:36 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [09:59:13] !log jynus@cumin1003 START - Cookbook sre.hosts.remove-downtime for 8 hosts [09:59:18] !log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 8 hosts [09:59:21] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1151.eqiad.wmnet with reason: host reimage [09:59:31] !log fceratto@cumin1003 START - Cookbook sre.mysql.decommission [09:59:32] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) [09:59:40] (03CR) 10Hashar: [C:03+1] ci: add ReadingLists to gitcache [puppet] - 10https://gerrit.wikimedia.org/r/1324278 (https://phabricator.wikimedia.org/T403560) (owner: 10Phedenskog) [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1000) [10:00:28] (03CR) 10JMeybohm: [C:03+2] wikikube: Update codfw to calico 3.30 [puppet] - 10https://gerrit.wikimedia.org/r/1325420 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [10:00:47] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-debug: apply [10:00:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx errors from gitlab-replica-a.discovery.wmnet in magru #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=magru&var-cluster=text&var-origin=gitlab-replica-a.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [10:00:58] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [10:01:11] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-debug: apply [10:01:18] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2214: Upgrade of db2214.codfw.wmnet completed [10:01:18] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for 1 hosts [10:01:33] RESOLVED: KubernetesAPINotScrapable: k8s-dse@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [10:01:38] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [10:01:46] (03CR) 10CI reject: [V:04-1] P:wmcs::novaproxy: Move rate limiting to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325428 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [10:02:06] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 7 hosts with reason: reboot [10:02:25] (03PS2) 10Majavah: P:wmcs::novaproxy: Move rate limiting to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325428 (https://phabricator.wikimedia.org/T429930) [10:03:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145503 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:03:51] FIRING: [5x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:04:28] PROBLEM - Router interfaces on mr1-eqsin is CRITICAL: CRITICAL: No response from remote host 103.102.166.128 for 1.3.6.1.2.1.2.2.1.2 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [10:04:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx errors from swift.discovery.wmnet in eqsin #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqsin&var-cluster=upload&var-origin=swift.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [10:05:24] checking [10:05:28] thumbor in eqiad looking sad [10:05:50] !incidents [10:05:50] 8246 (ACKED) ATSBackendErrorsHigh cache_upload sre (swift.discovery.wmnet eqsin) [10:05:50] 8245 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab-replica-a.discovery.wmnet magru) [10:06:31] hnowlan: very sad indeed, looking into 5xx [10:06:50] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-debug: apply [10:07:10] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [10:07:55] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:08:26] RECOVERY - Router interfaces on mr1-eqsin is OK: OK: host 103.102.166.128, interfaces up: 38, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [10:09:46] probably just overloaded...increased 2xx as well [10:09:59] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325431 [10:10:02] spike in codfw also but eqiad is significantly larger and harming pods [10:11:07] (03PS1) 10JavierMonton: pageview-trending-relative: Flink app alerts [alerts] - 10https://gerrit.wikimedia.org/r/1325432 (https://phabricator.wikimedia.org/T431536) [10:11:25] I'm failing to see an obvious pattern in requests or errors, could be indeed a case of too much legit load ? [10:12:03] (03CR) 10Ladsgroup: "I checked every single one manually to make sure they don't change." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324832 (owner: 10Ladsgroup) [10:12:09] unlikely given the sudden jump [10:12:21] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325433 [10:12:28] PROBLEM - Router interfaces on mr1-eqsin is CRITICAL: CRITICAL: No response from remote host 103.102.166.128 for 1.3.6.1.2.1.2.2.1.7 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [10:12:55] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:13:14] PROBLEM - Recursive DNS on 2001:df2:e500:2:103:102:166:36 is CRITICAL: DNS_QUERY CRITICAL - Connection timed out https://wikitech.wikimedia.org/wiki/DNS [10:13:20] PROBLEM - Recursive DNS on 2001:df2:e500:1:103:102:166:10 is CRITICAL: DNS_QUERY CRITICAL - Connection timed out https://wikitech.wikimedia.org/wiki/DNS [10:13:30] (03PS2) 10Majavah: P:wmcs::novaproxy: Add timeouts and limits to HTTP frontend [puppet] - 10https://gerrit.wikimedia.org/r/1325427 (https://phabricator.wikimedia.org/T429930) [10:13:30] (03PS10) 10Majavah: P:wmcs::novaproxy: Add HAProxy in front of all requests [puppet] - 10https://gerrit.wikimedia.org/r/1308623 (https://phabricator.wikimedia.org/T429930) [10:13:30] (03PS3) 10Majavah: P:wmcs::novaproxy: Move rate limiting to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325428 (https://phabricator.wikimedia.org/T429930) [10:13:30] (03PS1) 10Majavah: P:wmcs::novaproxy: Move various headers to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325434 (https://phabricator.wikimedia.org/T429930) [10:13:32] (03PS1) 10Majavah: P:wmcs::novaproxy: Move static bans to HAProxy configuration [puppet] - 10https://gerrit.wikimedia.org/r/1325435 (https://phabricator.wikimedia.org/T429930) [10:13:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145503 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:13:46] investigating gitlab-replica-a [10:14:12] RECOVERY - Recursive DNS on 2001:df2:e500:2:103:102:166:36 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [10:14:18] RECOVERY - Recursive DNS on 2001:df2:e500:1:103:102:166:10 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [10:15:20] RECOVERY - Router interfaces on mr1-eqsin is OK: OK: host 103.102.166.128, interfaces up: 38, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [10:15:25] FIRING: [2x] GanetiBGPDown: BGP session down between ganeti3005 and asw1-by27-esams - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPDown - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPDown [10:15:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx errors from gitlab-replica-a.discovery.wmnet in magru #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=magru&var-cluster=text&var-origin=gitlab-replica-a.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [10:15:56] !log installing bind9 security updates (client-side tools/libs only) [10:15:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:16:01] gitlab-replica-a is back [10:16:04] sorry for the noise [10:17:33] (03CR) 10Ladsgroup: [C:03+1] "I will never understand why we keep this many regexes so it's a pain every time we add a new section but out of scope of this patch (just " [puppet] - 10https://gerrit.wikimedia.org/r/1321907 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [10:17:59] (03PS22) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [10:18:12] (03CR) 10Marostegui: "Yeah 🤷‍♂️" [puppet] - 10https://gerrit.wikimedia.org/r/1321907 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [10:18:16] PROBLEM - NTP peers and stratum check on dns5004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/NTP [10:18:18] (03CR) 10Marostegui: [C:03+2] mariadb: Add test-s4 [puppet] - 10https://gerrit.wikimedia.org/r/1321907 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [10:19:33] (03PS23) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [10:19:51] FIRING: [2x] ATSBackendErrorsHigh: ATS: elevated 5xx errors from swift.discovery.wmnet in eqsin #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [10:20:21] arnaudb: https://gitlab-replica-a.wikimedia.org still fails to load for me [10:20:25] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12211462 (10BTullis) a:03BTullis [10:20:42] (03PS1) 10Marostegui: instances.yaml: Add test-s4 hosts to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1325438 (https://phabricator.wikimedia.org/T427059) [10:21:11] (03CR) 10Marostegui: [C:03+1] mariadb: Add db1274 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1325429 (https://phabricator.wikimedia.org/T407942) (owner: 10CWilliams) [10:22:14] PROBLEM - Auth DNS on dns5004 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [10:22:21] (03CR) 10CI reject: [V:04-1] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [10:22:26] (03PS2) 10Marostegui: instances.yaml: Add test-s4 hosts to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1325438 (https://phabricator.wikimedia.org/T427059) [10:22:48] (03PS24) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [10:24:16] urgh it's still failing indeed, sorry about that, checking again [10:24:20] (03PS2) 10CWilliams: mariadb: Add db1274 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1325429 (https://phabricator.wikimedia.org/T407942) [10:24:33] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1151.eqiad.wmnet with OS bookworm [10:24:51] FIRING: [2x] ATSBackendErrorsHigh: ATS: elevated 5xx errors from swift.discovery.wmnet in eqsin #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [10:25:09] 06SRE, 10SRE-SLO, 06SRE Observability, 06Traffic: Page on ATS backend errors relative to traffic - https://phabricator.wikimedia.org/T400675#12211488 (10fgiunchedi) Conversely, alerting on ratio will also cover low-traffic services that are erroring out just below the threshold. As it was the case today fo... [10:25:41] (03CR) 10CI reject: [V:04-1] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [10:26:12] RECOVERY - Auth DNS on dns5004 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [10:27:31] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1165: Pool db1165.eqiad.wmnet in after cloning [10:27:55] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:28:06] RECOVERY - NTP peers and stratum check on dns5004 is OK: NTP OK: Offset -4.4001e-05 secs, stratum=2 https://wikitech.wikimedia.org/wiki/NTP [10:28:16] (03CR) 10Phuedx: [C:03+1] Deploy GrowthBook 5.0.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325294 (https://phabricator.wikimedia.org/T434098) (owner: 10Santiago Faci) [10:28:48] godog: gitlab-replica-a is failing periodically, it's coming back up with a reconfigure but fails after a short while [10:28:52] digging [10:29:09] arnaudb: ack, thank you [10:30:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx errors from gitlab-replica-a.discovery.wmnet in magru #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=magru&var-cluster=text&var-origin=gitlab-replica-a.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [10:32:44] hopefully fixed now godog [10:33:33] FIRING: KubernetesAPINotScrapable: k8s-dse@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [10:33:44] 06SRE, 10SRE-SLO, 06SRE Observability, 06Traffic: Page on ATS backend errors relative to traffic - https://phabricator.wikimedia.org/T400675#12211522 (10fgiunchedi) Today also a swift.discovery.wmnet page came in, although IMHO it should have been a thumbor page instead. Swift itself was fine compared to i... [10:33:50] arnaudb: cheers [10:34:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx errors from swift.discovery.wmnet in eqsin #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqsin&var-cluster=upload&var-origin=swift.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [10:35:24] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet [10:36:15] (03PS1) 10Marostegui: instances.yaml: Remove db1151 [puppet] - 10https://gerrit.wikimedia.org/r/1325441 (https://phabricator.wikimedia.org/T434538) [10:37:34] (03CR) 10Marostegui: [C:03+2] instances.yaml: Remove db1151 [puppet] - 10https://gerrit.wikimedia.org/r/1325441 (https://phabricator.wikimedia.org/T434538) (owner: 10Marostegui) [10:37:53] (03PS1) 10Federico Ceratto: sre.mysql.parsercache: use shared ensure [cookbooks] - 10https://gerrit.wikimedia.org/r/1325440 [10:38:29] !log marostegui@cumin1003 dbctl commit (dc=all): 'Remove db1151 from dbctl T434538', diff saved to https://phabricator.wikimedia.org/P96055 and previous config saved to /var/cache/conftool/dbconfig/20260813-103828-marostegui.json [10:38:33] RESOLVED: KubernetesAPINotScrapable: k8s-dse@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [10:38:33] T434538: decommission db1151.eqiad.wmnet - https://phabricator.wikimedia.org/T434538 [10:38:38] (03CR) 10Santiago Faci: [C:03+2] Deploy GrowthBook 5.0.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325294 (https://phabricator.wikimedia.org/T434098) (owner: 10Santiago Faci) [10:39:31] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet [10:40:11] (03PS3) 10CWilliams: mariadb: Add db1274 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1325429 (https://phabricator.wikimedia.org/T407942) [10:40:52] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[1216,1225,1239-1240].eqiad.wmnet with reason: reboot [10:40:58] (03CR) 10JMeybohm: [C:03+2] wikikube-codfw: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325421 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [10:41:06] (03Merged) 10jenkins-bot: Deploy GrowthBook 5.0.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325294 (https://phabricator.wikimedia.org/T434098) (owner: 10Santiago Faci) [10:42:26] !log jayme@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [10:42:34] !log jayme@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [10:43:20] !log jayme@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [10:44:22] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply [10:45:38] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet [10:45:45] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet [10:46:03] (03PS3) 10Marostegui: instances.yaml: Add test-s4 hosts to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1325438 (https://phabricator.wikimedia.org/T427059) [10:47:48] (03CR) 10CWilliams: [C:03+2] mariadb: Add db1274 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1325429 (https://phabricator.wikimedia.org/T407942) (owner: 10CWilliams) [10:48:51] FIRING: [5x] ProbeDown: Service ganeti1033:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:50:34] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Enabling notifications and pooling [10:51:50] (03PS1) 10CWilliams: mariadb: Enabling notifications for db1274 [puppet] - 10https://gerrit.wikimedia.org/r/1325442 (https://phabricator.wikimedia.org/T407942) [10:53:12] (03CR) 10CWilliams: [C:03+2] mariadb: Enabling notifications for db1274 [puppet] - 10https://gerrit.wikimedia.org/r/1325442 (https://phabricator.wikimedia.org/T407942) (owner: 10CWilliams) [10:54:33] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply [10:55:47] FIRING: HelmReleaseBadStatus: Helm release kube-system/calico on k8s@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s&var-namespace=kube-system - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:57:18] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply [10:59:38] (03CR) 10Marostegui: "I'd appreciate a review to make sure the hosts I am adding are indeed test-s4 and I didn't miss any/made typos." [puppet] - 10https://gerrit.wikimedia.org/r/1325438 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [11:07:29] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply [11:12:44] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1165: Pool db1165.eqiad.wmnet in after cloning [11:12:52] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1165.eqiad.wmnet onto db1279.eqiad.wmnet [11:13:02] PROBLEM - Host wikikube-worker1067 is DOWN: PING CRITICAL - Packet loss = 71%, RTA = 6003.95 ms [11:14:08] RECOVERY - Host wikikube-worker1067 is UP: PING OK - Packet loss = 0%, RTA = 0.81 ms [11:14:21] (03PS1) 10Majavah: hieradata: Update striker-toolsbeta to 2026-08-13-111038-production [puppet] - 10https://gerrit.wikimedia.org/r/1325445 [11:14:21] (03PS1) 10Majavah: hieradata: Update production Striker to 2026-08-13-111038-production [puppet] - 10https://gerrit.wikimedia.org/r/1325446 [11:15:19] (03CR) 10Majavah: [C:03+2] hieradata: Update striker-toolsbeta to 2026-08-13-111038-production [puppet] - 10https://gerrit.wikimedia.org/r/1325445 (owner: 10Majavah) [11:16:13] (03PS7) 10Cparle: Baseline user-facing metrics for thumbnails, with an experiment mechanism [puppet] - 10https://gerrit.wikimedia.org/r/1324778 (https://phabricator.wikimedia.org/T431597) [11:17:20] PROBLEM - OSPF status on cr2-drmrs is CRITICAL: OSPFv2: 6/6 UP : OSPFv3: 5/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [11:17:34] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1213.eqiad.wmnet with OS bookworm [11:18:20] RECOVERY - OSPF status on cr2-drmrs is OK: OSPFv2: 6/6 UP : OSPFv3: 6/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [11:18:36] (03CR) 10CI reject: [V:04-1] Baseline user-facing metrics for thumbnails, with an experiment mechanism [puppet] - 10https://gerrit.wikimedia.org/r/1324778 (https://phabricator.wikimedia.org/T431597) (owner: 10Cparle) [11:19:58] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1034.eqiad.wmnet [11:20:59] (03CR) 10Majavah: [C:03+2] hieradata: Update production Striker to 2026-08-13-111038-production [puppet] - 10https://gerrit.wikimedia.org/r/1325446 (owner: 10Majavah) [11:22:02] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet [11:26:24] (03PS1) 10Kevin Bazira: ml-services: deploy qwen36-27b and qwen3-14b isvcs with support for thinking features [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325449 (https://phabricator.wikimedia.org/T433569) [11:26:44] (03PS1) 10Marostegui: mariadb: Productionize db1281 [puppet] - 10https://gerrit.wikimedia.org/r/1325450 (https://phabricator.wikimedia.org/T407942) [11:26:51] FIRING: [2x] ATSBackendErrorsHigh: ATS: elevated 5xx errors from gitlab-replica-a.discovery.wmnet in esams #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [11:27:06] Wow [11:27:08] !ack [11:27:08] 8248 (ACKED) [2x] ATSBackendErrorsHigh cache_text sre (gitlab-replica-a.discovery.wmnet) [11:27:38] fabfur: a repeat of a non-critical problem [11:27:42] ok [11:27:47] glad to hear [11:28:08] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet [11:28:13] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet [11:28:24] One that we were quite sure Arnaud fixed an hour ago [11:28:47] ok, given isn't critical let's wait for a confirmation [11:28:51] FIRING: [5x] ProbeDown: Service ganeti1034:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:29:42] (03PS8) 10Cparle: Baseline user-facing metrics for thumbnails, with an experiment mechanism [puppet] - 10https://gerrit.wikimedia.org/r/1324778 (https://phabricator.wikimedia.org/T431597) [11:29:52] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1281 [puppet] - 10https://gerrit.wikimedia.org/r/1325450 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [11:31:36] (03PS1) 10Ladsgroup: thumbor: Pin haproxy and prometheus-statsd-exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325451 (https://phabricator.wikimedia.org/T348856) [11:31:58] (03CR) 10CI reject: [V:04-1] Baseline user-facing metrics for thumbnails, with an experiment mechanism [puppet] - 10https://gerrit.wikimedia.org/r/1324778 (https://phabricator.wikimedia.org/T431597) (owner: 10Cparle) [11:32:12] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1213.eqiad.wmnet with reason: host reimage [11:32:16] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12211683 (10BTullis) I did the first of these, an-worker1213. The procedure, such as it is, is written out here: https://wikitech.wikimedia.org/wiki/Data_Plat... [11:32:31] (03CR) 10Bartosz Wójtowicz: [C:03+1] ml-services: deploy qwen36-27b and qwen3-14b isvcs with support for thinking features [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325449 (https://phabricator.wikimedia.org/T433569) (owner: 10Kevin Bazira) [11:32:52] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12211686 (10BTullis) [11:33:12] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy qwen36-27b and qwen3-14b isvcs with support for thinking features [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325449 (https://phabricator.wikimedia.org/T433569) (owner: 10Kevin Bazira) [11:33:55] 06SRE, 06Infrastructure-Foundations, 10netops: Problems using Homer with SSH Agent on Debian 13 - https://phabricator.wikimedia.org/T434773 (10cmooney) 03NEW p:05Triage→03Medium [11:34:34] !log remove ganeti3005 from esams03 cluster, hardware issues T434646 [11:34:34] (03PS9) 10Cparle: Baseline user-facing metrics for thumbnails, with an experiment mechanism [puppet] - 10https://gerrit.wikimedia.org/r/1324778 (https://phabricator.wikimedia.org/T431597) [11:34:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:34:38] T434646: ganeti3005 shows backplane error after reboot - https://phabricator.wikimedia.org/T434646 [11:34:52] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 22 hosts with reason: Cloning [11:35:30] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1167.eqiad.wmnet onto db1281.eqiad.wmnet [11:35:33] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1167: Depool db1167.eqiad.wmnet to then clone it to db1281.eqiad.wmnet - marostegui@cumin1003 [11:35:49] (03Merged) 10jenkins-bot: ml-services: deploy qwen36-27b and qwen3-14b isvcs with support for thinking features [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325449 (https://phabricator.wikimedia.org/T433569) (owner: 10Kevin Bazira) [11:35:53] !log cgoubert@cumin2003 START - Cookbook sre.ganeti.makevm for new host rdb-lock2003.codfw.wmnet [11:35:55] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [11:36:28] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1167: Depool db1167.eqiad.wmnet to then clone it to db1281.eqiad.wmnet - marostegui@cumin1003 [11:36:41] (03CR) 10CI reject: [V:04-1] Baseline user-facing metrics for thumbnails, with an experiment mechanism [puppet] - 10https://gerrit.wikimedia.org/r/1324778 (https://phabricator.wikimedia.org/T431597) (owner: 10Cparle) [11:38:21] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1213.eqiad.wmnet with reason: host reimage [11:39:21] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops, 07Wikimedia-Incident: codfw S2 db2207 master fail - https://phabricator.wikimedia.org/T432398#12211727 (10Marostegui) 05In progress→03Resolved a:03Marostegui I am going to close this for now, as there's an email thread with Dell. We can reopen if needed. [11:40:25] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" [11:40:30] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" [11:40:30] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:40:30] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors [11:40:33] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors [11:40:37] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2209 firmware upgrade after a crash - https://phabricator.wikimedia.org/T431952#12211743 (10Marostegui) @Jhancock.wm if you are okay, I am going to repool this host. There's the open email thread with Dell so we can follow up there and reopen this task if needed? [11:40:52] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [11:41:19] (03PS1) 10Cathal Mooney: junos.py: set allow_agent to true for use with PyEZ v2.8.0+ [software/homer] - 10https://gerrit.wikimedia.org/r/1325453 (https://phabricator.wikimedia.org/T434773) [11:41:38] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1038.eqiad.wmnet [11:41:51] FIRING: [4x] ATSBackendErrorsHigh: ATS: elevated 5xx errors from gitlab-replica-a.discovery.wmnet in drmrs #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [11:42:23] !incidents [11:42:23] 8248 (ACKED) [2x] ATSBackendErrorsHigh cache_text sre (gitlab-replica-a.discovery.wmnet) [11:42:24] 8246 (RESOLVED) ATSBackendErrorsHigh cache_upload sre (swift.discovery.wmnet eqsin) [11:42:24] 8247 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab-replica-a.discovery.wmnet magru) [11:42:24] 8245 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab-replica-a.discovery.wmnet magru) [11:44:42] !log installing Linux 5.10.262 on Bullseye hosts [11:44:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:44:51] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:45:02] (03PS1) 10Arnaudb: gitlab: discard firewall throttling on gitlab-replicas [puppet] - 10https://gerrit.wikimedia.org/r/1325454 (https://phabricator.wikimedia.org/T425441) [11:45:04] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Problems using Homer with SSH Agent on Debian 13 - https://phabricator.wikimedia.org/T434773#12211776 (10cmooney) [11:45:47] RESOLVED: HelmReleaseBadStatus: Helm release kube-system/calico on k8s@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s&var-namespace=kube-system - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:46:01] jmm@cumin2003 drain-node (PID 1293722) is awaiting input [11:46:45] (03PS25) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [11:47:02] cgoubert@cumin2003 makevm (PID 1293048) is awaiting input [11:47:37] (03PS1) 10Zabe: Api: Use correct query when continue prop=categories [core] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325455 (https://phabricator.wikimedia.org/T433922) [11:48:13] (03CR) 10Blake: [C:03+1] ingress: Introduce istio 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [11:49:47] (03CR) 10CI reject: [V:04-1] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [11:49:59] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host apus-be1004.eqiad.wmnet [11:51:51] RESOLVED: [4x] ATSBackendErrorsHigh: ATS: elevated 5xx errors from gitlab-replica-a.discovery.wmnet in drmrs #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [11:52:36] (03CR) 10Blake: [C:03+1] mediawiki: Bump ingress.istio from 1.2.0 to 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324344 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [11:53:04] 🎉 apologies for that [11:53:41] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet [11:53:49] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" [11:53:53] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" [11:53:53] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:53:53] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors [11:53:56] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors [11:54:08] !log cgoubert@cumin2003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rdb-lock2003.codfw.wmnet [11:55:14] (03CR) 10CI reject: [V:04-1] junos.py: set allow_agent to true for use with PyEZ v2.8.0+ [software/homer] - 10https://gerrit.wikimedia.org/r/1325453 (https://phabricator.wikimedia.org/T434773) (owner: 10Cathal Mooney) [11:55:32] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be1004.eqiad.wmnet [11:56:12] (03CR) 10Blake: mw-pretrain: Introduce dedicated Ingress Gateway for jobrunner (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324780 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [11:56:14] PROBLEM - Host kubestagemaster1003 is DOWN: PING CRITICAL - Packet loss = 100% [11:56:29] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host apus-be1005.eqiad.wmnet [11:57:17] FIRING: [2x] HelmReleaseBadStatus: Helm release kube-system/calico on k8s@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s&var-namespace=kube-system - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:58:56] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1213.eqiad.wmnet with OS bookworm [11:59:29] (03CR) 10Blake: [C:03+1] kubernetes: Enable and configure mw-pretrain scap deployments [puppet] - 10https://gerrit.wikimedia.org/r/1324826 (https://phabricator.wikimedia.org/T428972) (owner: 10Scott French) [12:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1200) [12:00:18] (03CR) 10Blake: [C:03+1] admin_ng: Add mw-pretrain-jobrunner to mw-pretrain tlsExtraSANs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324775 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [12:00:34] RECOVERY - Host kubestagemaster1003 is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms [12:00:46] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [12:01:06] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [12:01:20] (03PS26) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [12:01:28] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1038.eqiad.wmnet [12:01:29] (03PS1) 10Filippo Giunchedi: site: get cloudvirts ready for E4 -> C8 move [puppet] - 10https://gerrit.wikimedia.org/r/1325458 (https://phabricator.wikimedia.org/T431682) [12:01:32] (03PS1) 10Filippo Giunchedi: installserver: switch cloudvirt to efi standard recipes [puppet] - 10https://gerrit.wikimedia.org/r/1325459 (https://phabricator.wikimedia.org/T431682) [12:01:34] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1038.eqiad.wmnet [12:01:59] (03CR) 10Blake: wmnet: Add mw-pretrain-jobrunner k8s Ingress CNAMEs (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1324776 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [12:02:08] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be1005.eqiad.wmnet [12:02:58] (03PS1) 10Blake: api-gateway: remove deprecated service. [labs/private] - 10https://gerrit.wikimedia.org/r/1315832 (https://phabricator.wikimedia.org/T432460) [12:03:11] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host apus-be1006.eqiad.wmnet [12:03:51] FIRING: [7x] ProbeDown: Service ganeti1038:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:03:58] RESOLVED: KubernetesCalicoDown: kubestagemaster1003.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-staging&var-instance=kubestagemaster1003.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:04:09] (03CR) 10CI reject: [V:04-1] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [12:04:17] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:04:30] !log jayme@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [12:04:41] (03CR) 10Clément Goubert: [C:03+1] api-gateway: remove deprecated service. [labs/private] - 10https://gerrit.wikimedia.org/r/1315832 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake) [12:04:46] (03PS1) 10Cathal Mooney: eqsin: add reverse INCLUDEs for public1-604-eqsin vlan and tidy up [dns] - 10https://gerrit.wikimedia.org/r/1325460 (https://phabricator.wikimedia.org/T418439) [12:04:52] (03PS27) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [12:05:37] (03PS2) 10Blake: api-gateway: remove deprecated service. [labs/private] - 10https://gerrit.wikimedia.org/r/1315832 (https://phabricator.wikimedia.org/T432460) [12:05:44] (03CR) 10CI reject: [V:04-1] eqsin: add reverse INCLUDEs for public1-604-eqsin vlan and tidy up [dns] - 10https://gerrit.wikimedia.org/r/1325460 (https://phabricator.wikimedia.org/T418439) (owner: 10Cathal Mooney) [12:05:55] !log cgoubert@cumin2003 START - Cookbook sre.ganeti.makevm for new host rdb-lock2003.codfw.wmnet [12:05:56] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [12:06:23] (03CR) 10Blake: [C:03+2] api-gateway: remove deprecated service. [labs/private] - 10https://gerrit.wikimedia.org/r/1315832 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake) [12:06:59] (03CR) 10Blake: [V:03+2 C:03+2] api-gateway: remove deprecated service. [labs/private] - 10https://gerrit.wikimedia.org/r/1315832 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake) [12:07:17] RESOLVED: HelmReleaseBadStatus: Helm release kube-system/calico on k8s@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s&var-namespace=kube-system - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [12:07:28] (03CR) 10CI reject: [V:04-1] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [12:08:16] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1049.eqiad.wmnet [12:08:21] (03PS2) 10Cathal Mooney: junos.py: set allow_agent to true for use with PyEZ v2.8.0+ [software/homer] - 10https://gerrit.wikimedia.org/r/1325453 (https://phabricator.wikimedia.org/T434773) [12:09:22] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netflow7002.magru.wmnet [12:10:16] (03CR) 10Filippo Giunchedi: [C:03+1] P:wmcs::novaproxy: Add timeouts and limits to HTTP frontend [puppet] - 10https://gerrit.wikimedia.org/r/1325427 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [12:10:21] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" [12:10:26] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" [12:10:26] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:10:26] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors [12:10:29] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors [12:10:31] .51 [12:10:33] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be1006.eqiad.wmnet [12:10:48] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [12:10:59] !log mvernon@cumin2003 START - Cookbook sre.hosts.reboot-single for host moss-be1003.eqiad.wmnet [12:12:29] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1178.eqiad.wmnet with OS bookworm [12:12:41] claime: I'm fixing up some mess in the dns repo, I hope it won't mess up whatever you are doing - when I'm running the dns cookbook it tells me you have the lock right now [12:13:09] topranks: yeah i have a rollback going on a vm create [12:13:11] it's fine [12:13:28] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow7002.magru.wmnet [12:13:49] jmm@cumin2003 drain-node (PID 1299692) is awaiting input [12:13:54] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1049.eqiad.wmnet [12:14:00] shit sorry... yeah need to do this dance of updating the netbox-generated dns records without reloading gdnsd, then merging a patch I have in the dns repo [12:14:34] (03CR) 10Filippo Giunchedi: [C:03+1] "Nice!" [puppet] - 10https://gerrit.wikimedia.org/r/1308623 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [12:15:04] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1211.eqiad.wmnet with OS bookworm [12:15:08] (03CR) 10Filippo Giunchedi: [C:03+1] P:wmcs::novaproxy: Move rate limiting to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325428 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [12:15:08] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1210.eqiad.wmnet with OS bookworm [12:15:11] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1233.eqiad.wmnet with OS bookworm [12:15:29] (03CR) 10Filippo Giunchedi: [C:03+1] P:wmcs::novaproxy: Move various headers to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325434 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [12:15:56] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [12:16:07] (03CR) 10Filippo Giunchedi: [C:03+1] P:wmcs::novaproxy: Move static bans to HAProxy configuration [puppet] - 10https://gerrit.wikimedia.org/r/1325435 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [12:16:59] cgoubert@cumin2003 makevm (PID 1299445) is awaiting input [12:17:27] !log installin curl security updates [12:17:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:17:52] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet [12:18:13] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be1003.eqiad.wmnet [12:18:51] (03PS6) 10Blake: kube-state-metrics: update to upstream 7.3.0. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319069 (https://phabricator.wikimedia.org/T427405) [12:18:57] (03CR) 10Blake: "Added, with links to examples. Please let me know if there's more information you'd like to see included there. Thanks!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319069 (https://phabricator.wikimedia.org/T427405) (owner: 10Blake) [12:19:10] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" [12:19:16] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet [12:19:30] topranks: - 103.102.166.0/28: [12:19:32] (03PS2) 10Cathal Mooney: eqsin: add reverse INCLUDEs for public1-604-eqsin vlan and tidy up [dns] - 10https://gerrit.wikimedia.org/r/1325460 (https://phabricator.wikimedia.org/T418439) [12:19:32] + 103.102.166.0/27: [12:19:35] dis u? [12:19:39] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet [12:19:46] claime: indeed yes that's fine to merge thanks [12:19:52] ack [12:19:55] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003" [12:19:55] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:19:56] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors [12:19:59] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors [12:20:05] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox dns entries for expanded public1-603-eqsin subnet - cmooney@cumin1003" [12:20:10] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox dns entries for expanded public1-603-eqsin subnet - cmooney@cumin1003" [12:20:10] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:20:12] !log cgoubert@cumin2003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rdb-lock2003.codfw.wmnet [12:20:27] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1050.eqiad.wmnet [12:21:32] (03CR) 10Cathal Mooney: [C:03+2] eqsin: add reverse INCLUDEs for public1-604-eqsin vlan and tidy up [dns] - 10https://gerrit.wikimedia.org/r/1325460 (https://phabricator.wikimedia.org/T418439) (owner: 10Cathal Mooney) [12:21:51] !log cmooney@dns3003 START - running authdns-update [12:21:56] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet [12:23:31] jmm@cumin2003 drain-node (PID 1304297) is awaiting input [12:24:00] !log cmooney@dns3003 END - running authdns-update [12:24:14] claime: sorry for the hassle, dns records are now back in sync and everything should be happy [12:26:09] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Problems using Homer with SSH Agent on Debian 13 - https://phabricator.wikimedia.org/T434773#12211966 (10cmooney) p:05Medium→03Low [12:26:30] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1050.eqiad.wmnet [12:26:31] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netflow5003.eqsin.wmnet [12:26:42] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1325464 (owner: 10L10n-bot) [12:29:17] !log installing Wireshark security updates [12:29:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:29:22] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1211.eqiad.wmnet with reason: host reimage [12:29:45] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1210.eqiad.wmnet with reason: host reimage [12:31:52] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1050.eqiad.wmnet [12:31:53] (03PS1) 10Cparle: Update dependencies to make CI refresh [puppet] - 10https://gerrit.wikimedia.org/r/1325468 (https://phabricator.wikimedia.org/T427387) [12:31:57] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1050.eqiad.wmnet [12:32:05] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow5003.eqsin.wmnet [12:32:07] !log Drop SecurePoll tables from closed wikis T423128 [12:32:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:32:11] T423128: Drop SecurePoll tables from closed wikis - https://phabricator.wikimedia.org/T423128 [12:32:30] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host bast7002.wikimedia.org [12:32:34] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1051.eqiad.wmnet [12:33:26] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1233.eqiad.wmnet with reason: host reimage [12:33:31] (03PS2) 10Cparle: Update commit message validator dep [puppet] - 10https://gerrit.wikimedia.org/r/1325468 (https://phabricator.wikimedia.org/T427387) [12:34:52] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1211.eqiad.wmnet with reason: host reimage [12:36:04] (03Abandoned) 10Marostegui: wmnet: Update s6-master alias [dns] - 10https://gerrit.wikimedia.org/r/1276878 (https://phabricator.wikimedia.org/T424315) (owner: 10Gerrit maintenance bot) [12:36:12] (03Abandoned) 10Marostegui: mariadb: Promote db1201 to s6 master [puppet] - 10https://gerrit.wikimedia.org/r/1276877 (https://phabricator.wikimedia.org/T424315) (owner: 10Gerrit maintenance bot) [12:37:48] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1051.eqiad.wmnet [12:38:07] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1233.eqiad.wmnet with reason: host reimage [12:38:18] (03PS10) 10Cparle: Baseline user-facing metrics for thumbnails, with an experiment mechanism [puppet] - 10https://gerrit.wikimedia.org/r/1324778 (https://phabricator.wikimedia.org/T431597) [12:38:34] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast7002.wikimedia.org [12:39:08] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Added db1274', diff saved to https://phabricator.wikimedia.org/P96062 and previous config saved to /var/cache/conftool/dbconfig/20260813-123907-cwilliams.json [12:39:37] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db1274: New host [12:40:04] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [12:41:15] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1167: Pool db1167.eqiad.wmnet in after cloning [12:42:36] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1210.eqiad.wmnet with reason: host reimage [12:43:22] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet [12:43:28] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet [12:44:55] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns record for deleted IP reservations eqsin lvs vlan ints - cmooney@cumin1003" [12:44:57] PROBLEM - Host an-worker1204 is DOWN: PING CRITICAL - Packet loss = 100% [12:44:59] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns record for deleted IP reservations eqsin lvs vlan ints - cmooney@cumin1003" [12:45:00] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:46:39] !log cwilliams@cumin1003 START - Cookbook sre.mysql.clone of db1169.eqiad.wmnet onto db1277.eqiad.wmnet [12:46:42] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db1169: Depool db1169.eqiad.wmnet to then clone it to db1277.eqiad.wmnet - cwilliams@cumin1003 [12:46:56] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12212030 (10BTullis) [12:49:18] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1169: Depool db1169.eqiad.wmnet to then clone it to db1277.eqiad.wmnet - cwilliams@cumin1003 [12:51:59] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1052.eqiad.wmnet [12:52:18] cwilliams@cumin1003 clone (PID 3467073) is awaiting input [12:52:20] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet [12:53:48] (03PS3) 10Filippo Giunchedi: openstack: export compute service info to prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1313958 (https://phabricator.wikimedia.org/T284747) [12:53:54] (03CR) 10Filippo Giunchedi: openstack: export compute service info to prometheus (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1313958 (https://phabricator.wikimedia.org/T284747) (owner: 10Filippo Giunchedi) [12:54:24] (03CR) 10CI reject: [V:04-1] openstack: export compute service info to prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1313958 (https://phabricator.wikimedia.org/T284747) (owner: 10Filippo Giunchedi) [12:55:02] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1052.eqiad.wmnet [12:55:11] RECOVERY - Host an-worker1204 is UP: PING OK - Packet loss = 0%, RTA = 0.13 ms [12:55:20] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1211.eqiad.wmnet with OS bookworm [12:56:23] PROBLEM - SSH on an-worker1204 is CRITICAL: connect to address 10.64.161.7 and port 22: Connection refused https://wikitech.wikimedia.org/wiki/SSH/monitoring [12:57:13] PROBLEM - Host ml-etcd1003 is DOWN: PING CRITICAL - Packet loss = 100% [12:58:13] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1300). [13:00:05] nemo-yiannis: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:14] 👋 [13:00:24] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1052.eqiad.wmnet [13:00:30] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1052.eqiad.wmnet [13:00:45] RECOVERY - Host ml-etcd1003 is UP: PING OK - Packet loss = 0%, RTA = 0.78 ms [13:00:52] nemo-yiannis: need a deployer or can you self-deploy? :) [13:00:58] i can deploy [13:01:01] should i go ahead? [13:01:05] (03PS1) 10STran: Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob [extensions/CheckUser] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325476 (https://phabricator.wikimedia.org/T434199) [13:01:06] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1036.eqiad.wmnet [13:01:15] nemo-yiannis: go ahead :) [13:01:20] ok [13:01:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, August 13 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/CheckUser] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325476 (https://phabricator.wikimedia.org/T434199) (owner: 10STran) [13:02:19] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1210.eqiad.wmnet with OS bookworm [13:02:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jgiannelos@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324731 (owner: 10Jgiannelos) [13:03:26] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1036.eqiad.wmnet [13:03:29] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netflow3004.esams.wmnet [13:03:59] (03Merged) 10jenkins-bot: prv: Enable parsoid rendering for 5 wikisource wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324731 (owner: 10Jgiannelos) [13:04:21] !log jgiannelos@deploy1003 Started scap sync-world: Backport for [[gerrit:1324731|prv: Enable parsoid rendering for 5 wikisource wikis]] [13:06:15] (03PS1) 10Marostegui: installserver: Do not format new hosts [puppet] - 10https://gerrit.wikimedia.org/r/1325477 [13:06:24] !log jgiannelos@deploy1003 jgiannelos: Backport for [[gerrit:1324731|prv: Enable parsoid rendering for 5 wikisource wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:06:59] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1233.eqiad.wmnet with OS bookworm [13:07:38] !log jgiannelos@deploy1003 jgiannelos: Continuing with deployment [13:07:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow3004.esams.wmnet [13:08:44] (03CR) 10Bking: [C:03+2] opensearch: point gc+age tracing to a file, and stop logging safepoint [puppet] - 10https://gerrit.wikimedia.org/r/1324763 (https://phabricator.wikimedia.org/T434685) (owner: 10Bking) [13:10:42] (03CR) 10Marostegui: [C:03+2] installserver: Do not format new hosts [puppet] - 10https://gerrit.wikimedia.org/r/1325477 (owner: 10Marostegui) [13:11:02] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1036.eqiad.wmnet [13:11:07] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1036.eqiad.wmnet [13:11:20] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1039.eqiad.wmnet [13:11:47] !log jgiannelos@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324731|prv: Enable parsoid rendering for 5 wikisource wikis]] (duration: 07m 26s) [13:11:56] ok i am done [13:12:06] (03CR) 10CWilliams: [C:03+1] "LGTM" [cookbooks] - 10https://gerrit.wikimedia.org/r/1325440 (owner: 10Federico Ceratto) [13:13:31] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1325438 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [13:13:51] FIRING: [5x] ProbeDown: Service ganeti1036:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:14:25] jmm@cumin2003 drain-node (PID 1317740) is awaiting input [13:14:29] I tagged a patch in last minute and can deploy myself [13:14:38] starting now [13:15:03] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy1003 using scap backport" [extensions/CheckUser] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325476 (https://phabricator.wikimedia.org/T434199) (owner: 10STran) [13:16:19] (03PS4) 10Marostegui: instances.yaml: Add test-s4 hosts to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1325438 (https://phabricator.wikimedia.org/T427059) [13:16:49] (03Merged) 10jenkins-bot: Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob [extensions/CheckUser] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325476 (https://phabricator.wikimedia.org/T434199) (owner: 10STran) [13:17:06] (03CR) 10Ssingh: [C:03+1] cache::proxy: remove avalanche parameter from sdbm function [puppet] - 10https://gerrit.wikimedia.org/r/1325279 (https://phabricator.wikimedia.org/T426379) (owner: 10Fabfur) [13:17:09] !log stran@deploy1003 Started scap sync-world: Backport for [[gerrit:1325476|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] [13:17:13] T434199: MediaWiki\JobQueue\Exceptions\JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T434199 [13:17:15] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add test-s4 hosts to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1325438 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [13:17:55] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [13:18:29] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1149.eqiad.wmnet with OS bookworm [13:18:32] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1214.eqiad.wmnet with OS bookworm [13:18:35] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1215.eqiad.wmnet with OS bookworm [13:18:49] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus3004.esams.wmnet [13:19:12] !log stran@deploy1003 stran: Backport for [[gerrit:1325476|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:19:17] !log sukhe@cumin1003 START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough [13:19:44] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netflow2003.codfw.wmnet [13:20:19] nothing to test in debug, will have to look at logstash to see if error rates go down so continuing deployment. [13:20:25] !log stran@deploy1003 stran: Continuing with deployment [13:20:26] !log sukhe@cumin1003 START - Cookbook sre.hosts.decommission for hosts lvs1013.eqiad.wmnet [13:22:07] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1039.eqiad.wmnet [13:22:08] (03CR) 10Bking: [C:03+2] opensearch-semantic-search: bump to opensearch 3.7.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319845 (https://phabricator.wikimedia.org/T433697) (owner: 10DCausse) [13:22:42] (03PS1) 10Marostegui: Revert "instances.yaml: Add test-s4 hosts to dbctl" [puppet] - 10https://gerrit.wikimedia.org/r/1325479 [13:23:14] (03CR) 10Marostegui: [V:03+2 C:03+2] Revert "instances.yaml: Add test-s4 hosts to dbctl" [puppet] - 10https://gerrit.wikimedia.org/r/1325479 (owner: 10Marostegui) [13:23:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2003.codfw.wmnet [13:24:08] !log Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260803000000" --end-timestamp="20260804000000" --sleep="5" --batch-size="10"` for T434688 [13:24:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:24:12] T434688: Create maintenance script to evaluate revisions performed between timestamps against content policies - https://phabricator.wikimedia.org/T434688 [13:24:16] !log Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260804000000" --end-timestamp="20260805000000" --sleep="5" --batch-size="10"` for T434688 [13:24:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:24:28] !log stran@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325476|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] (duration: 07m 19s) [13:24:34] T434199: MediaWiki\JobQueue\Exceptions\JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T434199 [13:24:35] done [13:24:51] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: New host [13:24:54] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus3004.esams.wmnet [13:24:55] !log sukhe@cumin1003 START - Cookbook sre.dns.netbox [13:25:12] !log Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260802000000" --end-timestamp="20260803000000" --sleep="5" --batch-size="10"` for T434688 [13:25:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:25:16] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus4003.ulsfo.wmnet [13:26:32] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Pool db1167.eqiad.wmnet in after cloning [13:26:40] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1167.eqiad.wmnet onto db1281.eqiad.wmnet [13:27:11] (03CR) 10Fabfur: [C:03+2] cache::proxy: remove avalanche parameter from sdbm function [puppet] - 10https://gerrit.wikimedia.org/r/1325279 (https://phabricator.wikimedia.org/T426379) (owner: 10Fabfur) [13:27:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet [13:27:48] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet [13:28:10] FIRING: BFDdown: BFD session down between cr1-eqiad and 2620:0:861:4:208:80:155:112 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:28:47] (03CR) 10Ssingh: wikimedia.org: add TXT verification for Mentimeter (033 comments) [dns] - 10https://gerrit.wikimedia.org/r/1324790 (https://phabricator.wikimedia.org/T434620) (owner: 10CDobbins) [13:29:48] (03PS1) 10STran: Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob [extensions/CheckUser] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1325480 (https://phabricator.wikimedia.org/T434199) [13:30:00] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, August 13 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/CheckUser] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1325480 (https://phabricator.wikimedia.org/T434199) (owner: 10STran) [13:30:04] !log sukhe@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" [13:30:33] !log sukhe@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" [13:30:33] !log sukhe@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:30:34] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1013.eqiad.wmnet [13:30:42] Hm I wasn't going to backport to .14 but an affected wiki is still on there until tomorrow and logging errors atm so I'd like to backport that in this window too after tests pass [13:30:45] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 06Traffic: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12212220 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin1003 for hosts: `lvs1013.eqiad.wmnet` - lvs1013.eqiad.wmnet (**PASS**) - Downti... [13:30:48] (03PS3) 10Hashar: ci: allow inbound ssh from controller on prod agent [2] [puppet] - 10https://gerrit.wikimedia.org/r/1320916 (https://phabricator.wikimedia.org/T418521) [13:30:48] (03CR) 10Hashar: "PCC https://puppet-compiler.wmflabs.org/output/1320916/7521/" [puppet] - 10https://gerrit.wikimedia.org/r/1320916 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [13:30:52] !log sukhe@cumin1003 START - Cookbook sre.hosts.decommission for hosts lvs1014.eqiad.wmnet [13:31:21] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus4003.ulsfo.wmnet [13:31:33] !log bking@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin2003 [13:31:46] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus5003.eqsin.wmnet [13:33:18] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1040.eqiad.wmnet [13:33:21] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1214.eqiad.wmnet with reason: host reimage [13:33:30] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netflow2004.codfw.wmnet [13:33:31] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1215.eqiad.wmnet with reason: host reimage [13:35:41] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1149.eqiad.wmnet with reason: host reimage [13:35:44] !log sukhe@cumin1003 START - Cookbook sre.dns.netbox [13:37:11] (03CR) 10Lerickson: [C:03+2] Enable egress via urldownloader for WDQS V2. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324809 (https://phabricator.wikimedia.org/T434321) (owner: 10Lerickson) [13:38:01] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus5003.eqsin.wmnet [13:38:28] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus6002.drmrs.wmnet [13:38:48] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2004.codfw.wmnet [13:39:23] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1214.eqiad.wmnet with reason: host reimage [13:39:33] (03Merged) 10jenkins-bot: Enable egress via urldownloader for WDQS V2. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324809 (https://phabricator.wikimedia.org/T434321) (owner: 10Lerickson) [13:40:01] CI passed, starting deploy [13:40:11] !log sukhe@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" [13:40:15] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet [13:40:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy1003 using scap backport" [extensions/CheckUser] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1325480 (https://phabricator.wikimedia.org/T434199) (owner: 10STran) [13:40:27] !log sukhe@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" [13:40:27] !log sukhe@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:40:28] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1014.eqiad.wmnet [13:40:45] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, and 2 others: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12212255 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin1003 for hosts: `lvs1014.eqiad.wmnet` - lvs1014.eqiad.wmnet (**PASS**) - Downt... [13:40:57] !log sukhe@cumin1003 START - Cookbook sre.hosts.decommission for hosts lvs1015.eqiad.wmnet [13:41:02] (03PS1) 10JMeybohm: calico: Use maxUnavailable 2% instead of 1 for rollingUpdate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325484 (https://phabricator.wikimedia.org/T427400) [13:41:11] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netflow1003.eqiad.wmnet [13:42:17] (03Merged) 10jenkins-bot: Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob [extensions/CheckUser] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1325480 (https://phabricator.wikimedia.org/T434199) (owner: 10STran) [13:42:40] !log stran@deploy1003 Started scap sync-world: Backport for [[gerrit:1325480|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] [13:42:45] T434199: MediaWiki\JobQueue\Exceptions\JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T434199 [13:43:06] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1149.eqiad.wmnet with reason: host reimage [13:43:10] FIRING: [2x] BFDdown: BFD session down between cr1-eqiad and 2620:0:861:4:208:80:155:112 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:43:28] ^ yeah that's fine and expected [13:43:33] reboot [13:43:45] !log Running `mwscript-k8s WikimediaAntiAbuse:BackfillAbuseReview.php --wiki=enwiki --start-timestamp="20260805000000" --end-timestamp="20260806000000" --sleep="5" --batch-size="10"` for T434688 [13:43:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:43:50] T434688: Create maintenance script to evaluate revisions performed between timestamps against content policies - https://phabricator.wikimedia.org/T434688 [13:44:31] (03CR) 10Clément Goubert: [C:03+1] "No problem with the whitespace diff imo." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313111 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [13:44:33] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus6002.drmrs.wmnet [13:44:43] !log stran@deploy1003 stran: Backport for [[gerrit:1325480|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:44:48] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus7002.magru.wmnet [13:45:10] !log stran@deploy1003 stran: Continuing with deployment [13:45:16] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus1005.eqiad.wmnet [13:45:32] !log sukhe@cumin1003 START - Cookbook sre.dns.netbox [13:45:37] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1040.eqiad.wmnet [13:45:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1040.eqiad.wmnet [13:45:55] !log cmooney@cumin1003 conftool action : set/pooled=no; selector: name=dns5003.* [13:46:34] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1003.eqiad.wmnet [13:46:39] !log cmooney@cumin1003 START - Cookbook sre.hosts.reboot-single for host dns5003.wikimedia.org [13:47:05] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1215.eqiad.wmnet with reason: host reimage [13:47:08] (03CR) 10Clément Goubert: [C:03+1] "Done" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313111 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [13:48:10] FIRING: [5x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:38 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:49:04] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12212316 (10BTullis) an-worker1204 has three drives missing. I have booted it but then it stopped and I have logged into over the serial console. ` Physical Dr... [13:49:18] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12212317 (10BTullis) [13:49:19] !log stran@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325480|Guard against malformed headers in SuggestedInvestigationsMatchSignalsAgainstUserJob (T434199)]] (duration: 06m 39s) [13:49:24] T434199: MediaWiki\JobQueue\Exceptions\JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T434199 [13:49:44] done [13:49:54] !log sukhe@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" [13:50:18] !log sukhe@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003" [13:50:19] !log sukhe@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:50:20] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1015.eqiad.wmnet [13:50:36] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, and 2 others: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12212330 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin1003 for hosts: `lvs1015.eqiad.wmnet` - lvs1015.eqiad.wmnet (**PASS**) - Downt... [13:50:41] (03PS9) 10Tiziano Fogli: grafana/image-renderer: install grafana-image-renderer [puppet] - 10https://gerrit.wikimedia.org/r/1324724 (https://phabricator.wikimedia.org/T432970) [13:50:54] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus7002.magru.wmnet [13:51:10] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host apifeatureusage1001.eqiad.wmnet with OS bookworm [13:51:13] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus2005.codfw.wmnet [13:51:13] !log cgoubert@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw [13:51:35] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet [13:51:38] (03PS1) 10CDobbins: installserver: put all cp hosts on UEFI [puppet] - 10https://gerrit.wikimedia.org/r/1325485 (https://phabricator.wikimedia.org/T414411) [13:51:55] PROBLEM - Host an-worker1204 is DOWN: PING CRITICAL - Packet loss = 100% [13:52:23] RECOVERY - SSH on an-worker1204 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u5 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [13:52:25] RECOVERY - Host an-worker1204 is UP: PING OK - Packet loss = 0%, RTA = 0.16 ms [13:53:10] FIRING: [9x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:38 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:53:18] (03CR) 10JMeybohm: [C:03+2] calico: Use maxUnavailable 2% instead of 1 for rollingUpdate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325484 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [13:53:26] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, and 2 others: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12212337 (10ssingh) [13:53:37] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, and 2 others: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12212339 (10ssingh) a:03VRiley-WMF [13:54:54] (03CR) 10Ssingh: installserver: put all cp hosts on UEFI (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1325485 (https://phabricator.wikimedia.org/T414411) (owner: 10CDobbins) [13:55:00] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1005.eqiad.wmnet [13:55:14] !log cmooney@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dns5003.wikimedia.org [13:55:20] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus1007.eqiad.wmnet [13:55:34] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet [13:55:43] RECOVERY - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1204 is OK: communication: 0 OK : controller: 0 OK : physical_disk: 0 OK : virtual_disk: 0 OK : bbu: 0 OK : enclosure: 0 OK https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [13:56:17] !log cmooney@cumin1003 conftool action : set/pooled=yes; selector: name=dns5003.*,service=authdns-update [13:56:31] !log jayme@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [13:56:32] !log cmooney@dns3003 START - running authdns-update [13:57:10] !log cgoubert@cumin2003 conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw [13:57:13] !log bking@cumin2003 END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin2003 [13:57:22] !log cgoubert@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad [13:57:27] (03PS2) 10CDobbins: installserver: put all cp hosts on UEFI [puppet] - 10https://gerrit.wikimedia.org/r/1325485 (https://phabricator.wikimedia.org/T414411) [13:57:36] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet [13:57:39] (03CR) 10CDobbins: installserver: put all cp hosts on UEFI (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1325485 (https://phabricator.wikimedia.org/T414411) (owner: 10CDobbins) [13:58:36] !log cmooney@dns3003 END - running authdns-update [13:58:51] FIRING: [6x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:59:14] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netflow1002.eqiad.wmnet [13:59:19] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1041.eqiad.wmnet [13:59:29] !log cmooney@cumin1003 conftool action : set/pooled=yes; selector: name=dns5003.* [13:59:48] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1214.eqiad.wmnet with OS bookworm [14:00:32] !log installing libxml2 security updates [14:00:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:01:38] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet [14:02:02] !log cgoubert@cumin2003 conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad [14:02:23] jmm@cumin2003 drain-node (PID 1328477) is awaiting input [14:02:56] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on apifeatureusage1001.eqiad.wmnet with reason: host reimage [14:03:05] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1002.eqiad.wmnet [14:03:12] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1007.eqiad.wmnet [14:03:51] FIRING: [8x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:03:53] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2005.codfw.wmnet [14:03:57] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus2007.codfw.wmnet [14:04:12] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus1006.eqiad.wmnet [14:06:08] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1041.eqiad.wmnet [14:06:52] !log jayme@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [14:07:00] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1149.eqiad.wmnet with OS bookworm [14:08:21] (03CR) 10FNegri: [C:03+1] "LGTM!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1314730 (owner: 10Federico Ceratto) [14:08:22] PROBLEM - Host kubestagemaster1004 is DOWN: PING CRITICAL - Packet loss = 100% [14:08:24] PROBLEM - Host dse-k8s-etcd1002 is DOWN: PING CRITICAL - Packet loss = 100% [14:08:31] !log updated calico to v3.30.7 on wikikube codfw T427400 [14:08:34] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet [14:08:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:08:36] T427400: Update calico to 3.30 - https://phabricator.wikimedia.org/T427400 [14:08:40] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1215.eqiad.wmnet with OS bookworm [14:08:51] FIRING: [8x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:09:00] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on apifeatureusage1001.eqiad.wmnet with reason: host reimage [14:09:13] !log cgoubert@cumin2003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad [14:09:16] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet [14:09:18] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet [14:10:26] RECOVERY - Host dse-k8s-etcd1002 is UP: PING OK - Packet loss = 0%, RTA = 0.82 ms [14:10:50] RECOVERY - Host kubestagemaster1004 is UP: PING OK - Packet loss = 0%, RTA = 0.92 ms [14:11:30] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1041.eqiad.wmnet [14:11:36] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1041.eqiad.wmnet [14:11:38] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2007.codfw.wmnet [14:11:48] (03PS1) 10Majavah: O:dumps::distribution::server: Remove duplicate profiles [puppet] - 10https://gerrit.wikimedia.org/r/1325488 [14:12:17] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325489 [14:12:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor2003.codfw.wmnet [14:12:59] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus2006.codfw.wmnet [14:13:07] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9220/console" [puppet] - 10https://gerrit.wikimedia.org/r/1325488 (owner: 10Majavah) [14:13:51] FIRING: [10x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:14:03] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1006.eqiad.wmnet [14:15:45] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus1008.eqiad.wmnet [14:16:47] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet [14:16:49] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet [14:16:54] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet [14:16:56] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet [14:17:08] !log cdobbins@cumin1003 conftool action : set/pooled=yes; selector: name=cp5022.* [14:18:28] (03PS1) 10Majavah: P:dumps: Remove unused variable [puppet] - 10https://gerrit.wikimedia.org/r/1325490 (https://phabricator.wikimedia.org/T417824) [14:18:34] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1042.eqiad.wmnet [14:18:51] FIRING: [10x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:19:26] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host debmonitor-dev2001.codfw.wmnet [14:19:37] (03CR) 10Ssingh: [C:03+1] installserver: put all cp hosts on UEFI [puppet] - 10https://gerrit.wikimedia.org/r/1325485 (https://phabricator.wikimedia.org/T414411) (owner: 10CDobbins) [14:20:21] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681#12212493 (10Jhancock.wm) direct quote from support If system stuck at grub boat loader and makes no progress, you can ask Gemini on how to rebuild the grub. They sent a doc w... [14:21:39] jmm@cumin2003 drain-node (PID 1332884) is awaiting input [14:22:02] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12212496 (10BTullis) Now working on an-worker 1199 ` Physical Drives = 14 PD LIST : ======= -----------------------------------------------------------------... [14:22:54] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2006.codfw.wmnet [14:23:14] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor-dev2001.codfw.wmnet [14:23:32] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12212507 (10BTullis) [14:23:47] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1008.eqiad.wmnet [14:23:51] FIRING: [12x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:23:56] !log tappof@cumin1003 START - Cookbook sre.hosts.reboot-single for host prometheus2008.codfw.wmnet [14:24:10] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet [14:24:11] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet [14:24:17] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet [14:24:19] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet [14:24:30] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12212511 (10BTullis) [14:24:40] (03CR) 10JMeybohm: [C:03+1] kube-state-metrics: update to upstream 7.3.0. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319069 (https://phabricator.wikimedia.org/T427405) (owner: 10Blake) [14:25:05] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1216.eqiad.wmnet with OS bookworm [14:25:12] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1217.eqiad.wmnet with OS bookworm [14:25:38] RECOVERY - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1199 is OK: communication: 0 OK : controller: 0 OK : physical_disk: 0 OK : virtual_disk: 0 OK : bbu: 0 OK : enclosure: 0 OK https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [14:25:50] (03PS1) 10Bking: dse-k8s-codfw: Add a new etcd host [puppet] - 10https://gerrit.wikimedia.org/r/1325491 (https://phabricator.wikimedia.org/T434681) [14:26:12] !log sukhe@cumin1003 END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough [14:26:22] !log installing Django security updates [14:26:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:26:49] (03CR) 10Lerickson: [C:03+1] WDQS: Set default query timeout to 300s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325235 (https://phabricator.wikimedia.org/T433391) (owner: 10Trueg) [14:27:00] (03CR) 10Bking: [C:03+2] dse-k8s-codfw: Add a new etcd host [puppet] - 10https://gerrit.wikimedia.org/r/1325491 (https://phabricator.wikimedia.org/T434681) (owner: 10Bking) [14:27:12] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet [14:27:22] (03CR) 10Trueg: [C:03+2] WDQS: Set default query timeout to 300s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325235 (https://phabricator.wikimedia.org/T433391) (owner: 10Trueg) [14:29:39] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1042.eqiad.wmnet [14:29:45] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:29:51] (03Merged) 10jenkins-bot: WDQS: Set default query timeout to 300s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325235 (https://phabricator.wikimedia.org/T433391) (owner: 10Trueg) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1430) [14:31:11] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet [14:31:21] PROBLEM - Host ml-etcd1001 is DOWN: PING CRITICAL - Packet loss = 100% [14:31:34] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Site: CODFW VM etcd for dse-k8s-etcd - https://phabricator.wikimedia.org/T434796 (10bking) 03NEW [14:31:56] !log tappof@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus2008.codfw.wmnet [14:32:03] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet [14:32:05] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet [14:32:06] (03PS1) 10Ssingh: tofurkey: mask tofurkey.service for package [puppet] - 10https://gerrit.wikimedia.org/r/1325492 (https://phabricator.wikimedia.org/T355446) [14:32:10] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet [14:32:12] (03CR) 10JMeybohm: "It's not uncommon for the chart itself to use the latest tag (helps with dev environments etc.). We usually override the tags to use in de" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325451 (https://phabricator.wikimedia.org/T348856) (owner: 10Ladsgroup) [14:32:12] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet [14:32:37] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Site: CODFW VM etcd for dse-k8s-etcd - https://phabricator.wikimedia.org/T434796#12212578 (10bking) [14:32:57] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12212583 (10BTullis) Now working on an-worker1200 ` Physical Drives = 14 PD LIST : ======= ------------------------------------------------------------------... [14:33:06] (03CR) 10Ssingh: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9221/co" [puppet] - 10https://gerrit.wikimedia.org/r/1325492 (https://phabricator.wikimedia.org/T355446) (owner: 10Ssingh) [14:33:12] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12212584 (10BTullis) [14:33:51] FIRING: [10x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:34:47] jouncebot: nowandnext [14:34:47] For the next 0 hour(s) and 25 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1430) [14:34:47] In 0 hour(s) and 25 minute(s): Train log triage (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1500) [14:35:05] (03CR) 10JMeybohm: "This does actually makes sense 😊" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [14:37:48] (03CR) 10Fabfur: [C:03+1] tofurkey: mask tofurkey.service for package [puppet] - 10https://gerrit.wikimedia.org/r/1325492 (https://phabricator.wikimedia.org/T355446) (owner: 10Ssingh) [14:37:55] FIRING: [2x] JobUnavailable: Reduced availability for job ganeti in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:37:57] (03CR) 10Zabe: [C:03+2] Api: Use correct query when continue prop=categories [core] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325455 (https://phabricator.wikimedia.org/T433922) (owner: 10Zabe) [14:38:02] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet [14:38:04] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet [14:38:09] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet [14:38:11] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet [14:38:16] (03CR) 10Ssingh: [V:03+1 C:03+2] tofurkey: mask tofurkey.service for package [puppet] - 10https://gerrit.wikimedia.org/r/1325492 (https://phabricator.wikimedia.org/T355446) (owner: 10Ssingh) [14:39:30] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1216.eqiad.wmnet with reason: host reimage [14:39:32] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Site: CODFW VM etcd for dse-k8s-etcd - https://phabricator.wikimedia.org/T434796#12212609 (10bking) FYI, I'm having trouble running the `sre.ganeti.resource-report` cookbook, stacktrace [[ https://etherpad.wikimedia.org/p/ganeti-resource-report | here ]] [14:39:59] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db1169: Pool db1169.eqiad.wmnet in after cloning [14:40:00] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1217.eqiad.wmnet with reason: host reimage [14:43:47] (03CR) 10JMeybohm: rdb-lock: Create puppet role redis::lock::instance (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324693 (https://phabricator.wikimedia.org/T434188) (owner: 10Clément Goubert) [14:44:16] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1216.eqiad.wmnet with reason: host reimage [14:44:41] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet [14:44:42] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet [14:44:43] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad [14:46:59] (03PS6) 10Scott French: ingress: Introduce istio 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 (https://phabricator.wikimedia.org/T427666) [14:46:59] (03PS6) 10Scott French: mediawiki: Bump ingress.istio from 1.2.0 to 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324344 (https://phabricator.wikimedia.org/T427666) [14:46:59] (03PS3) 10Scott French: mw-pretrain: Introduce dedicated Ingress Gateway for jobrunner [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324780 (https://phabricator.wikimedia.org/T427666) [14:48:00] !log bking@cumin2003 START - Cookbook sre.ganeti.makevm for new host dse-k8s-etcd2004.codfw.wmnet [14:48:02] !log bking@cumin2003 START - Cookbook sre.dns.netbox [14:48:08] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1217.eqiad.wmnet with reason: host reimage [14:48:35] (03CR) 10Scott French: "Thanks for the review!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [14:49:04] (03Merged) 10jenkins-bot: Api: Use correct query when continue prop=categories [core] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325455 (https://phabricator.wikimedia.org/T433922) (owner: 10Zabe) [14:50:00] PROBLEM - Host ganeti1042 is DOWN: PING CRITICAL - Packet loss = 100% [14:50:12] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1325455|Api: Use correct query when continue prop=categories (T433922)]] [14:50:17] T433922: Missing categories from generator=categorymembers (just 1 instead of 2) in itwiki - https://phabricator.wikimedia.org/T433922 [14:51:53] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12212686 (10BTullis) Working on an-worker1201. ` btullis@an-worker1201:~$ sudo perccli64 /c0 show Physical Drives = 14 PD LIST : ======= -------------------... [14:52:18] !log zabe@deploy1003 zabe: Backport for [[gerrit:1325455|Api: Use correct query when continue prop=categories (T433922)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:52:26] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet [14:52:27] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet [14:52:35] !log cgoubert@cumin2003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw [14:52:39] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet [14:52:40] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet [14:53:22] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003" [14:53:27] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003" [14:53:27] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:53:27] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2004.codfw.wmnet on all recursors [14:53:30] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2004.codfw.wmnet on all recursors [14:53:48] !log bking@cumin2003 START - Cookbook sre.dns.netbox [14:53:51] FIRING: [5x] ProbeDown: Service ganeti1042:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:55:25] !log zabe@deploy1003 zabe: Continuing with deployment [14:55:54] (03PS4) 10CDobbins: wikimedia.org: add TXT verification for Mentimeter [dns] - 10https://gerrit.wikimedia.org/r/1324790 (https://phabricator.wikimedia.org/T434620) [14:56:00] (03CR) 10CDobbins: wikimedia.org: add TXT verification for Mentimeter (033 comments) [dns] - 10https://gerrit.wikimedia.org/r/1324790 (https://phabricator.wikimedia.org/T434620) (owner: 10CDobbins) [14:56:32] (03PS1) 10Bking: dse-k8s-codfw: Add SRV records for new etcd node [dns] - 10https://gerrit.wikimedia.org/r/1325498 (https://phabricator.wikimedia.org/T434793) [14:57:32] (03CR) 10CI reject: [V:04-1] dse-k8s-codfw: Add SRV records for new etcd node [dns] - 10https://gerrit.wikimedia.org/r/1325498 (https://phabricator.wikimedia.org/T434793) (owner: 10Bking) [14:58:08] (03CR) 10JMeybohm: [C:03+1] ingress: Introduce istio 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [14:58:10] RECOVERY - Host ganeti1042 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [14:58:10] FIRING: [7x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:38 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:58:46] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003" [14:58:51] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003" [14:58:51] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:58:51] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2004.codfw.wmnet on all recursors [14:58:54] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2004.codfw.wmnet on all recursors [14:59:00] (03PS1) 10Ladsgroup: [WIP] Upgrade to trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1325499 (https://phabricator.wikimedia.org/T419815) [14:59:07] !log bking@cumin2003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dse-k8s-etcd2004.codfw.wmnet [14:59:49] FIRING: [2x] JobUnavailable: Reduced availability for job ganeti in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:59:59] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325455|Api: Use correct query when continue prop=categories (T433922)]] (duration: 09m 47s) [15:00:03] T433922: Missing categories from generator=categorymembers (just 1 instead of 2) in itwiki - https://phabricator.wikimedia.org/T433922 [15:00:05] brennen and jnuche: gettimeofday() says it's time for Train log triage. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1500) [15:00:36] ^ i could have sworn i'd zapped that. [15:00:46] (03CR) 10Klausman: dse-k8s-codfw: Add SRV records for new etcd node (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1325498 (https://phabricator.wikimedia.org/T434793) (owner: 10Bking) [15:00:54] RECOVERY - Host ml-etcd1001 is UP: PING OK - Packet loss = 0%, RTA = 0.81 ms [15:01:25] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1042.eqiad.wmnet [15:01:30] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1042.eqiad.wmnet [15:02:08] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet [15:02:09] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet [15:02:15] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2002.codfw.wmnet [15:02:16] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2002.codfw.wmnet [15:02:54] (03PS2) 10Bking: dse-k8s-codfw: Add SRV records for new etcd node [dns] - 10https://gerrit.wikimedia.org/r/1325498 (https://phabricator.wikimedia.org/T434793) [15:03:17] (03CR) 10Bking: dse-k8s-codfw: Add SRV records for new etcd node (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1325498 (https://phabricator.wikimedia.org/T434793) (owner: 10Bking) [15:03:47] (03PS1) 10CWilliams: mariadb: Productionizing db1280 [puppet] - 10https://gerrit.wikimedia.org/r/1325501 (https://phabricator.wikimedia.org/T407942) [15:03:51] FIRING: [5x] ProbeDown: Service ganeti1042:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:04:11] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1216.eqiad.wmnet with OS bookworm [15:04:51] (03CR) 10Ssingh: [C:03+1] wikimedia.org: add TXT verification for Mentimeter [dns] - 10https://gerrit.wikimedia.org/r/1324790 (https://phabricator.wikimedia.org/T434620) (owner: 10CDobbins) [15:07:57] (03PS2) 10Ladsgroup: [WIP] Upgrade to trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1325499 (https://phabricator.wikimedia.org/T419815) [15:07:58] (03CR) 10Btullis: [C:03+1] "LGTM" [dns] - 10https://gerrit.wikimedia.org/r/1325498 (https://phabricator.wikimedia.org/T434793) (owner: 10Bking) [15:08:00] (03CR) 10Klausman: [C:03+1] dse-k8s-codfw: Add SRV records for new etcd node [dns] - 10https://gerrit.wikimedia.org/r/1325498 (https://phabricator.wikimedia.org/T434793) (owner: 10Bking) [15:09:04] (03CR) 10CDobbins: wikimedia.org: add TXT verification for Mentimeter (032 comments) [dns] - 10https://gerrit.wikimedia.org/r/1324790 (https://phabricator.wikimedia.org/T434620) (owner: 10CDobbins) [15:09:19] (03CR) 10Marostegui: [C:03+1] mariadb: Productionizing db1280 [puppet] - 10https://gerrit.wikimedia.org/r/1325501 (https://phabricator.wikimedia.org/T407942) (owner: 10CWilliams) [15:09:30] (03PS1) 10Gerrit maintenance bot: mariadb: Promote db2159 to s7 master [puppet] - 10https://gerrit.wikimedia.org/r/1325502 (https://phabricator.wikimedia.org/T434802) [15:09:42] (03CR) 10CWilliams: [C:03+2] mariadb: Productionizing db1280 [puppet] - 10https://gerrit.wikimedia.org/r/1325501 (https://phabricator.wikimedia.org/T407942) (owner: 10CWilliams) [15:09:59] !log bking@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: quash java safepoint logspam - bking@cumin2003 - T434685 [15:10:03] T434685: CirrusSearch: stop pushing Java safepoint logs to syslog pipeline - https://phabricator.wikimedia.org/T434685 [15:10:11] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2002.codfw.wmnet [15:10:12] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2002.codfw.wmnet [15:10:17] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2003.codfw.wmnet [15:10:18] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2003.codfw.wmnet [15:10:28] (03CR) 10CDobbins: [C:03+2] wikimedia.org: add TXT verification for Mentimeter [dns] - 10https://gerrit.wikimedia.org/r/1324790 (https://phabricator.wikimedia.org/T434620) (owner: 10CDobbins) [15:10:32] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1217.eqiad.wmnet with OS bookworm [15:11:24] (03CR) 10Scott French: "Thanks, Blake!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324780 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:13:35] (03CR) 10Scott French: "Done in I59fd57f335c55438d88c978cd6b8f5bd6a6a6964. Thanks for the reminder!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324344 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:13:39] !log cdobbins@dns1004 START - running authdns-update [15:15:40] !log cdobbins@dns1004 END - running authdns-update [15:17:03] 06SRE, 10DNS, 06Traffic, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Mentimeter - https://phabricator.wikimedia.org/T434620#12212808 (10CDobbins) I just updated the DNS records, @bcampbell. Let me know if there's any unexpected behavior or if I can close the t... [15:17:11] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2003.codfw.wmnet [15:17:12] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2003.codfw.wmnet [15:17:18] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2004.codfw.wmnet [15:17:19] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2004.codfw.wmnet [15:17:21] (03CR) 10Scott French: wmnet: Add mw-pretrain-jobrunner k8s Ingress CNAMEs (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1324776 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:18:10] FIRING: [5x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:38 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:18:21] !log bking@ganeti2048 sudo gnt-node failover -f ganeti2046.codfw.wmnet T434681 [15:18:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:18:24] T434681: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681 [15:19:40] FIRING: KubernetesAPINotScrapable: k8s-aux@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [15:20:44] (03PS28) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [15:21:22] (03CR) 10Federico Ceratto: [C:03+2] mysql.multiinstance_reboot_test: switch to mocker [cookbooks] - 10https://gerrit.wikimedia.org/r/1314730 (owner: 10Federico Ceratto) [15:22:58] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681#12212845 (10bking) a:03bking [15:23:42] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2004.codfw.wmnet [15:23:43] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2004.codfw.wmnet [15:23:45] (03CR) 10CI reject: [V:04-1] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [15:23:49] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2005.codfw.wmnet [15:23:50] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2005.codfw.wmnet [15:24:21] (03Merged) 10jenkins-bot: mysql.multiinstance_reboot_test: switch to mocker [cookbooks] - 10https://gerrit.wikimedia.org/r/1314730 (owner: 10Federico Ceratto) [15:24:40] RESOLVED: KubernetesAPINotScrapable: k8s-aux@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable [15:25:12] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1169: Pool db1169.eqiad.wmnet in after cloning [15:25:21] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1169.eqiad.wmnet onto db1277.eqiad.wmnet [15:26:29] (03CR) 10Scott French: "Thanks for the review!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324775 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:26:34] (03CR) 10Blake: [C:03+1] mw-pretrain: Introduce dedicated Ingress Gateway for jobrunner [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324780 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:26:34] (03CR) 10Scott French: [C:03+2] admin_ng: Add mw-pretrain-jobrunner to mw-pretrain tlsExtraSANs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324775 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:26:43] (03CR) 10Blake: [C:03+1] wmnet: Add mw-pretrain-jobrunner k8s Ingress CNAMEs [dns] - 10https://gerrit.wikimedia.org/r/1324776 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:28:08] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681#12212869 (10bking) Hello all, sorry for grabbing this but the state of this host is blocking VM creation. I'm following [[ https://wikitech.wikimedia.org/wiki/Ganeti#Failed_ha... [15:28:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:29:09] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806 (10cmooney) 03NEW p:05Triage→03Low [15:29:26] inflatador: you may want to ask moritzm about the ganeti node before doing stuff [15:29:38] it's got four etcd vms on it that are not redundant [15:30:11] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2005.codfw.wmnet [15:30:12] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2005.codfw.wmnet [15:30:12] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-codfw [15:30:57] 10ops-codfw, 06SRE, 10bacula, 06Data-Persistence, and 3 others: decommission backup2003.codfw.wmnet - https://phabricator.wikimedia.org/T433971#12212889 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [15:31:52] jouncebot: nowandnext [15:31:52] For the next 0 hour(s) and 28 minute(s): Train log triage (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1500) [15:31:53] In 0 hour(s) and 28 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1600) [15:32:06] !log cgoubert@deploy1003 Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot [15:32:48] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet [15:32:52] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12212903 (10cmooney) I should say this has no particular production impact, BGP establishes and everything works ok. It's only if BFD was already in... [15:33:43] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12212931 (10cmooney) [15:35:20] (03CR) 10Scott French: "Thanks, Blake!" [dns] - 10https://gerrit.wikimedia.org/r/1324776 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:35:21] (03CR) 10Ahmon Dancy: "Confirmed that unless you use --scope=pretrain, you should not get warnings with this configuration." [puppet] - 10https://gerrit.wikimedia.org/r/1324826 (https://phabricator.wikimedia.org/T428972) (owner: 10Scott French) [15:35:22] (03CR) 10Scott French: [C:03+2] wmnet: Add mw-pretrain-jobrunner k8s Ingress CNAMEs [dns] - 10https://gerrit.wikimedia.org/r/1324776 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:35:33] !log swfrench@dns1004 START - running authdns-update [15:36:20] (03PS2) 10Gerrit maintenance bot: mariadb: Promote db2159 to s7 master [puppet] - 10https://gerrit.wikimedia.org/r/1325502 (https://phabricator.wikimedia.org/T434802) [15:36:37] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet [15:37:20] (03Merged) 10jenkins-bot: admin_ng: Add mw-pretrain-jobrunner to mw-pretrain tlsExtraSANs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324775 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [15:37:30] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet [15:37:43] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 30 hosts with reason: Primary switchover s7 T434802 [15:37:47] T434802: Switchover s7 master (db2220 -> db2159) - https://phabricator.wikimedia.org/T434802 [15:37:48] !log swfrench@dns1004 END - running authdns-update [15:38:07] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Set db2159 with weight 0 T434802', diff saved to https://phabricator.wikimedia.org/P96077 and previous config saved to /var/cache/conftool/dbconfig/20260813-153806-cwilliams.json [15:38:10] FIRING: [3x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:38 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:39:14] !log swfrench@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [15:39:38] RECOVERY - Host logstash2023 is UP: PING OK - Packet loss = 0%, RTA = 30.71 ms [15:39:38] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [15:39:51] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [15:39:53] !log bking@ganeti2048] ~$ sudo gnt-node failover -f --ignore-consistency ganeti2046.codfw.wmnet T434681 [15:39:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:39:58] T434681: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681 [15:40:33] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681#12212963 (10bking) Per the etherpad, ganeti won't failover 4 of its VMs. 3 of the 4 VMs are not backed by DRBD so that is expected. The fourth, `logstash2023.codfw.wmnet` , is b... [15:41:30] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet [15:41:40] (03CR) 10CWilliams: [C:03+2] mariadb: Promote db2159 to s7 master [puppet] - 10https://gerrit.wikimedia.org/r/1325502 (https://phabricator.wikimedia.org/T434802) (owner: 10Gerrit maintenance bot) [15:41:49] !log cgoubert@deploy1003 Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 09m 42s) [15:42:53] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12212971 (10cmooney) [15:43:22] !log Starting s7 codfw failover from db2220 to db2159 - T434802 [15:43:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:43:27] T434802: Switchover s7 master (db2220 -> db2159) - https://phabricator.wikimedia.org/T434802 [15:43:51] FIRING: [4x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:44:07] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Promote db2159 to s7 primary T434802', diff saved to https://phabricator.wikimedia.org/P96078 and previous config saved to /var/cache/conftool/dbconfig/20260813-154405-cwilliams.json [15:44:39] !log depooling dns3004 to reimage to trixie [15:44:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:45:22] !log cdobbins@cumin1003 conftool action : set/pooled=no; selector: name=dns3004.* [15:46:26] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depool db2220 T434802', diff saved to https://phabricator.wikimedia.org/P96079 and previous config saved to /var/cache/conftool/dbconfig/20260813-154624-cwilliams.json [15:46:43] !log cdobbins@cumin1003 START - Cookbook sre.hosts.reimage for host dns3004.wikimedia.org with OS trixie [15:48:10] RESOLVED: [3x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:38 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:48:44] !log cwilliams@cumin1003 START - Cookbook sre.mysql.upgrade for 1 hosts [15:48:50] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 2 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [15:48:53] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db2220: Upgrading db2220.codfw.wmnet [15:48:57] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2220: Upgrading db2220.codfw.wmnet [15:50:23] PROBLEM - BFD status on asw1-bw27-esams.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [15:53:09] FIRING: HelmReleaseBadStatus: Helm release kube-system/calico on k8s-staging@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-staging&var-namespace=kube-system - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [15:53:40] FIRING: [5x] BFDdown: BFD session down between asw1-bw27-esams and 185.15.59.2 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:54:45] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@esams - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:56:35] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2220: Upgrade of db2220.codfw.wmnet completed [15:59:58] !log swfrench@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [16:00:05] jhathaway and rzl: Time to snap out of that daydream and deploy Puppet request window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1600). [16:00:05] No Gerrit patches in the queue for this window AFAICS. [16:00:17] !log swfrench@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [16:02:39] rzl/jhathaway: I have an unscheduled item: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324278 [16:04:39] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [16:05:00] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [16:07:45] !log cdobbins@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on dns3004.wikimedia.org with reason: host reimage [16:07:53] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [16:08:06] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [16:14:12] !log cdobbins@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns3004.wikimedia.org with reason: host reimage [16:17:00] PROBLEM - Recursive DNS on 185.15.59.2 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [16:20:14] !log bking@cumin2003 END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: quash java safepoint logspam - bking@cumin2003 - T434685 [16:20:18] T434685: CirrusSearch: stop pushing Java safepoint logs to syslog pipeline - https://phabricator.wikimedia.org/T434685 [16:22:10] !log dancy@deploy1003 Installing scap version "4.280.0" for 3 host(s) [16:23:00] PROBLEM - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2086 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:24:04] PROBLEM - Check unit status of push_cross_cluster_settings_9600 on cirrussearch2108 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9600 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:24:05] !log dancy@deploy1003 Installation of scap version "4.280.0" completed for 3 hosts [16:27:55] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@esams - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:31:40] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host an-worker1150.eqiad.wmnet with OS bookworm [16:32:01] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host an-worker1150 [16:32:02] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1150 [16:33:00] RECOVERY - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2086 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:33:47] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host an-worker1179.eqiad.wmnet with OS bookworm [16:34:04] RECOVERY - Check unit status of push_cross_cluster_settings_9600 on cirrussearch2108 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9600 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:34:09] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host an-worker1179 [16:34:09] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1179 [16:34:30] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host an-worker1218.eqiad.wmnet with OS bookworm [16:34:51] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host an-worker1218 [16:34:52] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1218 [16:37:00] PROBLEM - Recursive DNS on 2a02:ec80:300:1:185:15:59:2 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [16:38:06] !log kubectl delete node kubestagemaster2005.codfw.wmnet - T434681 [16:38:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:38:10] T434681: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681 [16:38:13] !log swfrench@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [16:41:23] RECOVERY - BFD status on asw1-bw27-esams.mgmt is OK: UP: 2 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [16:41:48] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2220: Upgrade of db2220.codfw.wmnet completed [16:41:48] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for 1 hosts [16:43:09] RESOLVED: HelmReleaseBadStatus: Helm release kube-system/calico on k8s-staging@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-staging&var-namespace=kube-system - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [16:43:40] RESOLVED: [2x] BFDdown: BFD session down between asw1-bw27-esams and 185.15.59.2 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-bw27-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:47:43] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1150.eqiad.wmnet with reason: host reimage [16:48:59] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1179.eqiad.wmnet with reason: host reimage [16:49:04] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1218.eqiad.wmnet with reason: host reimage [16:50:38] !log cwilliams@cumin1003 START - Cookbook sre.mysql.clone of db1166.eqiad.wmnet onto db1280.eqiad.wmnet [16:50:42] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db1166: Depool db1166.eqiad.wmnet to then clone it to db1280.eqiad.wmnet - cwilliams@cumin1003 [16:51:57] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1166: Depool db1166.eqiad.wmnet to then clone it to db1280.eqiad.wmnet - cwilliams@cumin1003 [16:53:56] !log dancy@deploy1003 Started scap sync-world: Backport for [[gerrit:1324810|deployment-info.php: Report dbname and branch for the requested wiki (T434726)]] [16:54:01] T434726: Where is my change running tool - https://phabricator.wikimedia.org/T434726 [16:54:27] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1150.eqiad.wmnet with reason: host reimage [16:56:11] !log dancy@deploy1003 dancy: Backport for [[gerrit:1324810|deployment-info.php: Report dbname and branch for the requested wiki (T434726)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:56:39] !log dancy@deploy1003 dancy: Continuing with deployment [16:57:34] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1179.eqiad.wmnet with reason: host reimage [16:58:02] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [16:59:47] wikibugs is kill -9ed [16:59:51] can someone bring it back up? [17:00:05] bd808: Time to snap out of that daydream and deploy Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1700). [17:00:05] swfrench-wmf: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for MediaWiki infrastructure (UTC late). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1700). [17:00:21] o/ [17:00:48] !log dancy@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324810|deployment-info.php: Report dbname and branch for the requested wiki (T434726)]] (duration: 06m 52s) [17:00:52] T434726: Where is my change running tool - https://phabricator.wikimedia.org/T434726 [17:00:58] dancy: any further backport changes? [17:01:04] I'm done [17:01:09] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [17:01:12] * swfrench-wmf thumbs up [17:01:34] !log jhancock@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [17:01:42] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1218.eqiad.wmnet with reason: host reimage [17:01:57] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [17:02:18] !log jhancock@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [17:02:39] getting started on the infra window ... waiting on CI, then deploying mediawiki [17:02:48] Nothing for my window today, but I am going to us the time to do a code review that his been lingering. [17:07:44] ... and waiting on chart-museum ... [17:09:11] !log swfrench@deploy1003 Started scap sync-world: Helmfile-only deployment for mediawiki chart bump - T427666 [17:09:15] T427666: Route testwiki traffic to the Pretrain MVP environment - https://phabricator.wikimedia.org/T427666 [17:09:45] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:09:57] RECOVERY - Recursive DNS on 185.15.59.2 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [17:09:57] RECOVERY - Recursive DNS on 2a02:ec80:300:1:185:15:59:2 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [17:10:24] (03CR) 10Scott French: [C:03+2] mw-pretrain: Introduce dedicated Ingress Gateway for jobrunner [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324780 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [17:12:14] !log swfrench@deploy1003 Finished scap sync-world: Helmfile-only deployment for mediawiki chart bump - T427666 (duration: 03m 03s) [17:12:29] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [17:12:33] (03Merged) 10jenkins-bot: mw-pretrain: Introduce dedicated Ingress Gateway for jobrunner [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324780 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [17:12:55] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:13:55] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [17:14:14] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [17:14:36] (03PS1) 10Ssingh: P:pybal: remove references to decommed hosts lvs10[13-15] [puppet] - 10https://gerrit.wikimedia.org/r/1325533 (https://phabricator.wikimedia.org/T433508) [17:14:42] (03CR) 10Bking: [C:04-1] "Looks like we might be able to get away with manually replacing the existing hostname, let's hold off on merging this until we're sure." [dns] - 10https://gerrit.wikimedia.org/r/1325498 (https://phabricator.wikimedia.org/T434793) (owner: 10Bking) [17:15:26] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681#12213422 (10bking) Per conversation in IRC, I have approval to move forward, so I'm starting this work now. [17:15:55] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [17:18:15] !log cdobbins@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns3004.wikimedia.org with OS trixie [17:18:34] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1150.eqiad.wmnet with OS bookworm [17:19:18] (03PS1) 10Pppery: Update translations [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1325534 [17:19:18] (03PS1) 10Pppery: Add two new locale files [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1325535 (https://phabricator.wikimedia.org/T412651) [17:19:21] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1179.eqiad.wmnet with OS bookworm [17:19:30] (03PS2) 10Ssingh: P:pybal: remove references to decommed hosts lvs10[13-16] [puppet] - 10https://gerrit.wikimedia.org/r/1325533 (https://phabricator.wikimedia.org/T433508) [17:20:29] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [17:20:44] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [17:21:37] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1218.eqiad.wmnet with OS bookworm [17:23:53] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [17:24:05] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [17:26:28] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [17:26:39] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [17:28:00] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [17:28:12] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12213457 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS t... [17:32:25] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [17:32:36] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [17:32:50] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, and 2 others: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12213469 (10ssingh) [17:33:49] (03CR) 10BCornwall: [C:03+1] P:pybal: remove references to decommed hosts lvs10[13-16] [puppet] - 10https://gerrit.wikimedia.org/r/1325533 (https://phabricator.wikimedia.org/T433508) (owner: 10Ssingh) [17:34:03] (03CR) 10Ssingh: [C:03+2] P:pybal: remove references to decommed hosts lvs10[13-16] [puppet] - 10https://gerrit.wikimedia.org/r/1325533 (https://phabricator.wikimedia.org/T433508) (owner: 10Ssingh) [17:36:27] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:37:35] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, and 2 others: decommission lvs10[13-16] - https://phabricator.wikimedia.org/T433508#12213505 (10ssingh) [17:38:57] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:39:13] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:40:34] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:40:42] !log bking@ganeti2048] `sudo gnt-instance remove --force --ignore-failures --shutdown-timeout=0` on non-DRBD VMs T434681 [17:40:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:40:46] T434681: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681 [17:40:55] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:41:32] !log bking@cumin2003 START - Cookbook sre.ganeti.makevm for new host dse-k8s-etcd2001.codfw.wmnet [17:41:33] !log bking@cumin2003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dse-k8s-etcd2001.codfw.wmnet [17:42:30] FIRING: Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-bw27-esams.mgmt.esams.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [17:43:29] ^ yeah, dns3004 [17:43:33] will be pooled shortly [17:43:50] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.admin rebooting P{lvs7003*} and A:liberica [17:44:56] (03PS1) 10Scott French: mw-pretrain: Inherit VirtualService route hosts from Gateway hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325538 (https://phabricator.wikimedia.org/T427666) [17:45:03] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host an-worker1220.eqiad.wmnet with OS bookworm [17:45:32] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host an-worker1220 [17:45:38] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host an-worker1221.eqiad.wmnet with OS bookworm [17:46:04] !log ryankemper@cumin2003 START - Cookbook sre.dns.netbox [17:46:19] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host an-worker1234.eqiad.wmnet with OS bookworm [17:47:25] !log sukhe@cumin1003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7003*} and A:liberica [17:47:54] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:48:30] (03CR) 10Scott French: [C:03+2] mw-pretrain: Inherit VirtualService route hosts from Gateway hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325538 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [17:50:35] !log ryankemper@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1220 - ryankemper@cumin2003" [17:50:46] (03Merged) 10jenkins-bot: mw-pretrain: Inherit VirtualService route hosts from Gateway hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325538 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [17:50:59] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1220 - ryankemper@cumin2003" [17:50:59] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:50:59] !log ryankemper@cumin2003 START - Cookbook sre.dns.wipe-cache an-worker1220.eqiad.wmnet 11.36.64.10.in-addr.arpa 1.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [17:51:02] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1220.eqiad.wmnet 11.36.64.10.in-addr.arpa 1.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [17:51:03] !log ryankemper@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host an-worker1220 [17:51:13] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [17:51:25] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [17:51:42] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1220 [17:51:42] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1220 [17:51:57] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host an-worker1221 [17:52:14] !log ryankemper@cumin2003 START - Cookbook sre.dns.netbox [17:52:39] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply [17:53:18] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply [17:53:52] (03CR) 10AOkoth: "Great. Thank you." [puppet] - 10https://gerrit.wikimedia.org/r/1321581 (https://phabricator.wikimedia.org/T377889) (owner: 10AOkoth) [17:54:16] alright ... that was underwhelming :) in any case, I believe I'm done with things that will fit in the infra window [17:56:22] (03CR) 10Subramanya Sastry: [C:03+1] Add Produnto to extension-list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324963 (https://phabricator.wikimedia.org/T421436) (owner: 10Tim Starling) [17:56:29] !log ryankemper@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1221 - ryankemper@cumin2003" [17:57:11] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [17:57:16] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [17:58:14] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [17:58:55] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [17:59:05] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [17:59:34] ryankemper@cumin2003 reimage (PID 1383382) is awaiting input [18:00:05] brennen and jnuche: I, the Bot under the Fountain, call upon thee, The Deployer, to do MediaWiki train - Utc-7+Utc-0 Version deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1800). [18:00:27] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1221 - ryankemper@cumin2003" [18:00:27] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:00:27] !log ryankemper@cumin2003 START - Cookbook sre.dns.wipe-cache an-worker1221.eqiad.wmnet 18.36.64.10.in-addr.arpa 8.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:00:27] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1221.eqiad.wmnet 18.36.64.10.in-addr.arpa 8.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:00:27] !log ryankemper@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host an-worker1221 [18:00:48] o/ [18:01:26] brennen: Can I squeeze in a scap update? [18:01:28] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1221 [18:01:28] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1221 [18:01:36] dancy: yeah, go to town [18:01:40] !log dancy@deploy1003 Installing scap version "4.280.1" for 3 host(s) [18:01:42] Thanks! [18:01:46] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host an-worker1234 [18:01:52] !log ryankemper@cumin2003 START - Cookbook sre.dns.netbox [18:01:57] (03PS4) 10Bernard Wang: Transition reading list experiment to instrument [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1259251 (https://phabricator.wikimedia.org/T421939) (owner: 10LorenMora) [18:02:02] (03PS5) 10Bernard Wang: Transition reading list experiment to instrument [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1259251 (https://phabricator.wikimedia.org/T421939) (owner: 10LorenMora) [18:02:06] taking a couple minutes to look over logs before i roll anyhow [18:02:35] !log bking@dse-k8s-etcd2002 etcdctl member remove ${UUID of dse-k8s-etcd2001} T434681 T434793 [18:02:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:02:42] T434681: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681 [18:02:43] T434793: Replace dse-k8s-etcd2001 which was only available on ganeti2046 - https://phabricator.wikimedia.org/T434793 [18:02:52] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [18:02:57] (03CR) 10Subramanya Sastry: [C:03+1] Enable Produnto on Beta (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324964 (https://phabricator.wikimedia.org/T421436) (owner: 10Tim Starling) [18:03:40] !log dancy@deploy1003 Installation of scap version "4.280.1" completed for 3 hosts [18:03:48] brennen: Done [18:04:05] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [18:04:47] !log bking@cumin2003 START - Cookbook sre.ganeti.makevm for new host dse-k8s-etcd2001.codfw.wmnet [18:05:46] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1220.eqiad.wmnet with reason: host reimage [18:06:51] dancy: thx [18:07:05] !log ryankemper@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1234 - ryankemper@cumin2003" [18:07:08] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1234 - ryankemper@cumin2003" [18:07:09] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:07:09] !log ryankemper@cumin2003 START - Cookbook sre.dns.wipe-cache an-worker1234.eqiad.wmnet 10.36.64.10.in-addr.arpa 0.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:07:09] !log bking@cumin2003 START - Cookbook sre.dns.netbox [18:07:12] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1234.eqiad.wmnet 10.36.64.10.in-addr.arpa 0.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:07:12] !log ryankemper@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host an-worker1234 [18:07:28] !log 1.47.0-wmf.15 train status (T430834) - no current blockers, rolling to all wikis [18:07:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:07:33] T430834: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834 [18:08:11] (03PS1) 10TrainBranchBot: group2 to 1.47.0-wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325543 (https://phabricator.wikimedia.org/T430834) [18:08:14] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by brennen@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325543 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [18:09:14] (03Merged) 10jenkins-bot: group2 to 1.47.0-wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325543 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [18:09:17] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1220.eqiad.wmnet with reason: host reimage [18:09:40] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1234 [18:09:40] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1234 [18:10:33] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [18:10:34] !log bking@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: quash java safepoint logspam - bking@cumin2003 - T434685 [18:10:39] T434685: CirrusSearch: stop pushing Java safepoint logs to syslog pipeline - https://phabricator.wikimedia.org/T434685 [18:12:26] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" [18:12:30] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" [18:12:30] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:12:30] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2001.codfw.wmnet on all recursors [18:12:33] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2001.codfw.wmnet on all recursors [18:12:36] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [18:12:52] !log bking@cumin2003 START - Cookbook sre.dns.netbox [18:14:17] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [18:15:18] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1221.eqiad.wmnet with reason: host reimage [18:15:32] !log brennen@deploy1003 rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.15 refs T430834 [18:15:36] T430834: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834 [18:17:42] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db1166: Pool db1166.eqiad.wmnet in after cloning [18:19:00] bking@cumin2003 makevm (PID 1388724) is awaiting input [18:19:20] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1221.eqiad.wmnet with reason: host reimage [18:25:20] !log jhancock@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie [18:25:36] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12213746 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-ctrl2006.co... [18:25:38] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1234.eqiad.wmnet with reason: host reimage [18:27:20] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" [18:27:24] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003" [18:27:24] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:27:25] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2001.codfw.wmnet on all recursors [18:27:28] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2001.codfw.wmnet on all recursors [18:27:36] !log bking@cumin2003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dse-k8s-etcd2001.codfw.wmnet [18:29:06] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1234.eqiad.wmnet with reason: host reimage [18:29:28] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681#12213759 (10bking) I still can't remove `ganeti2046` from the cluster until I reassign its secondaries: ` sudo gnt-instance list -o name,pnode,snodes | grep ganeti2046.codfw.wm... [18:29:59] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1220.eqiad.wmnet with OS bookworm [18:33:10] jhancock@cumin1003 reimage (PID 3730719) is awaiting input [18:33:46] (03PS1) 10Bearloga: EventStreamConfig: Remove Watchlist click stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325546 (https://phabricator.wikimedia.org/T434790) [18:34:16] (03CR) 10Ssingh: "PCC NOOP" [puppet] - 10https://gerrit.wikimedia.org/r/1324380 (owner: 10BBlack) [18:35:07] vriley@cumin1003 reimage (PID 3689104) is awaiting input [18:35:28] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie [18:35:41] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12213798 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS trixi... [18:36:52] !log [bking@ganeti2048] ~$ sudo gnt-instance replace-disks -n ganeti2030.codfw.wmnet aux-k8s-worker2002.codfw.wmnet T434681 [18:36:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:36:59] T434681: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681 [18:39:13] (03CR) 10Ssingh: "modules/profile/files/cache/update_ocsp_haproxy_hook.sh can also be removed I think." [puppet] - 10https://gerrit.wikimedia.org/r/1324380 (owner: 10BBlack) [18:39:53] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1221.eqiad.wmnet with OS bookworm [18:40:16] (03CR) 10Phuedx: [C:03+1] EventStreamConfig: Remove Watchlist click stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325546 (https://phabricator.wikimedia.org/T434790) (owner: 10Bearloga) [18:44:20] !log cdobbins@cumin1003 conftool action : set/pooled=yes; selector: name=dns3004.* [18:46:17] !log jhancock@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS trixie [18:46:35] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12213836 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-ctrl2006.codfw.... [18:46:51] !log jhancock@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie [18:46:59] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [18:47:05] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12213840 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-ctrl2006.co... [18:47:30] RESOLVED: Not accepting/receiving prefixes from anycast BGP peer: Device asw1-bw27-esams.mgmt.esams.wmnet recovered from Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [18:47:41] nice [18:49:35] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [18:51:20] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [18:52:00] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [18:52:23] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12213849 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS t... [18:58:18] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1234.eqiad.wmnet with OS bookworm [18:58:47] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Create Kerberos identity for Randall Scout - https://phabricator.wikimedia.org/T430598#12213865 (10Rscout) 05Open→03Invalid [18:59:15] !log jhancock@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage [19:00:59] (03PS1) 10Ladsgroup: Enable desktop lazy loading on group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325548 (https://phabricator.wikimedia.org/T148047) [19:01:27] (03PS2) 10Ladsgroup: Enable image lazy loading on desktop in group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325548 (https://phabricator.wikimedia.org/T148047) [19:01:56] jouncebot: nowandnext [19:01:56] For the next 0 hour(s) and 58 minute(s): MediaWiki train - Utc-7+Utc-0 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T1800) [19:01:56] In 0 hour(s) and 58 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T2000) [19:02:55] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1166: Pool db1166.eqiad.wmnet in after cloning [19:03:04] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage [19:04:01] wmf.15 is deployed, so we are free to some backports now, yeah? [19:04:06] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1166.eqiad.wmnet onto db1280.eqiad.wmnet [19:05:19] I'm doing a backport right now [19:05:26] but after I'm done, I think it's fine [19:05:33] okay thanks [19:05:37] (03PS1) 10Arlolra: Deploy PRV to several LC wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325549 (https://phabricator.wikimedia.org/T423785) [19:05:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325548 (https://phabricator.wikimedia.org/T148047) (owner: 10Ladsgroup) [19:06:53] (03Merged) 10jenkins-bot: Enable image lazy loading on desktop in group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325548 (https://phabricator.wikimedia.org/T148047) (owner: 10Ladsgroup) [19:07:06] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1325548|Enable image lazy loading on desktop in group1 (T148047)]] [19:07:13] T148047: Enable native image lazy loading on desktop - https://phabricator.wikimedia.org/T148047 [19:08:05] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, August 13 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325549 (https://phabricator.wikimedia.org/T423785) (owner: 10Arlolra) [19:10:16] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1325548|Enable image lazy loading on desktop in group1 (T148047)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:10:42] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [19:14:34] (03CR) 10Subramanya Sastry: [C:03+1] Deploy PRV to several LC wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325549 (https://phabricator.wikimedia.org/T423785) (owner: 10Arlolra) [19:14:49] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325548|Enable image lazy loading on desktop in group1 (T148047)]] (duration: 07m 43s) [19:14:54] T148047: Enable native image lazy loading on desktop - https://phabricator.wikimedia.org/T148047 [19:17:02] vriley@cumin1003 reimage (PID 3750009) is awaiting input [19:18:19] (03PS1) 10MusikAnimal: Render the focused module view as a full-screen page [extensions/PersonalDashboard] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325551 (https://phabricator.wikimedia.org/T433896) [19:18:54] !log jhancock@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" [19:20:36] !log jhancock@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" [19:20:38] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl2006.codfw.wmnet with OS trixie [19:20:52] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12213928 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-ctrl2006.codfw.... [19:20:52] FIRING: [2x] GitLabReplicaDataStale: GitLab - replica gitlab1003:0 serves data older than 30h - https://wikitech.wikimedia.org/wiki/GitLab/Backup_and_Restore - https://grafana.wikimedia.org/d/R_1IvBZnz/gitlab-omnibus-overview - https://alerts.wikimedia.org/?q=alertname%3DGitLabReplicaDataStale [19:21:18] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12213935 (10Jhancock.wm) so that first one failed because the firmware upgrade screwed up the boot order. the network connection was... [19:21:48] PROBLEM - Check unit status of push_cross_cluster_settings_9600 on cirrussearch1073 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9600 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:22:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by musikanimal@deploy1003 using scap backport" [extensions/PersonalDashboard] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325551 (https://phabricator.wikimedia.org/T433896) (owner: 10MusikAnimal) [19:24:07] (03Merged) 10jenkins-bot: Render the focused module view as a full-screen page [extensions/PersonalDashboard] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325551 (https://phabricator.wikimedia.org/T433896) (owner: 10MusikAnimal) [19:24:20] !log musikanimal@deploy1003 Started scap sync-world: Backport for [[gerrit:1325551|Render the focused module view as a full-screen page (T433896)]] [19:24:24] T433896: Render the focused module view as a full-screen page - https://phabricator.wikimedia.org/T433896 [19:28:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:34:00] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie [19:34:11] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12213976 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS trixi... [19:34:13] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [19:34:28] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12213981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS t... [19:34:33] !log bking@cumin2003 END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: quash java safepoint logspam - bking@cumin2003 - T434685 [19:34:41] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, August 13 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1320163 (https://phabricator.wikimedia.org/T428147) (owner: 10Milazg) [19:34:42] T434685: CirrusSearch: stop pushing Java safepoint logs to syslog pipeline - https://phabricator.wikimedia.org/T434685 [19:38:30] (03PS1) 10DLynch: InstrumentConstructiveEdits: exclude mw-reverted as well [extensions/WikimediaEvents] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325553 (https://phabricator.wikimedia.org/T431493) [19:38:41] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, August 13 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1325553 (https://phabricator.wikimedia.org/T431493) (owner: 10DLynch) [19:39:34] (03CR) 10Eevans: [C:03+2] aqs: canary Cassandra 5.0.8 upgrade [puppet] - 10https://gerrit.wikimedia.org/r/1324812 (https://phabricator.wikimedia.org/T433026) (owner: 10Eevans) [19:41:25] !log musikanimal@deploy1003 musikanimal: Backport for [[gerrit:1325551|Render the focused module view as a full-screen page (T433896)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:41:30] T433896: Render the focused module view as a full-screen page - https://phabricator.wikimedia.org/T433896 [19:41:48] RECOVERY - Check unit status of push_cross_cluster_settings_9600 on cirrussearch1073 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9600 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:42:29] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12214024 (10VRiley-WMF) zuul1004, zuul1005 and zuul1007 have been completed. However, on zuul1006, I'm getting this error. {F98339657} I eve... [19:42:37] !log musikanimal@deploy1003 musikanimal: Continuing with deployment [19:44:06] FIRING: [2x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:44:31] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching aqs[2001,1016]*: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003 [19:44:35] T433028: Upgrade restbase cluster to Cassandra 5.0.8 - https://phabricator.wikimedia.org/T433028 [19:52:45] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs[2001,1016]*: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003 [19:52:50] T433028: Upgrade restbase cluster to Cassandra 5.0.8 - https://phabricator.wikimedia.org/T433028 [19:54:57] !log musikanimal@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325551|Render the focused module view as a full-screen page (T433896)]] (duration: 30m 37s) [19:55:02] T433896: Render the focused module view as a full-screen page - https://phabricator.wikimedia.org/T433896 [19:56:19] (03PS2) 10Arlolra: Deploy PRV to several LC wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325549 (https://phabricator.wikimedia.org/T423785) [19:58:20] !log robh@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs2001.codfw.wmnet with reason: updating firmware [19:58:34] 10ops-codfw, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Noisy alerts: investigate fstrim.service failure on dse-k8s-wdqs2001 (new Supermicro host) - https://phabricator.wikimedia.org/T434299#12214068 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0525ce6b-a5e4-405d... [19:59:44] (03CR) 10Subramanya Sastry: [C:03+1] Deploy PRV to several LC wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325549 (https://phabricator.wikimedia.org/T423785) (owner: 10Arlolra) [20:00:04] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: Time to do the UTC late backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260813T2000). [20:00:04] apergos, arlolra, obvious-dust, and kemayo: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:13] o/ [20:00:15] o/ [20:00:20] o/ [20:01:09] 06SRE, 10SRE-Access-Requests, 10SRE-tools, 10Cumin, and 2 others: add dcops group to run sre.hosts.downtime cookbook - https://phabricator.wikimedia.org/T433409#12214073 (10Jhancock.wm) 05Resolved→03Open Sorry to reopen but i don't seem to be able to run downtime cookbooks. it prompts me for a sudo pas... [20:01:59] o/ [20:02:13] who's running the window? I cannot, I'm in a meeting at the same time so already with split attention [20:02:47] feel free to self-deploy per queue for those who can - i'm happy to deploy for those who need a deployer [20:03:24] apergos: do you want me to deploy for you? [20:03:27] I think I'm first up [20:03:37] I'm already on the host so ok to do it [20:03:45] cool beans [20:03:48] * TheresNoTime is also around if a deployer is needed [20:04:15] * cjming bows to TheresNoTime [20:04:22] I also have no problem deploying for myself. [20:05:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ariel@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319906 (https://phabricator.wikimedia.org/T433736) (owner: 10ArielGlenn)