[00:00:39] FIRING: TransitBGPDown: Transit BGP session down between cr1-magru and Ufinet (187.108.235.25) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Transit4&var-bgp_neighbor=Ufinet - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [00:05:39] RESOLVED: TransitBGPDown: Transit BGP session down between cr1-magru and Ufinet (187.108.235.25) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Transit4&var-bgp_neighbor=Ufinet - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [00:19:44] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:11:16] (03PS1) 10TrainBranchBot: Branch commit for wmf/1.47.0-wmf.15 [core] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1324028 (https://phabricator.wikimedia.org/T430834) [01:11:19] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/1.47.0-wmf.15 [core] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1324028 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [01:12:13] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1324029 [01:12:13] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1324029 (owner: 10TrainBranchBot) [01:13:10] FIRING: BFDdown: BFD session down between cr1-eqiad and fe80::7a4f:9b00:d4e:7c0c - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [01:14:49] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [01:18:10] RESOLVED: BFDdown: BFD session down between cr1-eqiad and fe80::7a4f:9b00:d4e:7c0c - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [01:21:47] (03Merged) 10jenkins-bot: Branch commit for wmf/1.47.0-wmf.15 [core] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1324028 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [01:22:03] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1324029 (owner: 10TrainBranchBot) [01:31:32] (03PS2) 10Abijeet Patro: ULS: Remove wgULSLanguageSelectorV2Enabled config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324031 (https://phabricator.wikimedia.org/T429122) [02:00:05] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T0200) [02:00:44] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:20] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 35s) [02:07:54] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:10:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [02:14:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:00:05] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T0300) [03:01:58] (03PS1) 10TrainBranchBot: testwikis to 1.47.0-wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324032 (https://phabricator.wikimedia.org/T430834) [03:02:01] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by mwpresync@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324032 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [03:02:56] (03Merged) 10jenkins-bot: testwikis to 1.47.0-wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324032 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [03:03:18] !log mwpresync@deploy1003 Started scap sync-world: testwikis to 1.47.0-wmf.15 refs T430834 [03:03:22] T430834: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834 [03:36:51] !log mwpresync@deploy1003 Finished scap sync-world: testwikis to 1.47.0-wmf.15 refs T430834 (duration: 33m 33s) [03:36:55] T430834: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834 [03:40:13] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:00:04] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T0400) [04:02:32] !log mwpresync@deploy1003 Pruned MediaWiki: 1.47.0-wmf.12 (duration: 02m 26s) [05:09:44] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:10:39] FIRING: CoreBGPDown: Core BGP session down between cr2-eqord and cr2-codfw (208.80.153.193) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=cr2-eqord:9804&var-bgp_group=Confed_codfw&var-bgp_neighbor=cr2-codfw - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [05:14:44] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T0600) [06:00:05] marostegui, Amir1, and federico3: How many deployers does it take to do Primary database switchover deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T0600). [06:10:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:14:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:28:05] jouncebot: nowandnext [06:28:05] For the next 0 hour(s) and 31 minute(s): MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T0600) [06:28:05] For the next 0 hour(s) and 1 minute(s): Primary database switchover (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T0600) [06:28:05] In 0 hour(s) and 31 minute(s): UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T0700) [06:29:12] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for iberker - https://phabricator.wikimedia.org/T433258#12200511 (10jcrespo) @IBerker-WMF While I process your request, may I ask you to link the ilanberker account to here on phabricator as your LDAP/developer account, to ease futu... [06:36:25] (03PS1) 10Jcrespo: admin: Add ilanberker as an ssh-less user with analytics privatedata [puppet] - 10https://gerrit.wikimedia.org/r/1324200 (https://phabricator.wikimedia.org/T433258) [06:37:46] (03PS2) 10Jcrespo: admin: Add ilanberker as an ssh-less user with analytics privatedata [puppet] - 10https://gerrit.wikimedia.org/r/1324200 (https://phabricator.wikimedia.org/T433258) [06:40:40] (03CR) 10Jcrespo: [C:03+1] "This is ready to go, but requires review." [puppet] - 10https://gerrit.wikimedia.org/r/1324200 (https://phabricator.wikimedia.org/T433258) (owner: 10Jcrespo) [06:42:46] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12200531 (10jcrespo) This is blocked right now on getting legal confirmation that an NDA has been signed. [06:43:54] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12200533 (10jcrespo) This is blocked on clarification seeked from the original requester. [06:48:45] 06SRE, 10LDAP-Access-Requests: Grant Access to NDA for jaleman-vdr-wmf - https://phabricator.wikimedia.org/T433417#12200540 (10jcrespo) > I can look up the details for you if it is necessary Yes please, whoever is handier to look this up should be set as contact, and access will be extended automatically. Thi... [06:50:21] 10SRE-swift-storage, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): RdfStreamingUpdaterSpaceUsageTooHigh - https://phabricator.wikimedia.org/T431506#12200541 (10Gehel) [06:50:26] 07sre-alert-triage, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 06Machine-Learning-Team (Q1 FY2026-27): Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100) - https://phabricator.wikimedia.org/T414969#12200545 (10Gehel) [06:50:32] 07sre-alert-triage, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Alert in need of triage: PuppetFailure (instance an-test-client1002:9100) - https://phabricator.wikimedia.org/T427399#12200551 (10Gehel) [06:50:42] 07sre-alert-triage, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Alert in need of triage: AlertLintProblem (instance localhost:9123) - https://phabricator.wikimedia.org/T430139#12200555 (10Gehel) [06:50:52] 07sre-alert-triage, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Alert in need of triage: Dell PowerEdge or Supermicro Broadcom RAID Controller (instance an-worker1208) - https://phabricator.wikimedia.org/T430138#12200559 (10Gehel) [06:54:34] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12200653 (10Gehel) [06:54:40] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Degraded RAID on an-presto1013 - https://phabricator.wikimedia.org/T433030#12200656 (10Gehel) [06:55:06] 10ops-eqiad, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T433348#12200662 (10Gehel) [06:55:12] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Degraded RAID on an-worker1191 - https://phabricator.wikimedia.org/T431828#12200661 (10Gehel) [06:55:49] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): eqiad row A&B host migration details request for Search Platform - https://phabricator.wikimedia.org/T432651#12200670 (10Gehel) [06:58:32] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), and 3 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12200721 (10Gehel) [06:58:52] 06SRE, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 3 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12200725 (10Gehel) [07:00:05] Amir1, urbanecm, and awight: Time to snap out of that daydream and deploy UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T0700). [07:00:05] No Gerrit patches in the queue for this window AFAICS. [07:00:35] 10SRE-SLO, 10observability, 10Wikidata, 06Wikidata Platform Team, and 3 others: Update WDQS SLOs to reflect graph split changes - https://phabricator.wikimedia.org/T393966#12200773 (10Gehel) [07:02:21] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 13Patch-For-Review: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12200812 (10Gehel) [07:02:37] 10ops-codfw, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Noisy alerts: investigate fstrim.service failure on dse-k8s-wdqs2001 (new Supermicro host) - https://phabricator.wikimedia.org/T434299#12200818 (10Gehel) [07:09:16] (03PS1) 10Marostegui: db1164: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1324207 (https://phabricator.wikimedia.org/T434043) [07:09:55] (03CR) 10Marostegui: [C:03+2] db1164: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1324207 (https://phabricator.wikimedia.org/T434043) (owner: 10Marostegui) [07:14:03] (03PS1) 10Marostegui: mariadb: Promote db1164 to m1 master [puppet] - 10https://gerrit.wikimedia.org/r/1324209 (https://phabricator.wikimedia.org/T434043) [07:22:54] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:25:39] RESOLVED: CoreBGPDown: Core BGP session down between cr2-eqord and cr2-codfw (208.80.153.193) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=cr2-eqord:9804&var-bgp_group=Confed_codfw&var-bgp_neighbor=cr2-codfw - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [07:35:40] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1323955 (owner: 10L10n-bot) [07:36:04] (03Abandoned) 10Abijeet Patro: Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1321925 (owner: 10L10n-bot) [07:36:17] (03Abandoned) 10Abijeet Patro: Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1320140 (owner: 10L10n-bot) [07:36:33] (03Abandoned) 10Abijeet Patro: Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1319459 (owner: 10L10n-bot) [07:36:35] (03Abandoned) 10Abijeet Patro: Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1318103 (owner: 10L10n-bot) [07:40:13] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:43:48] (03PS2) 10Mpostoronca: Register the mediawiki.wikimedia_antiabuse.content_policy_score stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1321579 (https://phabricator.wikimedia.org/T432848) [07:44:15] FIRING: [2x] ProbeDown: Service idp2005:443 has failed probes (http_idp_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/CAS-SSO#Alerting - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:47:57] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1324200 (https://phabricator.wikimedia.org/T433258) (owner: 10Jcrespo) [07:49:05] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [07:49:15] RESOLVED: [2x] ProbeDown: Service idp2005:443 has failed probes (http_idp_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/CAS-SSO#Alerting - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:50:22] (03CR) 10Jcrespo: [C:03+2] admin: Add ilanberker as an ssh-less user with analytics privatedata [puppet] - 10https://gerrit.wikimedia.org/r/1324200 (https://phabricator.wikimedia.org/T433258) (owner: 10Jcrespo) [07:56:29] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for iberker - https://phabricator.wikimedia.org/T433258#12200948 (10jcrespo) 05In progress→03Resolved Access has been deployed, it will take up to 30 minutes to propagate to all servers. After that, ple... [08:00:56] (03PS1) 10Slyngshede: IDP: Switch active server [dns] - 10https://gerrit.wikimedia.org/r/1324220 [08:10:55] (03PS1) 10Btullis: Remove last mentions of an-test-master100[1-2] and an-test-coord1001 [puppet] - 10https://gerrit.wikimedia.org/r/1324248 (https://phabricator.wikimedia.org/T433494) [08:11:06] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [dns] - 10https://gerrit.wikimedia.org/r/1324220 (owner: 10Slyngshede) [08:12:49] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12200991 (10Chlod) >>! In T433791#12195091, @jcrespo wrote: > As you can see in the amended PR, you will be provided deployment rights, which includes some spiderp... [08:16:18] !log imported jenkins 2.568.2 to thirdparty/jenkins for trixie-wikimedia [08:16:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:16:58] (03CR) 10Slyngshede: [C:03+2] IDP: Switch active server [dns] - 10https://gerrit.wikimedia.org/r/1324220 (owner: 10Slyngshede) [08:17:19] !log slyngshede@dns1004 START - running authdns-update [08:17:58] (03PS1) 10Bartosz Wójtowicz: ml-services: Declare ephemeral-storage on non-kserve web services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324249 (https://phabricator.wikimedia.org/T431089) [08:18:39] !log installing openjdk-21 security updates [08:18:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:19:12] !log slyngshede@dns1004 END - running authdns-update [08:20:24] (03CR) 10Klausman: [C:03+1] Remove last mentions of an-test-master100[1-2] and an-test-coord1001 [puppet] - 10https://gerrit.wikimedia.org/r/1324248 (https://phabricator.wikimedia.org/T433494) (owner: 10Btullis) [08:20:47] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1152: Switching over ms1 [08:20:47] !log marostegui@cumin1003 START - Cookbook sre.mysql.parsercache [08:20:48] !log marostegui@cumin1003 END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99) [08:20:48] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Switching over ms1 [08:22:02] !log marostegui@cumin1003 dbctl commit (dc=all): 'Depool ms1 T434288', diff saved to https://phabricator.wikimedia.org/P95960 and previous config saved to /var/cache/conftool/dbconfig/20260811-082201-marostegui.json [08:22:07] T434288: Switchover ms1, ms2 and ms3 master - https://phabricator.wikimedia.org/T434288 [08:22:37] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2251.codfw.wmnet,db[1152,1267].eqiad.wmnet with reason: Switching over ms1 [08:23:43] (03CR) 10Btullis: [C:03+2] Remove last mentions of an-test-master100[1-2] and an-test-coord1001 [puppet] - 10https://gerrit.wikimedia.org/r/1324248 (https://phabricator.wikimedia.org/T433494) (owner: 10Btullis) [08:24:07] (03PS2) 10Marostegui: mariadb: Promote db1164 to m1 master [puppet] - 10https://gerrit.wikimedia.org/r/1324209 (https://phabricator.wikimedia.org/T434043) [08:24:07] (03PS1) 10Marostegui: instances.yaml: Add db1267 [puppet] - 10https://gerrit.wikimedia.org/r/1324250 (https://phabricator.wikimedia.org/T434288) [08:24:44] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:25:01] 06SRE, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission an-test-coord1001.eqiad.wmnet - https://phabricator.wikimedia.org/T433494#12201045 (10BTullis) [08:25:04] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2232].codfw.wmnet,db[1164,1213,1217].eqiad.wmnet with reason: Primary switchover m1 T434043 [08:25:09] T434043: Switchover m1 master (db1213 -> db1164) - https://phabricator.wikimedia.org/T434043 [08:25:21] 06SRE, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission an-test-coord1001.eqiad.wmnet - https://phabricator.wikimedia.org/T433494#12201047 (10BTullis) 05Open→03Resolved [08:25:57] 06SRE, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission an-test-master100[1-2] - https://phabricator.wikimedia.org/T433495#12201050 (10BTullis) 05Open→03Resolved [08:27:35] (03CR) 10Marostegui: [C:03+2] mariadb: Promote db1164 to m1 master [puppet] - 10https://gerrit.wikimedia.org/r/1324209 (https://phabricator.wikimedia.org/T434043) (owner: 10Marostegui) [08:27:48] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1267 [puppet] - 10https://gerrit.wikimedia.org/r/1324250 (https://phabricator.wikimedia.org/T434288) (owner: 10Marostegui) [08:29:58] !log Failover m1 from db1213 to db1164 - T434043 [08:30:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:30:57] (03CR) 10Marostegui: [C:03+2] mariadb: Switchover backup statistics primary db for dbbackups [puppet] - 10https://gerrit.wikimedia.org/r/1321468 (https://phabricator.wikimedia.org/T434043) (owner: 10Jcrespo) [08:31:20] (03CR) 10Marostegui: [C:03+2] "Per our irc chat, merging after the switchover." [puppet] - 10https://gerrit.wikimedia.org/r/1321468 (https://phabricator.wikimedia.org/T434043) (owner: 10Jcrespo) [08:32:35] (03Abandoned) 10Urbanecm: chore: update mediawiki/mediawiki-codesniffer to fix CI [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323971 (https://phabricator.wikimedia.org/T434187) (owner: 10Urbanecm) [08:33:57] (03PS18) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [08:34:21] (03PS1) 10Marostegui: db1164: Enable master [puppet] - 10https://gerrit.wikimedia.org/r/1324253 [08:34:36] (03CR) 10Marostegui: [V:03+2 C:03+2] db1164: Enable master [puppet] - 10https://gerrit.wikimedia.org/r/1324253 (owner: 10Marostegui) [08:35:50] (03CR) 10Federico Ceratto: "The cookbook runs the decommission cookbook at line 130 as:" [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [08:36:29] (03CR) 10Marostegui: "Then it failed for me past friday.It did everything but the decommissioning itself." [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [08:38:01] (03CR) 10CI reject: [V:04-1] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [08:39:33] (03CR) 10Marostegui: "Test with this REAL decommission: https://phabricator.wikimedia.org/T433474" [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [08:40:55] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1267 to dbctl T434288', diff saved to https://phabricator.wikimedia.org/P95961 and previous config saved to /var/cache/conftool/dbconfig/20260811-084054-marostegui.json [08:41:00] T434288: Switchover ms1, ms2 and ms3 master - https://phabricator.wikimedia.org/T434288 [08:42:41] (03PS1) 10Marostegui: db1267: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1324254 (https://phabricator.wikimedia.org/T434288) [08:43:17] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Declare ephemeral-storage on non-kserve web services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324249 (https://phabricator.wikimedia.org/T431089) (owner: 10Bartosz Wójtowicz) [08:43:43] (03CR) 10Marostegui: [C:03+2] db1267: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1324254 (https://phabricator.wikimedia.org/T434288) (owner: 10Marostegui) [08:48:05] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repool ms1 T434288', diff saved to https://phabricator.wikimedia.org/P95962 and previous config saved to /var/cache/conftool/dbconfig/20260811-084804-marostegui.json [08:48:09] T434288: Switchover ms1, ms2 and ms3 master - https://phabricator.wikimedia.org/T434288 [08:48:19] !log Switchover ms1 master in eqiad T434288 [08:48:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:49:05] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [08:49:47] (03PS1) 10Marostegui: wmnet: Change ms1-master [dns] - 10https://gerrit.wikimedia.org/r/1324257 (https://phabricator.wikimedia.org/T434288) [08:51:02] (03CR) 10Marostegui: [C:03+2] wmnet: Change ms1-master [dns] - 10https://gerrit.wikimedia.org/r/1324257 (https://phabricator.wikimedia.org/T434288) (owner: 10Marostegui) [08:51:07] !log marostegui@dns1004 START - running authdns-update [08:53:00] !log marostegui@dns1004 END - running authdns-update [08:56:33] (03CR) 10Federico Ceratto: "Ah, I found this in the logs:" [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [09:03:48] (03PS19) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [09:04:50] (03PS1) 10Marostegui: db1213: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1324260 (https://phabricator.wikimedia.org/T433463) [09:05:03] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Reimage [09:05:55] (03CR) 10Marostegui: [C:03+2] db1213: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1324260 (https://phabricator.wikimedia.org/T433463) (owner: 10Marostegui) [09:06:34] !log fceratto@cumin1003 START - Cookbook sre.mysql.decommission [09:06:59] !log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host db1213.eqiad.wmnet with OS trixie [09:07:07] (03CR) 10CI reject: [V:04-1] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [09:07:42] (03PS1) 10Marostegui: installserveR: Move db1267 to reuse UEFI [puppet] - 10https://gerrit.wikimedia.org/r/1324261 [09:08:14] (03PS20) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [09:08:17] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) [09:08:30] (03PS2) 10Marostegui: installserver: Move db1267 to reuse UEFI [puppet] - 10https://gerrit.wikimedia.org/r/1324261 [09:08:37] !log fceratto@cumin1003 START - Cookbook sre.mysql.decommission [09:08:41] !log fceratto@cumin1003 START - Cookbook sre.mysql.depool depool db1177: Decommission [09:09:17] !log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1177: Decommission [09:10:42] (03CR) 10Marostegui: [C:03+2] installserver: Move db1267 to reuse UEFI [puppet] - 10https://gerrit.wikimedia.org/r/1324261 (owner: 10Marostegui) [09:11:08] (03CR) 10CI reject: [V:04-1] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [09:12:17] fceratto@cumin1003 decommission (PID 2082362) is awaiting input [09:12:53] (03PS1) 10Federico Ceratto: instances.yaml: Remove db1177 from dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1324262 (https://phabricator.wikimedia.org/T433474) [09:14:44] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [09:18:22] (03CR) 10Marostegui: [C:03+1] instances.yaml: Remove db1177 from dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1324262 (https://phabricator.wikimedia.org/T433474) (owner: 10Federico Ceratto) [09:22:49] !log marostegui@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db1213.eqiad.wmnet with reason: host reimage [09:24:30] (03PS1) 10Marostegui: Revert "mariadb: Switchover backup statistics primary db for dbbackups" [puppet] - 10https://gerrit.wikimedia.org/r/1324263 [09:24:57] (03PS1) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [09:27:33] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1213.eqiad.wmnet with reason: host reimage [09:28:23] !log installing node-tar security updates [09:28:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:35:41] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-coord1004.eqiad.wmnet with OS bookworm [09:36:36] (03PS2) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [09:38:28] (03PS1) 10CWilliams: mariadb: Productionize db1272 [puppet] - 10https://gerrit.wikimedia.org/r/1324265 (https://phabricator.wikimedia.org/T407942) [09:39:41] (03PS3) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [09:40:25] (03PS1) 10STran: Document cusi_case_property table [puppet] - 10https://gerrit.wikimedia.org/r/1324266 (https://phabricator.wikimedia.org/T433853) [09:40:29] (03CR) 10Hnowlan: [C:03+1] profile::opensearch: add scap to roles [puppet] - 10https://gerrit.wikimedia.org/r/1321141 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [09:40:47] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-presto1006.eqiad.wmnet with OS bookworm [09:40:51] (03CR) 10Jcrespo: [C:03+1] Revert "mariadb: Switchover backup statistics primary db for dbbackups" [puppet] - 10https://gerrit.wikimedia.org/r/1324263 (owner: 10Marostegui) [09:41:18] !log installing Linux 6.12.101 on Trixie hosts [09:41:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:45:44] (03PS5) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [09:47:27] (03CR) 10Federico Ceratto: [C:03+2] instances.yaml: Remove db1177 from dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1324262 (https://phabricator.wikimedia.org/T433474) (owner: 10Federico Ceratto) [09:49:06] (03CR) 10Dreamy Jazz: Document cusi_case_property table (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324266 (https://phabricator.wikimedia.org/T433853) (owner: 10STran) [09:49:43] !log fceratto@cumin1003 START - Cookbook sre.hosts.decommission for hosts db1177.eqiad.wmnet [09:49:51] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1213.eqiad.wmnet with OS trixie [09:51:58] (03PS1) 10Marostegui: mariadb: Promote db1213 to m1 master [puppet] - 10https://gerrit.wikimedia.org/r/1324271 (https://phabricator.wikimedia.org/T434493) [09:52:28] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-coord1004.eqiad.wmnet with reason: host reimage [09:53:40] (03PS1) 10Btullis: Configure partition retention for all production HDFS nodes [puppet] - 10https://gerrit.wikimedia.org/r/1324272 (https://phabricator.wikimedia.org/T434494) [09:54:37] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2232].codfw.wmnet,db[1164,1213,1217].eqiad.wmnet with reason: Primary switchover m1 T434493 [09:54:41] T434493: Switchover m1 master (db1164 -> db1213) - https://phabricator.wikimedia.org/T434493 [09:55:33] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [09:55:47] (03CR) 10Marostegui: [C:03+2] mariadb: Promote db1213 to m1 master [puppet] - 10https://gerrit.wikimedia.org/r/1324271 (https://phabricator.wikimedia.org/T434493) (owner: 10Marostegui) [09:56:18] (03PS2) 10STran: Document cusi_case_property table [puppet] - 10https://gerrit.wikimedia.org/r/1324266 (https://phabricator.wikimedia.org/T433853) [09:56:24] (03CR) 10STran: Document cusi_case_property table (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324266 (https://phabricator.wikimedia.org/T433853) (owner: 10STran) [09:57:12] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1006.eqiad.wmnet with reason: host reimage [09:57:56] !log Failover m1 from db1164 to db1213 - T434493 [09:58:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:58:06] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-coord1004.eqiad.wmnet with reason: host reimage [09:58:13] (03CR) 10Dreamy Jazz: Document cusi_case_property table (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324266 (https://phabricator.wikimedia.org/T433853) (owner: 10STran) [09:58:53] (03CR) 10Marostegui: [C:03+2] Revert "mariadb: Switchover backup statistics primary db for dbbackups" [puppet] - 10https://gerrit.wikimedia.org/r/1324263 (owner: 10Marostegui) [09:59:25] (03CR) 10Ladsgroup: [C:03+1] "Ping me once it's renamed in production and I handle it." [puppet] - 10https://gerrit.wikimedia.org/r/1290814 (https://phabricator.wikimedia.org/T426102) (owner: 10Daimona Eaytoy) [10:00:03] (03CR) 10STran: Document cusi_case_property table (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324266 (https://phabricator.wikimedia.org/T433853) (owner: 10STran) [10:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1000) [10:00:06] (03CR) 10Blake: [C:03+2] kubernetes: Add a jobrunner deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1322782 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [10:00:21] (03CR) 10Ladsgroup: [C:04-1] "hmm, it's private, it'd be easier if we introduce a new one, add it to the filter and once it's been moved, we drop the old name" [puppet] - 10https://gerrit.wikimedia.org/r/1290814 (https://phabricator.wikimedia.org/T426102) (owner: 10Daimona Eaytoy) [10:00:49] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1006.eqiad.wmnet with reason: host reimage [10:00:56] 06SRE, 10LDAP-Access-Requests: Grant Access to NDA for jaleman-vdr-wmf - https://phabricator.wikimedia.org/T433417#12201621 (10jcrespo) 05Open→03Invalid So, given that this is a request for the NDA group given the target is already part of the WMF group, this request makes no sense. It could be that t... [10:01:38] fceratto@cumin1003 decommission (PID 2082362) is awaiting input [10:01:48] (03CR) 10Dreamy Jazz: [C:03+1] Document cusi_case_property table [puppet] - 10https://gerrit.wikimedia.org/r/1324266 (https://phabricator.wikimedia.org/T433853) (owner: 10STran) [10:02:05] PROBLEM - Uncommitted dbctl configuration changes- check dbctl config diff on cumin2003 is CRITICAL: CRITICAL - Uncommitted dbctl configuration changes, check dbctl config diff https://wikitech.wikimedia.org/wiki/Dbctl%23Uncommitted_dbctl_diffs [10:02:05] PROBLEM - Uncommitted dbctl configuration changes- check dbctl config diff on cumin2002 is CRITICAL: CRITICAL - Uncommitted dbctl configuration changes, check dbctl config diff https://wikitech.wikimedia.org/wiki/Dbctl%23Uncommitted_dbctl_diffs [10:02:07] PROBLEM - Uncommitted dbctl configuration changes- check dbctl config diff on cumin1003 is CRITICAL: CRITICAL - Uncommitted dbctl configuration changes, check dbctl config diff https://wikitech.wikimedia.org/wiki/Dbctl%23Uncommitted_dbctl_diffs [10:03:24] (03CR) 10Jcrespo: "This may need an amend to add contract end date and contact for one person. See: https://phabricator.wikimedia.org/T433417#12201621 for de" [puppet] - 10https://gerrit.wikimedia.org/r/1318975 (owner: 10Slyngshede) [10:03:37] !log fceratto@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1177.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" [10:04:19] !log fceratto@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1177.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003" [10:04:20] !log fceratto@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:04:21] !log fceratto@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1177.eqiad.wmnet [10:04:25] !log fceratto@cumin1003 Removing db1177 from zarcillo T433474 [10:04:29] !log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) [10:04:32] T433474: decommission db1177.eqiad.wmnet - https://phabricator.wikimedia.org/T433474 [10:06:22] 10ops-eqiad, 06DC-Ops, 10decommission-hardware: decommission db1177.eqiad.wmnet - https://phabricator.wikimedia.org/T433474#12201642 (10FCeratto-WMF) 05In progress→03Open a:05FCeratto-WMF→03None [10:06:37] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 1 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [10:07:33] (03CR) 10Federico Ceratto: "The cookbook ran successfully for https://phabricator.wikimedia.org/T433474#12201631" [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [10:08:07] (03CR) 10Marostegui: "Maybe it was a glitch then, who knows!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [10:09:16] !log blake@deploy1003 Started scap sync-world: Non-deployment run to populate release values for T427668 [10:09:20] T427668: Turn up the Pretrain MVP environment - https://phabricator.wikimedia.org/T427668 [10:09:37] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [10:09:48] !log blake@deploy1003 Stopping before sync operations [10:10:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:10:43] (03CR) 10Marostegui: "@fceratto@wikimedia.org you've not removed all the files for db1177 yet." [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [10:11:05] (03CR) 10Blake: [C:03+2] mw-pretrain: Add a jobrunner values.yaml. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321561 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [10:12:10] (03PS1) 10Marostegui: mariadb: Productionize db1278 [puppet] - 10https://gerrit.wikimedia.org/r/1324274 (https://phabricator.wikimedia.org/T407942) [10:13:09] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1278 [puppet] - 10https://gerrit.wikimedia.org/r/1324274 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [10:13:48] (03Merged) 10jenkins-bot: mw-pretrain: Add a jobrunner values.yaml. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321561 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [10:14:31] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply [10:17:35] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply [10:17:50] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [10:18:06] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1179.eqiad.wmnet onto db1278.eqiad.wmnet [10:18:10] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1179: Depool db1179.eqiad.wmnet to then clone it to db1278.eqiad.wmnet - marostegui@cumin1003 [10:19:50] (03CR) 10Federico Ceratto: "See the previous comment on ensure_shell_is_durable: it seems it did not run for you due to not having a tmux session running. I also adde" [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [10:20:10] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [10:20:45] (03CR) 10Huei Tan: [C:03+1] ULS: Remove wgULSLanguageSelectorV2Enabled config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324031 (https://phabricator.wikimedia.org/T429122) (owner: 10Abijeet Patro) [10:21:10] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1006.eqiad.wmnet with OS bookworm [10:21:51] (03CR) 10Hnowlan: logstash: enable reversing security plugin enable (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1320291 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [10:22:00] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-presto1007.eqiad.wmnet with OS bookworm [10:22:07] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-presto1008.eqiad.wmnet with OS bookworm [10:22:09] (03CR) 10Marostegui: "But that's weird, did only the non decommissioning steps ran due to that? (eg: removing from zarcillo etc)?" [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [10:23:40] (03PS1) 10Krinkle: Improve Math preference labels for SVG/MathJax/MathML [extensions/Math] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1324277 (https://phabricator.wikimedia.org/T433891) [10:24:10] (03PS1) 10Federico Ceratto: db1177.yaml: delete old file [puppet] - 10https://gerrit.wikimedia.org/r/1324276 (https://phabricator.wikimedia.org/T433474) [10:24:25] (03PS1) 10Phedenskog: ci: add ReadingLists to gitcache [puppet] - 10https://gerrit.wikimedia.org/r/1324278 (https://phabricator.wikimedia.org/T403560) [10:24:44] (03PS3) 10Dreamy Jazz: Register the mediawiki.wikimedia_antiabuse.content_policy_score stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1321579 (https://phabricator.wikimedia.org/T432848) (owner: 10Mpostoronca) [10:25:00] (03CR) 10CI reject: [V:04-1] ci: add ReadingLists to gitcache [puppet] - 10https://gerrit.wikimedia.org/r/1324278 (https://phabricator.wikimedia.org/T403560) (owner: 10Phedenskog) [10:25:28] !log fceratto@cumin1003 dbctl commit (dc=all): 'Remove db1177 T433474', diff saved to https://phabricator.wikimedia.org/P95964 and previous config saved to /var/cache/conftool/dbconfig/20260811-102527-fceratto.json [10:25:32] T433474: decommission db1177.eqiad.wmnet - https://phabricator.wikimedia.org/T433474 [10:26:14] (03CR) 10Marostegui: [C:04-1] "Needs to be removed from site.pp" [puppet] - 10https://gerrit.wikimedia.org/r/1324276 (https://phabricator.wikimedia.org/T433474) (owner: 10Federico Ceratto) [10:27:07] RECOVERY - Uncommitted dbctl configuration changes- check dbctl config diff on cumin2002 is OK: OK - no diffs https://wikitech.wikimedia.org/wiki/Dbctl%23Uncommitted_dbctl_diffs [10:27:07] RECOVERY - Uncommitted dbctl configuration changes- check dbctl config diff on cumin2003 is OK: OK - no diffs https://wikitech.wikimedia.org/wiki/Dbctl%23Uncommitted_dbctl_diffs [10:27:07] RECOVERY - Uncommitted dbctl configuration changes- check dbctl config diff on cumin1003 is OK: OK - no diffs https://wikitech.wikimedia.org/wiki/Dbctl%23Uncommitted_dbctl_diffs [10:28:10] (03PS2) 10Phedenskog: ci: add ReadingLists to gitcache [puppet] - 10https://gerrit.wikimedia.org/r/1324278 (https://phabricator.wikimedia.org/T403560) [10:28:58] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission db1177.eqiad.wmnet - https://phabricator.wikimedia.org/T433474#12201720 (10Marostegui) [10:29:10] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission db1177.eqiad.wmnet - https://phabricator.wikimedia.org/T433474#12201722 (10Marostegui) This is ready for DCOps. [10:29:10] (03PS2) 10Federico Ceratto: db1177.yaml: delete old file [puppet] - 10https://gerrit.wikimedia.org/r/1324276 (https://phabricator.wikimedia.org/T433474) [10:29:34] (03CR) 10Marostegui: [C:03+1] db1177.yaml: delete old file [puppet] - 10https://gerrit.wikimedia.org/r/1324276 (https://phabricator.wikimedia.org/T433474) (owner: 10Federico Ceratto) [10:29:56] (03PS2) 10Arnaudb: gitlab: point gitlab.wikimedia.org at the CDN [dns] - 10https://gerrit.wikimedia.org/r/1324281 (https://phabricator.wikimedia.org/T425441) [10:29:56] (03CR) 10Arnaudb: "that last change in the relation chain will effectively move Gitlab behind CDN." [dns] - 10https://gerrit.wikimedia.org/r/1324281 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [10:30:19] (03PS2) 10Arnaudb: gitlab: point gitlab-replica-b at the CDN [dns] - 10https://gerrit.wikimedia.org/r/1324275 (https://phabricator.wikimedia.org/T425441) [10:30:23] (03CR) 10Federico Ceratto: [C:03+2] db1177.yaml: delete old file [puppet] - 10https://gerrit.wikimedia.org/r/1324276 (https://phabricator.wikimedia.org/T433474) (owner: 10Federico Ceratto) [10:30:24] (03PS2) 10Arnaudb: gitlab: point gitlab-replica-a at the CDN [dns] - 10https://gerrit.wikimedia.org/r/1324280 (https://phabricator.wikimedia.org/T425441) [10:31:47] btullis@cumin1003 reimage (PID 2144008) is awaiting input [10:34:00] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1008.eqiad.wmnet with OS bookworm [10:34:13] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1179: Depool db1179.eqiad.wmnet to then clone it to db1278.eqiad.wmnet - marostegui@cumin1003 [10:35:52] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-presto1008.eqiad.wmnet with OS bookworm [10:37:13] marostegui@cumin1003 clone (PID 2136497) is awaiting input [10:37:34] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-coord1004.eqiad.wmnet with OS bookworm [10:38:14] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1007.eqiad.wmnet with reason: host reimage [10:42:10] (03PS2) 10CWilliams: mariadb: Productionize db1272 [puppet] - 10https://gerrit.wikimedia.org/r/1324265 (https://phabricator.wikimedia.org/T407942) [10:43:49] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1007.eqiad.wmnet with reason: host reimage [10:45:11] !log bump space for prometheus k8s-dse in codfw [10:45:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:45:28] (03CR) 10CWilliams: [C:03+2] mariadb: Productionize db1272 [puppet] - 10https://gerrit.wikimedia.org/r/1324265 (https://phabricator.wikimedia.org/T407942) (owner: 10CWilliams) [10:46:38] jouncebot: nowandnext [10:46:38] For the next 0 hour(s) and 13 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1000) [10:46:38] In 1 hour(s) and 13 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1200) [10:48:13] (03PS1) 10Ladsgroup: Use maximum compression level in SqlBlobStore and SqlBagOStuff [core] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324283 (https://phabricator.wikimedia.org/T428377) [10:48:33] (03CR) 10Ladsgroup: [C:03+2] Use maximum compression level in SqlBlobStore and SqlBagOStuff [core] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324283 (https://phabricator.wikimedia.org/T428377) (owner: 10Ladsgroup) [10:48:55] (03PS1) 10Slyngshede: data.yaml add gturkington to wmf [puppet] - 10https://gerrit.wikimedia.org/r/1324284 [10:57:30] (03PS1) 10Btullis: Migrate hive and presto to the new bookworm host [dns] - 10https://gerrit.wikimedia.org/r/1324285 (https://phabricator.wikimedia.org/T434494) [10:57:54] (03CR) 10Atsuko: [C:03+1] Configure partition retention for all production HDFS nodes [puppet] - 10https://gerrit.wikimedia.org/r/1324272 (https://phabricator.wikimedia.org/T434494) (owner: 10Btullis) [10:59:00] (03CR) 10Hnowlan: "This change doesn't modify paging, just an attempt to clean up the use of tags in alerts and make alerts.wikimedia.org more directly usefu" [alerts] - 10https://gerrit.wikimedia.org/r/1319827 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [11:00:31] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-presto1007 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 Dgrd : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T434505 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [11:00:44] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1007 - https://phabricator.wikimedia.org/T434505 (10ops-monitoring-bot) 03NEW [11:00:51] (03Merged) 10jenkins-bot: Use maximum compression level in SqlBlobStore and SqlBagOStuff [core] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324283 (https://phabricator.wikimedia.org/T428377) (owner: 10Ladsgroup) [11:01:35] (03CR) 10Btullis: [C:03+2] Configure partition retention for all production HDFS nodes [puppet] - 10https://gerrit.wikimedia.org/r/1324272 (https://phabricator.wikimedia.org/T434494) (owner: 10Btullis) [11:04:45] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1324284 (owner: 10Slyngshede) [11:07:12] (03PS1) 10Marostegui: db1152: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1324288 (https://phabricator.wikimedia.org/T434480) [11:07:59] (03CR) 10Marostegui: [C:03+2] db1152: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1324288 (https://phabricator.wikimedia.org/T434480) (owner: 10Marostegui) [11:09:36] (03CR) 10Slyngshede: [C:03+2] data.yaml add gturkington to wmf [puppet] - 10https://gerrit.wikimedia.org/r/1324284 (owner: 10Slyngshede) [11:10:04] (03PS1) 10Muehlenhoff: Remove LDAP access for ospingou [puppet] - 10https://gerrit.wikimedia.org/r/1324290 [11:11:47] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1007.eqiad.wmnet with OS bookworm [11:20:12] !log installing curl security updates [11:20:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:20:22] !log btullis@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-presto1008.eqiad.wmnet with OS bookworm [11:20:57] (03CR) 10Btullis: [C:03+2] Migrate hive and presto to the new bookworm host [dns] - 10https://gerrit.wikimedia.org/r/1324285 (https://phabricator.wikimedia.org/T434494) (owner: 10Btullis) [11:21:23] !log btullis@dns1004 START - running authdns-update [11:23:11] btullis@cumin1003 provision (PID 2186471) is awaiting input [11:23:19] !log btullis@dns1004 END - running authdns-update [11:24:44] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [11:25:23] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:26:47] !log btullis@cumin1003 START - Cookbook sre.hosts.provision for host an-presto1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:27:18] (03PS1) 10Gerrit maintenance bot: mariadb: Promote db2161 to s8 master [puppet] - 10https://gerrit.wikimedia.org/r/1324292 (https://phabricator.wikimedia.org/T434514) [11:28:55] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-presto1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:29:22] !log installing Python 3.11 security updates [11:29:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:31:33] jouncebot: nowandnext [11:31:33] No deployments scheduled for the next 0 hour(s) and 28 minute(s) [11:31:33] In 0 hour(s) and 28 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1200) [11:34:17] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: Declare ephemeral-storage on non-kserve web services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324249 (https://phabricator.wikimedia.org/T431089) (owner: 10Bartosz Wójtowicz) [11:34:31] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s8 T434514 [11:34:36] T434514: Switchover s8 master (db2165 -> db2161) - https://phabricator.wikimedia.org/T434514 [11:34:51] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Set db2161 with weight 0 T434514', diff saved to https://phabricator.wikimedia.org/P95966 and previous config saved to /var/cache/conftool/dbconfig/20260811-113449-cwilliams.json [11:36:41] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2003.codfw.wmnet [11:36:47] (03Merged) 10jenkins-bot: ml-services: Declare ephemeral-storage on non-kserve web services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324249 (https://phabricator.wikimedia.org/T431089) (owner: 10Bartosz Wójtowicz) [11:36:52] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet [11:39:15] (03CR) 10CWilliams: [C:03+2] mariadb: Promote db2161 to s8 master [puppet] - 10https://gerrit.wikimedia.org/r/1324292 (https://phabricator.wikimedia.org/T434514) (owner: 10Gerrit maintenance bot) [11:40:13] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:40:22] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1179: Pool db1179.eqiad.wmnet in after cloning [11:40:46] !log Starting s8 codfw failover from db2165 to db2161 - T434514 [11:40:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:40:49] T434514: Switchover s8 master (db2165 -> db2161) - https://phabricator.wikimedia.org/T434514 [11:41:38] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Promote db2161 to s8 primary T434514', diff saved to https://phabricator.wikimedia.org/P95968 and previous config saved to /var/cache/conftool/dbconfig/20260811-114136-cwilliams.json [11:42:53] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet [11:42:55] !log jmm@cumin2003 END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2003.codfw.wmnet [11:43:17] !log jnuche@deploy1003 Started deploy [releng/jenkins-deploy@050d19e] (releasing): test jenkins deploy for T434186 [11:43:53] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depool db2165 T434514', diff saved to https://phabricator.wikimedia.org/P95969 and previous config saved to /var/cache/conftool/dbconfig/20260811-114352-cwilliams.json [11:44:14] !log jnuche@deploy1003 Finished deploy [releng/jenkins-deploy@050d19e] (releasing): test jenkins deploy for T434186 (duration: 01m 08s) [11:45:37] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 2 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [11:47:36] !log jnuche@deploy1003 Started deploy [releng/jenkins-deploy@050d19e] (releasing): T434186 [11:48:42] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1324283|Use maximum compression level in SqlBlobStore and SqlBagOStuff (T428377)]] [11:48:44] !log jnuche@deploy1003 Finished deploy [releng/jenkins-deploy@050d19e] (releasing): T434186 (duration: 01m 14s) [11:48:45] T428377: Consider using lzma compression for storing wikidata entries in external storage - https://phabricator.wikimedia.org/T428377 [11:49:51] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance [11:50:55] !log jmm@cumin2002 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2002.codfw.wmnet [11:51:03] !log jmm@cumin2002 START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet [11:52:28] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1324283|Use maximum compression level in SqlBlobStore and SqlBagOStuff (T428377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:54:19] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [11:56:44] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2165: Security update [11:57:08] !log jmm@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet [11:57:11] !log jmm@cumin2002 END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2002.codfw.wmnet [12:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1200) [12:00:19] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324283|Use maximum compression level in SqlBlobStore and SqlBagOStuff (T428377)]] (duration: 11m 37s) [12:00:24] T428377: Consider using lzma compression for storing wikidata entries in external storage - https://phabricator.wikimedia.org/T428377 [12:01:23] PROBLEM - OSPF status on cr2-eqdfw is CRITICAL: OSPFv2: 5/7 UP : OSPFv3: 5/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:01:25] PROBLEM - OSPF status on cr2-drmrs is CRITICAL: OSPFv2: 5/6 UP : OSPFv3: 5/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:02:25] PROBLEM - OSPF status on cr2-esams is CRITICAL: OSPFv2: 5/6 UP : OSPFv3: 6/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:03:10] FIRING: BFDdown: BFD session down between cr2-eqdfw and 208.80.153.205 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:03:17] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4005.ulsfo.wmnet [12:03:23] RECOVERY - OSPF status on cr2-eqdfw is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:03:25] RECOVERY - OSPF status on cr2-esams is OK: OSPFv2: 6/6 UP : OSPFv3: 6/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:03:25] RECOVERY - OSPF status on cr2-drmrs is OK: OSPFv2: 6/6 UP : OSPFv3: 6/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:03:34] !log jmm@cumin2003 END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti4005.ulsfo.wmnet [12:04:05] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet [12:04:59] PROBLEM - ganeti-wconfd running on ganeti-test2001 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [12:07:38] !log failover ganeti master in ganeti/test to ganeti-test2003 [12:07:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:08:10] RESOLVED: BFDdown: BFD session down between cr2-eqdfw and 208.80.153.205 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:09:51] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [12:11:22] jmm@cumin2003 drain-node (PID 726585) is awaiting input [12:14:08] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2033.codfw.wmnet [12:16:15] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12202165 (10fgiunchedi) >>! In T431682#12174859, @taavi wrote: > Good question. Looking at the Netbox changelog that change was first made in [[ https://netbox.wik... [12:18:20] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-presto1008.eqiad.wmnet with OS bookworm [12:18:38] !log jmm@cumin2002 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet [12:18:44] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-presto1009.eqiad.wmnet with OS bookworm [12:19:06] !log jmm@cumin2002 START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet [12:22:12] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2033.codfw.wmnet [12:22:15] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet [12:22:52] !log failover ganeti master in codfw/routed to ganeti2033 [12:22:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:23:41] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-coord1003.eqiad.wmnet with OS bookworm [12:24:44] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:24:54] PROBLEM - ganeti-wconfd running on ganeti2034 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 111 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [12:25:35] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1179: Pool db1179.eqiad.wmnet in after cloning [12:25:50] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1179.eqiad.wmnet onto db1278.eqiad.wmnet [12:26:19] 07sre-alert-triage, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 06Machine-Learning-Team (Q1 FY2026-27): Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100) - https://phabricator.wikimedia.org/T414969#12202222 (10klausman) 05Open→03Resolved Disk situation has been resolved, see T4... [12:30:24] !log jmm@cumin2002 END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti-test2001.codfw.wmnet [12:30:24] !log jmm@cumin2002 END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet [12:30:50] PROBLEM - ganeti-wconfd running on ganeti-test2001 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [12:32:30] FIRING: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [12:34:37] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet [12:37:47] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . [12:38:13] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . [12:38:54] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . [12:39:49] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [12:40:36] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-coord1003.eqiad.wmnet with reason: host reimage [12:41:23] jmm@cumin2003 drain-node (PID 732200) is awaiting input [12:42:06] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2165: Security update [12:42:10] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12202330 (10fgiunchedi) >>! In T431682#12180113, @VRiley-WMF wrote: > @Andrew Awesome, sounds good. Would we like to start on 1049? I was thinking of doing the re... [12:42:14] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' . [12:42:55] RESOLVED: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [12:43:10] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [12:43:15] jouncebot: nowandnext [12:43:15] For the next 0 hour(s) and 16 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1200) [12:43:16] In 0 hour(s) and 16 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1300) [12:43:41] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-coord1003.eqiad.wmnet with reason: host reimage [12:43:58] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [12:44:29] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [12:44:59] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [12:45:31] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [12:46:15] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [12:46:46] (03PS1) 10Kosta Harlan: Revert "Lazily reject pre-fix parser-cache entries for noreferrer/noopener links" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324295 [12:46:59] (03PS3) 10Blake: mw-pretrain: Add a jobrunner-canary values.yaml. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321588 (https://phabricator.wikimedia.org/T427668) [12:47:19] (03PS2) 10Kosta Harlan: Revert "Lazily reject pre-fix parser-cache entries for noreferrer/noopener links" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324295 (https://phabricator.wikimedia.org/T429090) [12:47:21] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [12:49:42] (03PS1) 10Blake: kubernetes: Add a jobrunner-canary deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1324296 (https://phabricator.wikimedia.org/T427668) [12:51:30] !log marostegui@cumin1003 dbctl commit (dc=all): 'Depool ms2 T434288', diff saved to https://phabricator.wikimedia.org/P95977 and previous config saved to /var/cache/conftool/dbconfig/20260811-125129-marostegui.json [12:51:34] T434288: Switchover ms1, ms2 and ms3 master - https://phabricator.wikimedia.org/T434288 [12:52:30] RESOLVED: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [12:53:32] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db1157: Using as clone source [12:54:10] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1157: Using as clone source [12:59:01] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2253.codfw.wmnet,db[1151,1266].eqiad.wmnet with reason: Switching over ms2 [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: It is that lovely time of the day again! You are hereby commanded to deploy UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1300). [13:00:05] Superpes: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:35] Superpes: around? :) [13:00:51] Hi @urbanecm Yes :) [13:01:31] (03PS3) 10Superpes15: [ukwiki] Remove reviewer usergroup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323779 (https://phabricator.wikimedia.org/T434252) [13:02:06] (03CR) 10Urbanecm: [C:03+2] [ukwiki] Remove reviewer usergroup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323779 (https://phabricator.wikimedia.org/T434252) (owner: 10Superpes15) [13:02:26] (03PS1) 10Marostegui: instances.yaml: Add db1266 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1324300 (https://phabricator.wikimedia.org/T407942) [13:02:41] (03PS2) 10Superpes15: [frwiktionary] Add new Schème namespace and its talk [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323312 (https://phabricator.wikimedia.org/T415716) [13:02:45] (03CR) 10Urbanecm: [C:03+2] [frwiktionary] Add new Schème namespace and its talk [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323312 (https://phabricator.wikimedia.org/T415716) (owner: 10Superpes15) [13:03:03] (03Merged) 10jenkins-bot: [ukwiki] Remove reviewer usergroup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323779 (https://phabricator.wikimedia.org/T434252) (owner: 10Superpes15) [13:03:14] (03PS1) 10Marostegui: wmnet: Update ms2-master [dns] - 10https://gerrit.wikimedia.org/r/1324301 (https://phabricator.wikimedia.org/T407942) [13:03:43] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1266 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1324300 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [13:03:54] (03Merged) 10jenkins-bot: [frwiktionary] Add new Schème namespace and its talk [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323312 (https://phabricator.wikimedia.org/T415716) (owner: 10Superpes15) [13:03:56] (03CR) 10TrainBranchBot: [C:03+2] "Copied votes on follow-up patch sets have been updated:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323312 (https://phabricator.wikimedia.org/T415716) (owner: 10Superpes15) [13:04:13] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1323779|[ukwiki] Remove reviewer usergroup (T434252)]], [[gerrit:1323312|[frwiktionary] Add new Schème namespace and its talk (T415716)]] [13:04:19] T434252: Remove reviewer user group from ukwiki - https://phabricator.wikimedia.org/T434252 [13:04:20] T415716: New Namespace for fr.wiktionary.org: "Schème" - https://phabricator.wikimedia.org/T415716 [13:04:42] (03PS1) 10Tiziano Fogli: swift/rsyslog: filter out thumb deletion messages [puppet] - 10https://gerrit.wikimedia.org/r/1324302 (https://phabricator.wikimedia.org/T434502) [13:04:45] (03CR) 10Urbanecm: [C:03+2] [tgwiki] Remove temporary logo for Wikipedia 25 (Vector 2022 + Vector Legacy) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323350 (https://phabricator.wikimedia.org/T415307) (owner: 10Superpes15) [13:04:45] (03CR) 10Urbanecm: [C:03+2] [slwiki] Revert temporary logo for Wikipedia 25 (Vector legacy + Vector 2022) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1322961 (https://phabricator.wikimedia.org/T414265) (owner: 10Superpes15) [13:04:47] (03CR) 10Urbanecm: [C:03+2] [itwiki] Remove temporary logo for Wikipedia 25 (Vector 2022 + Vector Legacy) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323827 (https://phabricator.wikimedia.org/T414320) (owner: 10Superpes15) [13:05:51] (03Merged) 10jenkins-bot: [tgwiki] Remove temporary logo for Wikipedia 25 (Vector 2022 + Vector Legacy) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323350 (https://phabricator.wikimedia.org/T415307) (owner: 10Superpes15) [13:05:58] (03Merged) 10jenkins-bot: [slwiki] Revert temporary logo for Wikipedia 25 (Vector legacy + Vector 2022) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1322961 (https://phabricator.wikimedia.org/T414265) (owner: 10Superpes15) [13:06:02] (03Merged) 10jenkins-bot: [itwiki] Remove temporary logo for Wikipedia 25 (Vector 2022 + Vector Legacy) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323827 (https://phabricator.wikimedia.org/T414320) (owner: 10Superpes15) [13:06:13] !log urbanecm@deploy1003 urbanecm, superpes: Backport for [[gerrit:1323779|[ukwiki] Remove reviewer usergroup (T434252)]], [[gerrit:1323312|[frwiktionary] Add new Schème namespace and its talk (T415716)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:06:21] Superpes: can you test, please? [13:06:28] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1266 to dbctl T434288', diff saved to https://phabricator.wikimedia.org/P95979 and previous config saved to /var/cache/conftool/dbconfig/20260811-130627-marostegui.json [13:06:32] T434288: Switchover ms1, ms2 and ms3 master - https://phabricator.wikimedia.org/T434288 [13:06:48] @Urbanecm Both work fine :) [13:06:58] !log urbanecm@deploy1003 urbanecm, superpes: Continuing with deployment [13:07:01] that was quick [13:07:21] Yep I was ready lmao [13:07:26] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repool ms2 T434288', diff saved to https://phabricator.wikimedia.org/P95980 and previous config saved to /var/cache/conftool/dbconfig/20260811-130725-marostegui.json [13:07:36] I had just to update the page :P [13:08:17] (03PS2) 10Tiziano Fogli: swift/rsyslog: filter out thumb deletion messages [puppet] - 10https://gerrit.wikimedia.org/r/1324302 (https://phabricator.wikimedia.org/T434502) [13:08:17] (03CR) 10Marostegui: [C:03+2] wmnet: Update ms2-master [dns] - 10https://gerrit.wikimedia.org/r/1324301 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [13:08:21] !log marostegui@dns1004 START - running authdns-update [13:10:14] !log marostegui@dns1004 END - running authdns-update [13:10:23] (03CR) 10Slyngshede: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1260730 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [13:11:03] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1323779|[ukwiki] Remove reviewer usergroup (T434252)]], [[gerrit:1323312|[frwiktionary] Add new Schème namespace and its talk (T415716)]] (duration: 06m 49s) [13:11:09] T434252: Remove reviewer user group from ukwiki - https://phabricator.wikimedia.org/T434252 [13:11:09] T415716: New Namespace for fr.wiktionary.org: "Schème" - https://phabricator.wikimedia.org/T415716 [13:11:26] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1323350|[tgwiki] Remove temporary logo for Wikipedia 25 (Vector 2022 + Vector Legacy) (T415307)]], [[gerrit:1322961|[slwiki] Revert temporary logo for Wikipedia 25 (Vector legacy + Vector 2022) (T414265)]], [[gerrit:1323827|[itwiki] Remove temporary logo for Wikipedia 25 (Vector 2022 + Vector Legacy) (T414320)]] [13:11:31] RECOVERY - Host ml-serve1015 is UP: PING OK - Packet loss = 0%, RTA = 0.59 ms [13:11:34] T415307: Requesting temporary logo change for tg.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T415307 [13:11:34] T414265: Requesting temporary logo change for sl.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T414265 [13:11:35] T414320: Requesting temporary logo change for it.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T414320 [13:12:00] cwilliams@cumin1003 clone (PID 2266957) is awaiting input [13:13:31] !log urbanecm@deploy1003 urbanecm, superpes: Backport for [[gerrit:1323350|[tgwiki] Remove temporary logo for Wikipedia 25 (Vector 2022 + Vector Legacy) (T415307)]], [[gerrit:1322961|[slwiki] Revert temporary logo for Wikipedia 25 (Vector legacy + Vector 2022) (T414265)]], [[gerrit:1323827|[itwiki] Remove temporary logo for Wikipedia 25 (Vector 2022 + Vector Legacy) (T414320)]] synced to the testservers (see https://wiki [13:13:31] tech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:13:39] Superpes: what about now? :) [13:13:44] Ah give me some time lol [13:14:26] sure [13:14:38] btullis@cumin1003 reimage (PID 2227825) is awaiting input [13:14:44] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [13:15:19] Everything's fine! Thanks for your patience @urbanecm :) [13:15:24] !log urbanecm@deploy1003 urbanecm, superpes: Continuing with deployment [13:15:33] Superpes: imagine a patch that fixes all of the bugs! [13:15:36] even the ones no one reported yet [13:15:53] (03PS1) 10Marostegui: db1266: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1324305 [13:16:09] LOL It's a dream (but also a utopia) [13:16:50] I'll definitively remove the temporary logos next week as you taught me lol [13:16:54] sounds good [13:16:54] (03PS4) 10Ssingh: wmf-config/ProductionServices: set URL for urldownloader to service record [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313985 (https://phabricator.wikimedia.org/T429175) [13:17:03] (03CR) 10Ssingh: "rebased, no code change" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313985 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [13:17:08] (03CR) 10Ssingh: wmf-config/ProductionServices: set URL for urldownloader to service record [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313985 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [13:17:36] (03PS40) 10Slyngshede: P:tofurkey Add tofurkey [puppet] - 10https://gerrit.wikimedia.org/r/1260730 (https://phabricator.wikimedia.org/T355446) [13:19:06] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-coord1003.eqiad.wmnet with OS bookworm [13:19:18] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, August 11 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313985 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [13:19:22] (03PS1) 10Dreamy Jazz: WikimediaAntiAbuse: Enable personal info for enwiki with no display [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324306 (https://phabricator.wikimedia.org/T431292) [13:19:26] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1323350|[tgwiki] Remove temporary logo for Wikipedia 25 (Vector 2022 + Vector Legacy) (T415307)]], [[gerrit:1322961|[slwiki] Revert temporary logo for Wikipedia 25 (Vector legacy + Vector 2022) (T414265)]], [[gerrit:1323827|[itwiki] Remove temporary logo for Wikipedia 25 (Vector 2022 + Vector Legacy) (T414320)]] (duration: 08m 00s) [13:19:34] T415307: Requesting temporary logo change for tg.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T415307 [13:19:35] T414265: Requesting temporary logo change for sl.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T414265 [13:19:35] T414320: Requesting temporary logo change for it.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T414320 [13:19:37] (03CR) 10Ssingh: [V:03+1 C:03+2] P:kubernetes::deployment_server::global_config: update IPs for urldownloader [puppet] - 10https://gerrit.wikimedia.org/r/1321987 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [13:20:12] (03CR) 10Marostegui: [C:03+2] db1266: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1324305 (owner: 10Marostegui) [13:20:14] Thanks @urbanecm :3 [13:20:22] any time [13:20:57] !log urbanecm@deploy1003 mwscript-k8s job started: namespaceDupes.php --wiki=frwiktionary # T415716 [13:21:01] T415716: New Namespace for fr.wiktionary.org: "Schème" - https://phabricator.wikimedia.org/T415716 [13:21:07] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet [13:21:11] !log urbanecm@deploy1003 mwscript-k8s job started: namespaceDupes.php --wiki=frwiktionary --fix # T415716 [13:21:12] (03PS3) 10Tiziano Fogli: swift/rsyslog: filter out thumb deletion messages [puppet] - 10https://gerrit.wikimedia.org/r/1324302 (https://phabricator.wikimedia.org/T434502) [13:22:54] (03CR) 10JMeybohm: miscweb: add pod_annotiations to deployment tpl (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320943 (https://phabricator.wikimedia.org/T433588) (owner: 10AOkoth) [13:24:06] (03PS1) 10Ladsgroup: mariadb: Mark cx_corpora as private [puppet] - 10https://gerrit.wikimedia.org/r/1324309 (https://phabricator.wikimedia.org/T415219) [13:24:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:25:29] !log cwilliams@cumin1003 START - Cookbook sre.mysql.clone of db1157.eqiad.wmnet onto db1272.eqiad.wmnet [13:25:37] !log cwilliams@cumin1003 END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db1157.eqiad.wmnet onto db1272.eqiad.wmnet [13:25:59] (03PS1) 10JMeybohm: wikikube: Update staging eqiad to calico 3.30 [puppet] - 10https://gerrit.wikimedia.org/r/1324310 (https://phabricator.wikimedia.org/T427400) [13:28:15] (03PS1) 10JMeybohm: wikikube-staging eqiad: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324311 (https://phabricator.wikimedia.org/T427400) [13:29:07] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet [13:29:15] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet [13:29:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:29:40] 10ops-codfw, 10Data-Persistence-Backup, 10database-backups, 06DBA, 06DC-Ops: db2201's both mysql instances crashed in the last few days - https://phabricator.wikimedia.org/T434532#12202718 (10jcrespo) > DIMM A8 DRAM DDR4 32 GB Presence Detected|Critical Error Dual Rank 2933 MT/s @ DC ops, could you help... [13:31:10] urbanecm: are you done deploying? [13:31:13] yes [13:31:16] go ahead :) [13:31:17] ok, I'll sync a patch now [13:31:33] (03CR) 10Dreamy Jazz: [C:03+1] Revert "Lazily reject pre-fix parser-cache entries for noreferrer/noopener links" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324295 (https://phabricator.wikimedia.org/T429090) (owner: 10Kosta Harlan) [13:31:35] 10ops-eqiad, 06SRE, 06DC-Ops: wmcs PXE issues - https://phabricator.wikimedia.org/T433538#12202745 (10Jhancock.wm) @fgiunchedi welcome back. no rush, but let us know when works for you on these servers. thanks! [13:32:25] (03PS1) 10Blake: envoy-future: Update envoy-future to 1.39.0. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1324312 (https://phabricator.wikimedia.org/T421418) [13:32:43] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313985 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [13:34:01] (03Merged) 10jenkins-bot: wmf-config/ProductionServices: set URL for urldownloader to service record [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313985 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [13:34:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 2.762% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:34:20] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1324310 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [13:34:21] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1313985|wmf-config/ProductionServices: set URL for urldownloader to service record (T429175)]] [13:34:23] 10ops-codfw, 10Data-Persistence-Backup, 10database-backups, 06DBA, 06DC-Ops: db2201's both mysql instances crashed in the last few days - https://phabricator.wikimedia.org/T434532#12202787 (10Jhancock.wm) server is still under warranty. starting the process to getting it replaced. [13:34:25] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [13:36:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 4.993% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:36:21] !log kharlan@deploy1003 kharlan, sukhe: Backport for [[gerrit:1313985|wmf-config/ProductionServices: set URL for urldownloader to service record (T429175)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:36:26] 10ops-codfw, 10Data-Persistence-Backup, 10database-backups, 06DBA, 06DC-Ops: db2201's both mysql instances crashed in the last few days - https://phabricator.wikimedia.org/T434532#12202796 (10jcrespo) Great news, then! [13:36:28] (03CR) 10Hnowlan: [C:03+1] "lgtm! please mention to DPE before rolling out" [puppet] - 10https://gerrit.wikimedia.org/r/1324302 (https://phabricator.wikimedia.org/T434502) (owner: 10Tiziano Fogli) [13:36:39] <_joe_> kostajh: maybe this is ytou? [13:36:57] it also fired before this patch though [13:37:00] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4005.ulsfo.wmnet [13:37:00] <_joe_> ah no just to testservers [13:37:02] <_joe_> so no [13:37:02] 09:24:15 <+jinxer-wm> FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: [13:37:05] The patch hasn't synced yet [13:37:15] <_joe_> yea yeah sorry [13:37:16] (03PS1) 10Btullis: Revert "Migrate hive and presto to the new bookworm host" [dns] - 10https://gerrit.wikimedia.org/r/1324313 [13:37:16] _joe_: do you want me to wait on syncing it? [13:37:19] <_joe_> but please hold yes [13:37:23] no worries, good to check [13:37:24] ok [13:37:29] <_joe_> !ack [13:37:30] 8240 (ACKED) PHPFPMTooBusy sre (mw-web main eqiad) [13:38:15] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.248s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [13:38:26] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1008.eqiad.wmnet with OS bookworm [13:38:48] !log sukhe@cumin1003 START - Cookbook sre.hosts.reimage for host dns7001.wikimedia.org with OS trixie [13:39:06] (03CR) 10Marostegui: [C:03+1] "As we do not replicate x1, there's no need to restart sanitarium hosts." [puppet] - 10https://gerrit.wikimedia.org/r/1324309 (https://phabricator.wikimedia.org/T415219) (owner: 10Ladsgroup) [13:39:24] !log sukhe@cumin1003 START - Cookbook sre.hosts.reimage for host dns4003.wikimedia.org with OS trixie [13:39:36] 10ops-eqiad, 06DC-Ops: Alert for device ps1-c3-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T434547 (10phaultfinder) 03NEW [13:40:35] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-presto1008.eqiad.wmnet with OS bookworm [13:40:54] jmm@cumin2003 drain-node (PID 743371) is awaiting input [13:41:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 4.993% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:41:43] _joe_: ok to proceed with syncing the urldownloader patch? [13:41:51] <_joe_> no please wait [13:41:52] ok [13:41:58] I'll wait to hear from you [13:42:03] <_joe_> the issue is still ongoing even if it's below paging threshold [13:42:31] PROBLEM - BFD status on asw1-b3-magru.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:43:15] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.029s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [13:44:10] FIRING: [2x] BFDdown: BFD session down between asw1-b3-magru and 195.200.68.4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:44:27] ^ expeced, reimage [13:44:52] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti4005.ulsfo.wmnet [13:45:00] (03CR) 10JMeybohm: [C:03+2] wikikube-staging eqiad: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324311 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [13:45:19] (03CR) 10JMeybohm: [C:03+2] wikikube: Update staging eqiad to calico 3.30 [puppet] - 10https://gerrit.wikimedia.org/r/1324310 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [13:45:34] (03CR) 10MVernon: [C:03+1] "Looks reasonable to me." [puppet] - 10https://gerrit.wikimedia.org/r/1324302 (https://phabricator.wikimedia.org/T434502) (owner: 10Tiziano Fogli) [13:47:55] FIRING: [5x] JobUnavailable: Reduced availability for job haproxy in ops@magru - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:48:29] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4005.ulsfo.wmnet [13:48:50] FIRING: ProbeDown: Service ganeti4005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:48:54] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4005.ulsfo.wmnet [13:49:10] FIRING: [2x] GanetiBGPDown: BGP session down between ganeti4005 and asw1-22-ulsfo - group ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPDown - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPDown [13:49:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:50:51] sukhe: I think I need to cancel the deployment, and make a revert of the urldownloader patch and merge that. [13:51:52] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4006.ulsfo.wmnet [13:52:14] kostajh: ok +1, let's try again in a later window I guess? even tomorrow works [13:52:23] <_joe_> kostajh: please proceed [13:52:52] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7001.magru.wmnet [13:52:52] !log Failover ms1 T434288 [13:52:54] !log Failover ms2 T434288 [13:52:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:52:59] T434288: Switchover ms1, ms2 and ms3 master - https://phabricator.wikimedia.org/T434288 [13:53:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:53:50] RESOLVED: ProbeDown: Service ganeti4005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:54:10] RESOLVED: [2x] GanetiBGPDown: BGP session down between ganeti4005 and asw1-22-ulsfo - group ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPDown - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPDown [13:54:44] FIRING: [5x] JobUnavailable: Reduced availability for job haproxy in ops@magru - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:54:54] jmm@cumin2003 drain-node (PID 745392) is awaiting input [13:55:12] <_joe_> kostajh: not sure if it was clear, but you have a green light to finish the deployment [13:55:19] _joe_: thanks, will proceed [13:55:22] !log kharlan@deploy1003 kharlan, sukhe: Continuing with deployment [13:55:52] !log jayme@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'sync'. [13:55:56] !log jayme@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. [13:55:57] (03Merged) 10jenkins-bot: wikikube-staging eqiad: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324311 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [13:55:59] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1008.eqiad.wmnet with reason: host reimage [13:56:25] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7001.magru.wmnet [13:56:51] !log jayme@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'sync'. [13:57:06] !log sukhe@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on dns4003.wikimedia.org with reason: host reimage [13:58:01] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti4006.ulsfo.wmnet [13:58:04] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7002.magru.wmnet [13:59:28] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1313985|wmf-config/ProductionServices: set URL for urldownloader to service record (T429175)]] (duration: 25m 06s) [13:59:32] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [13:59:35] sukhe: seems to not be working [13:59:36] !log jayme@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. [13:59:48] (03CR) 10Btullis: [C:03+2] Revert "Migrate hive and presto to the new bookworm host" [dns] - 10https://gerrit.wikimedia.org/r/1324313 (owner: 10Btullis) [13:59:49] 10ops-codfw, 10Data-Persistence-Backup, 10database-backups, 06DBA, 06DC-Ops: db2201's both mysql instances crashed in the last few days - https://phabricator.wikimedia.org/T434532#12203102 (10jcrespo) While we wait for vendor servicing, x1 had redundancy and it was not the active host, so we are good the... [13:59:58] kostajh: oh really? [13:59:58] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1008.eqiad.wmnet with reason: host reimage [13:59:58] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:00:04] Deploy window Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1400) [14:00:06] !log btullis@dns1004 START - running authdns-update [14:00:07] sukhe: yes, reverting [14:00:11] kostajh: that's surprising :( [14:00:19] (03PS1) 10Kosta Harlan: Revert "wmf-config/ProductionServices: set URL for urldownloader to service record" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324318 [14:00:20] ok thanks. so hcaptcha was borken? [14:00:24] sukhe: yes [14:00:25] (03PS1) 10Andrew Bogott: ceph.conf: move bluestore slow ops config into [osd] section [puppet] - 10https://gerrit.wikimedia.org/r/1324319 (https://phabricator.wikimedia.org/T429387) [14:00:46] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324318 (owner: 10Kosta Harlan) [14:00:57] that's surprising because I checked against the service itself and that was working. ok. we will check what happened. [14:01:09] (03PS2) 10Kosta Harlan: Revert "wmf-config/ProductionServices: set URL for urldownloader to service record" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324318 (https://phabricator.wikimedia.org/T429175) [14:01:18] (03CR) 10TrainBranchBot: "Approved by kharlan@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324318 (https://phabricator.wikimedia.org/T429175) (owner: 10Kosta Harlan) [14:01:39] let's discuss it once you are done (in meetings now) [14:01:47] I wasn't sure if there was an issue with WikimediaDebug, so I proceeded to a full sync. In retrospect, the issue was visible via WikimediaDebug [14:02:03] !log btullis@dns1004 FAIL - running authdns-update [14:02:18] (03Merged) 10jenkins-bot: Revert "wmf-config/ProductionServices: set URL for urldownloader to service record" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324318 (https://phabricator.wikimedia.org/T429175) (owner: 10Kosta Harlan) [14:02:21] jmm@cumin2003 drain-node (PID 748119) is awaiting input [14:02:38] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1324318|Revert "wmf-config/ProductionServices: set URL for urldownloader to service record" (T429175)]] [14:02:38] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti7002.magru.wmnet [14:02:55] FIRING: [4x] JobUnavailable: Reduced availability for job haproxy in ops@magru - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:03:20] !log sukhe@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on dns7001.wikimedia.org with reason: host reimage [14:03:51] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns4003.wikimedia.org with reason: host reimage [14:04:46] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4006.ulsfo.wmnet [14:04:46] !log kharlan@deploy1003 kharlan: Backport for [[gerrit:1324318|Revert "wmf-config/ProductionServices: set URL for urldownloader to service record" (T429175)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:04:47] !log updated calico to v3.30.7 on staging-eqiad - T427400 [14:04:52] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [14:04:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:04:55] T427400: Update calico to 3.30 - https://phabricator.wikimedia.org/T427400 [14:04:58] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:05:11] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4006.ulsfo.wmnet [14:05:21] !log kharlan@deploy1003 kharlan: Continuing with deployment [14:05:57] (03PS1) 10Superpes15: [itwiki/slwiki/tgwiki] Remove temporary Wikipedia 25 logos permanently (already reverted) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324320 (https://phabricator.wikimedia.org/T414265) [14:06:13] (03CR) 10Tiziano Fogli: [C:03+2] swift/rsyslog: filter out thumb deletion messages [puppet] - 10https://gerrit.wikimedia.org/r/1324302 (https://phabricator.wikimedia.org/T434502) (owner: 10Tiziano Fogli) [14:06:36] (03PS1) 10Muehlenhoff: Record LDAP access for mmilenkovicwmf [puppet] - 10https://gerrit.wikimedia.org/r/1324321 [14:06:38] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4007.ulsfo.wmnet [14:06:38] PROBLEM - Recursive DNS on 198.35.26.7 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [14:06:45] (03CR) 10Lerickson: [C:03+1] WDQS: Enable prometheus metrics on Qlever [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321948 (https://phabricator.wikimedia.org/T433040) (owner: 10Trueg) [14:07:05] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2209 firmware upgrade after a crash - https://phabricator.wikimedia.org/T431952#12203152 (10Jhancock.wm) couldn't get a list. just "all of them" [14:07:16] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7001.wikimedia.org with reason: host reimage [14:09:12] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-master1004.eqiad.wmnet with OS bookworm [14:09:24] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324318|Revert "wmf-config/ProductionServices: set URL for urldownloader to service record" (T429175)]] (duration: 06m 46s) [14:09:44] sukhe: reverted [14:09:59] kostajh: thanks. I will sync up after meeting block and plan the next steps [14:10:07] thanks [14:11:02] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7002.magru.wmnet [14:11:07] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7002.magru.wmnet [14:11:12] jmm@cumin2003 drain-node (PID 749035) is awaiting input [14:11:39] PROBLEM - Recursive DNS on 2620:0:863:1:198:35:26:7 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [14:12:41] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti4007.ulsfo.wmnet [14:12:43] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7003.magru.wmnet [14:13:01] (03CR) 10Muehlenhoff: [C:03+2] Record LDAP access for mmilenkovicwmf [puppet] - 10https://gerrit.wikimedia.org/r/1324321 (owner: 10Muehlenhoff) [14:13:46] (03PS2) 10Daimona Eaytoy: tables-catalog: Rename ce_worklist_articles to ce_invitation_list_articles [puppet] - 10https://gerrit.wikimedia.org/r/1290814 (https://phabricator.wikimedia.org/T426102) [14:13:55] PROBLEM - Recursive DNS on 2a02:ec80:700:1:195:200:68:4 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [14:14:11] ^ expected [14:14:57] (03CR) 10Muehlenhoff: [C:03+2] Remove LDAP access for ospingou [puppet] - 10https://gerrit.wikimedia.org/r/1324290 (owner: 10Muehlenhoff) [14:15:09] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti7003.magru.wmnet [14:15:52] (03PS3) 10Daimona Eaytoy: tables-catalog: Add ce_invitation_list_articles [puppet] - 10https://gerrit.wikimedia.org/r/1290814 (https://phabricator.wikimedia.org/T426102) [14:15:53] (03CR) 10AOkoth: miscweb: add pod_annotiations to deployment tpl (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320943 (https://phabricator.wikimedia.org/T433588) (owner: 10AOkoth) [14:17:07] (03PS2) 10AOkoth: miscweb: add pod_annotiations to deployment tpl [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320943 (https://phabricator.wikimedia.org/T433588) [14:17:11] (03PS1) 10Daimona Eaytoy: tables-catalog: Drop ce_worklist_articles [puppet] - 10https://gerrit.wikimedia.org/r/1324325 (https://phabricator.wikimedia.org/T426102) [14:17:33] (03CR) 10AOkoth: miscweb: add pod_annotiations to deployment tpl (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320943 (https://phabricator.wikimedia.org/T433588) (owner: 10AOkoth) [14:17:48] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1008.eqiad.wmnet with OS bookworm [14:18:13] (03CR) 10Daimona Eaytoy: "Done, this patch is now for adding only, and I made a separate one to remove. Does the plan in T426102#12185841 for the rename sound good " [puppet] - 10https://gerrit.wikimedia.org/r/1290814 (https://phabricator.wikimedia.org/T426102) (owner: 10Daimona Eaytoy) [14:18:21] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-presto1010.eqiad.wmnet with OS bookworm [14:19:27] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4007.ulsfo.wmnet [14:19:51] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4007.ulsfo.wmnet [14:21:35] (03CR) 10Trueg: [C:03+2] WDQS: Enable prometheus metrics on Qlever [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321948 (https://phabricator.wikimedia.org/T433040) (owner: 10Trueg) [14:21:37] !log btullis@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-master1004.eqiad.wmnet with OS bookworm [14:22:27] !log btullis@cumin1003 START - Cookbook sre.hosts.provision for host an-master1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:23:18] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-master1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:23:21] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7003.magru.wmnet [14:23:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7003.magru.wmnet [14:23:55] (03Merged) 10jenkins-bot: WDQS: Enable prometheus metrics on Qlever [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321948 (https://phabricator.wikimedia.org/T433040) (owner: 10Trueg) [14:24:46] (03PS2) 10Arnaudb: gitlab: expose staleness of the backup used by replica restores [puppet] - 10https://gerrit.wikimedia.org/r/1324308 (https://phabricator.wikimedia.org/T425441) [14:24:47] (03CR) 10Arnaudb: [C:03+2] "alerts to come in https://gerrit.wikimedia.org/r/c/operations/alerts/+/1324322" [puppet] - 10https://gerrit.wikimedia.org/r/1324308 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [14:27:11] (03CR) 10Cathal Mooney: "Nice work! One nit in-line which isn't too important. I'll catch up with you on how I can use this in place of the rule I added in 13194" [puppet] - 10https://gerrit.wikimedia.org/r/1322963 (https://phabricator.wikimedia.org/T433601) (owner: 10JHathaway) [14:27:30] (03CR) 10David Caro: [C:03+1] ceph.conf: move bluestore slow ops config into [osd] section (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324319 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [14:27:43] btullis@cumin1003 reimage (PID 2320256) is awaiting input [14:27:55] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:28:49] !log failover ganeti master in ulsfo to ganeti4005 [14:28:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:29:56] PROBLEM - Recursive DNS on 195.200.68.4 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [14:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1430) [14:30:13] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7004.magru.wmnet [14:31:08] PROBLEM - ganeti-wconfd running on ganeti4008 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [14:31:08] (03PS2) 10Daimona Eaytoy: Enable campaignEvents on bdwikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1275938 (https://phabricator.wikimedia.org/T424016) (owner: 10Yahya) [14:31:09] (03PS1) 10Marostegui: installserver: Do not format db1266 [puppet] - 10https://gerrit.wikimedia.org/r/1324326 [14:31:13] !log cwilliams@cumin1003 START - Cookbook sre.mysql.clone of db1157.eqiad.wmnet onto db1272.eqiad.wmnet [14:31:37] (03PS3) 10Daimona Eaytoy: Enable campaignEvents on bdwikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1275938 (https://phabricator.wikimedia.org/T424016) (owner: 10Yahya) [14:32:07] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 12 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1275938 (https://phabricator.wikimedia.org/T424016) (owner: 10Yahya) [14:32:10] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-master1004.eqiad.wmnet with OS bookworm [14:32:41] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1009.eqiad.wmnet with OS bookworm [14:34:25] (03CR) 10Marostegui: [C:03+2] installserver: Do not format db1266 [puppet] - 10https://gerrit.wikimedia.org/r/1324326 (owner: 10Marostegui) [14:34:30] RECOVERY - BFD status on asw1-b3-magru.mgmt is OK: UP: 2 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [14:35:09] jmm@cumin2003 drain-node (PID 754083) is awaiting input [14:35:09] (03PS2) 10Andrew Bogott: ceph.conf: move bluestore slow ops config into [osd] section [puppet] - 10https://gerrit.wikimedia.org/r/1324319 (https://phabricator.wikimedia.org/T429387) [14:35:17] (03PS1) 10Ssingh: dns4003/dns7001: use new pdns-rec config [puppet] - 10https://gerrit.wikimedia.org/r/1324327 (https://phabricator.wikimedia.org/T401832) [14:36:37] (03CR) 10Ssingh: [C:03+2] dns4003/dns7001: use new pdns-rec config [puppet] - 10https://gerrit.wikimedia.org/r/1324327 (https://phabricator.wikimedia.org/T401832) (owner: 10Ssingh) [14:36:39] btullis@cumin1003 provision (PID 2334347) is awaiting input [14:36:41] !log btullis@cumin1003 START - Cookbook sre.hosts.provision for host an-presto1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:37:18] (03CR) 10David Caro: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1324319 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [14:37:55] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:38:30] (03PS3) 10Andrew Bogott: ceph.conf: move bluestore slow ops config into [osd] section [puppet] - 10https://gerrit.wikimedia.org/r/1324319 (https://phabricator.wikimedia.org/T429387) [14:38:35] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1324319 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [14:38:41] (03CR) 10Scott French: [C:03+1] kubernetes: Add a jobrunner-canary deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1324296 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [14:39:09] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-presto1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:39:10] RESOLVED: [2x] BFDdown: BFD session down between asw1-b3-magru and 195.200.68.4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:39:30] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-presto1009.eqiad.wmnet with OS bookworm [14:39:52] RECOVERY - Recursive DNS on 195.200.68.4 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [14:39:52] RECOVERY - Recursive DNS on 2a02:ec80:700:1:195:200:68:4 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [14:40:35] !log btullis@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-presto1010.eqiad.wmnet with OS bookworm [14:40:56] (03PS2) 10Cwhite: profile::opensearch: add scap to roles [puppet] - 10https://gerrit.wikimedia.org/r/1321141 (https://phabricator.wikimedia.org/T350516) [14:41:29] !log btullis@cumin1003 START - Cookbook sre.hosts.provision for host an-presto1010.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:41:36] RECOVERY - Recursive DNS on 198.35.26.7 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [14:41:36] RECOVERY - Recursive DNS on 2620:0:863:1:198:35:26:7 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [14:42:55] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:43:10] (03CR) 10Scott French: "Thanks, Blake!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321588 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [14:44:05] I am going to restart the CI Jenkins in a few minutes to catch up with a java upgraded which has happened this morning. [14:44:21] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-presto1010.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:44:42] (03CR) 10Andrew Bogott: [C:03+2] ceph.conf: move bluestore slow ops config into [osd] section (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324319 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [14:44:58] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [14:47:16] (03CR) 10Andrew Bogott: "Everyone agrees that we would like a metric about when and how often this fires. Of course it's hard to record a metric while the network " [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [14:47:20] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7001.wikimedia.org with OS trixie [14:47:34] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-presto1010.eqiad.wmnet with OS bookworm [14:47:36] (03PS1) 10Tiziano Fogli: swift/rsyslog: move comment to avoid parsing issues [puppet] - 10https://gerrit.wikimedia.org/r/1324330 (https://phabricator.wikimedia.org/T434502) [14:48:25] (03CR) 10JMeybohm: [C:03+1] miscweb: add pod_annotiations to deployment tpl [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320943 (https://phabricator.wikimedia.org/T433588) (owner: 10AOkoth) [14:48:29] (03CR) 10Tiziano Fogli: [C:03+2] swift/rsyslog: move comment to avoid parsing issues [puppet] - 10https://gerrit.wikimedia.org/r/1324330 (https://phabricator.wikimedia.org/T434502) (owner: 10Tiziano Fogli) [14:49:22] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns4003.wikimedia.org with OS trixie [14:49:50] (03CR) 10Cwhite: [C:03+2] profile::opensearch: add scap to roles [puppet] - 10https://gerrit.wikimedia.org/r/1321141 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [14:51:06] (03PS1) 10TChin: [eventgate] Bump to v1.32.0, make eventgate-analytics auto-refresh stream configs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324331 (https://phabricator.wikimedia.org/T430154) [14:52:55] FIRING: [4x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:54:22] (03Abandoned) 10Andrew Bogott: Revert "ceph.conf: support adjusting slow ops health messages" [puppet] - 10https://gerrit.wikimedia.org/r/1322877 (owner: 10Andrew Bogott) [14:55:38] (03PS1) 10Gerrit maintenance bot: mariadb: Promote db2207 to s2 master [puppet] - 10https://gerrit.wikimedia.org/r/1324332 (https://phabricator.wikimedia.org/T434565) [14:55:48] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1010.eqiad.wmnet with OS bookworm [14:56:54] (03PS1) 10Tiziano Fogli: swift/rsyslog: use re_match instead of regex [puppet] - 10https://gerrit.wikimedia.org/r/1324333 (https://phabricator.wikimedia.org/T434502) [14:57:39] (03CR) 10Tiziano Fogli: [C:03+2] swift/rsyslog: use re_match instead of regex [puppet] - 10https://gerrit.wikimedia.org/r/1324333 (https://phabricator.wikimedia.org/T434502) (owner: 10Tiziano Fogli) [14:57:52] (03PS5) 10Cwhite: logstash: enable reversing security plugin enable [puppet] - 10https://gerrit.wikimedia.org/r/1320291 (https://phabricator.wikimedia.org/T350516) [14:58:12] (03CR) 10Cwhite: logstash: enable reversing security plugin enable (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1320291 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [14:59:44] FIRING: [4x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:59:47] (03PS1) 10Reedy: InitialiseSettings: Enable 2FA enforcement on various private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324335 (https://phabricator.wikimedia.org/T428103) [15:00:05] jelto, arnoldokoth, mutante, and arnaudb: OwO what's this, a deployment window?? SRE Collaboration Services office hours. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1500). nyaa~ [15:01:18] (03CR) 10Dreamy Jazz: [C:03+1] Register the mediawiki.wikimedia_antiabuse.content_policy_score stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1321579 (https://phabricator.wikimedia.org/T432848) (owner: 10Mpostoronca) [15:01:36] 06SRE: expanddblist without parameters doesn't fail cleanly - https://phabricator.wikimedia.org/T434567 (10Reedy) 03NEW [15:02:38] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s2 T434565 [15:02:42] T434565: Switchover s2 master (db2204 -> db2207) - https://phabricator.wikimedia.org/T434565 [15:02:42] (03PS1) 10JavierMonton: alerts: webrequest-pageview webrequest-pageview is a new application that needs alerts. Here are new alerts defined for when the application is failing, lagging, or losing records for any unexpected reason. [alerts] - 10https://gerrit.wikimedia.org/r/1324336 (https://phabricator.wikimedia.org/T431536) [15:02:50] (03CR) 10CI reject: [V:04-1] alerts: webrequest-pageview webrequest-pageview is a new application that needs alerts. Here are new alerts defined for when the application is failing, lagging, or losing records for any unexpected reason. [alerts] - 10https://gerrit.wikimedia.org/r/1324336 (https://phabricator.wikimedia.org/T431536) (owner: 10JavierMonton) [15:03:30] 06SRE: expanddblist without arguments doesn't fail cleanly - https://phabricator.wikimedia.org/T434567#12203506 (10Reedy) [15:03:44] (03CR) 10Andrew Bogott: [C:03+1] icinga: remove absented wikitech-static checks [puppet] - 10https://gerrit.wikimedia.org/r/1312958 (https://phabricator.wikimedia.org/T362397) (owner: 10Filippo Giunchedi) [15:04:03] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Set db2207 with weight 0 T434565', diff saved to https://phabricator.wikimedia.org/P95982 and previous config saved to /var/cache/conftool/dbconfig/20260811-150402-cwilliams.json [15:04:15] (03CR) 10Filippo Giunchedi: [C:03+2] icinga: remove absented wikitech-static checks [puppet] - 10https://gerrit.wikimedia.org/r/1312958 (https://phabricator.wikimedia.org/T362397) (owner: 10Filippo Giunchedi) [15:06:04] (03PS2) 10Gerrit maintenance bot: mariadb: Promote db2207 to s2 master [puppet] - 10https://gerrit.wikimedia.org/r/1324332 (https://phabricator.wikimedia.org/T434565) [15:06:19] (03CR) 10CWilliams: [C:03+2] mariadb: Promote db2207 to s2 master [puppet] - 10https://gerrit.wikimedia.org/r/1324332 (https://phabricator.wikimedia.org/T434565) (owner: 10Gerrit maintenance bot) [15:08:24] !log Starting s2 codfw failover from db2204 to db2207 - T434565 [15:08:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:08:28] T434565: Switchover s2 master (db2204 -> db2207) - https://phabricator.wikimedia.org/T434565 [15:08:37] (03CR) 10Ottomata: [C:03+1] [eventgate] Bump to v1.32.0, make eventgate-analytics auto-refresh stream configs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324331 (https://phabricator.wikimedia.org/T430154) (owner: 10TChin) [15:09:06] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Promote db2207 to s2 primary T434565', diff saved to https://phabricator.wikimedia.org/P95983 and previous config saved to /var/cache/conftool/dbconfig/20260811-150905-cwilliams.json [15:10:02] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti7004.magru.wmnet [15:10:39] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4008.ulsfo.wmnet [15:10:43] (03CR) 10Andrew Bogott: [C:04-1] cloud-vps: kick networkd if network breaks [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [15:11:27] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depool db2204 T434565', diff saved to https://phabricator.wikimedia.org/P95984 and previous config saved to /var/cache/conftool/dbconfig/20260811-151126-cwilliams.json [15:11:37] !log Restarting CI Jenkins on contint1003 due to Java upgrade. [15:11:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:11:58] !log btullis@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-master1004.eqiad.wmnet with OS bookworm [15:12:26] Jenkins is back up [15:12:36] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-master1004.eqiad.wmnet with OS bookworm [15:12:59] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 2 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [15:14:02] (03CR) 10Ahmon Dancy: "This is working in beta." [puppet] - 10https://gerrit.wikimedia.org/r/1321653 (https://phabricator.wikimedia.org/T434114) (owner: 10Ahmon Dancy) [15:14:49] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2204.codfw.wmnet with reason: Maintenance [15:16:28] jmm@cumin2003 drain-node (PID 760640) is awaiting input [15:16:33] jouncebot now [15:16:33] For the next 0 hour(s) and 43 minute(s): SRE Collaboration Services office hours (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1500) [15:16:49] !log dancy@deploy1003 Installing scap version "4.278.0" for 3 host(s) [15:18:12] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7004.magru.wmnet [15:18:18] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7004.magru.wmnet [15:18:46] !log dancy@deploy1003 Installation of scap version "4.278.0" completed for 3 hosts [15:20:48] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2204: Security update [15:22:02] (03PS1) 10Reedy: InitialiseSettings: Enable 2FA warnings on more private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324338 (https://phabricator.wikimedia.org/T428103) [15:22:14] (03CR) 10BBlack: [C:04-1] Varnish: Reject non-thumb requests to thumbs (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [15:22:41] (03PS1) 10Ahmon Dancy: Add /w/deployment-info.php entrypoint [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324339 [15:22:50] (03PS2) 10Reedy: InitialiseSettings: Enable 2FA warnings on more private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324338 (https://phabricator.wikimedia.org/T428103) [15:23:18] (03CR) 10Ahmon Dancy: "bd808 reviewed in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1321096" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324339 (owner: 10Ahmon Dancy) [15:23:29] (03Abandoned) 10Ahmon Dancy: Add /w/deployment-info.php entrypoint [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1321096 (owner: 10Ahmon Dancy) [15:23:52] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dancy@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324339 (owner: 10Ahmon Dancy) [15:24:56] (03Merged) 10jenkins-bot: Add /w/deployment-info.php entrypoint [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324339 (owner: 10Ahmon Dancy) [15:25:15] !log dancy@deploy1003 Started scap sync-world: Backport for [[gerrit:1324339|Add /w/deployment-info.php entrypoint]] [15:25:38] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:27:18] !log dancy@deploy1003 dancy: Backport for [[gerrit:1324339|Add /w/deployment-info.php entrypoint]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:27:55] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:28:21] !log remove 2026-05 swift log archives from centrallog to free some space (T434502) [15:28:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:28:25] T434502: Swift logging is swamping centrallog hosts - https://phabricator.wikimedia.org/T434502 [15:28:38] !log dancy@deploy1003 dancy: Continuing with deployment [15:28:50] jouncebot: nowandnext [15:28:50] For the next 0 hour(s) and 31 minute(s): SRE Collaboration Services office hours (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1500) [15:28:50] In 0 hour(s) and 31 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1600) [15:29:13] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti4008.ulsfo.wmnet [15:30:03] !log failover ganeti master in magru to ganeti7004 [15:30:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:32:31] PROBLEM - ganeti-wconfd running on ganeti7001 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [15:32:41] !log dancy@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324339|Add /w/deployment-info.php entrypoint]] (duration: 07m 25s) [15:32:44] !log dancy@deploy1003 Installing scap version "4.279.0" for 3 host(s) [15:33:52] !log btullis@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-master1004.eqiad.wmnet with OS bookworm [15:34:18] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-master1004.eqiad.wmnet with OS bookworm [15:34:22] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-master1004.eqiad.wmnet with OS bookworm [15:34:39] !log dancy@deploy1003 Installation of scap version "4.279.0" completed for 3 hosts [15:34:56] !log dancy@deploy1003 Started scap sync-world: Testing [15:34:59] (03CR) 10JavierMonton: "recheck" [alerts] - 10https://gerrit.wikimedia.org/r/1324336 (https://phabricator.wikimedia.org/T431536) (owner: 10JavierMonton) [15:35:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4008.ulsfo.wmnet [15:35:38] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4008.ulsfo.wmnet [15:35:55] (03PS1) 10Ahmon Dancy: Merge remote-tracking branch 'origin/master' into train-dev [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1324346 [15:36:26] (03CR) 10Ahmon Dancy: [C:03+2] Merge remote-tracking branch 'origin/master' into train-dev [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1324346 (owner: 10Ahmon Dancy) [15:36:53] (03PS1) 10Reedy: expanddblist: Better handle no expression being passed [puppet] - 10https://gerrit.wikimedia.org/r/1324348 (https://phabricator.wikimedia.org/T434567) [15:38:03] (03Merged) 10jenkins-bot: Merge remote-tracking branch 'origin/master' into train-dev [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1324346 (owner: 10Ahmon Dancy) [15:38:58] (03CR) 10Ahmon Dancy: [C:03+1] ci: add ReadingLists to gitcache [puppet] - 10https://gerrit.wikimedia.org/r/1324278 (https://phabricator.wikimedia.org/T403560) (owner: 10Phedenskog) [15:39:14] (03PS2) 10Scott French: ingress: Introduce istio 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 [15:39:46] (03CR) 10Hnowlan: [C:03+1] logstash: enable reversing security plugin enable [puppet] - 10https://gerrit.wikimedia.org/r/1320291 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [15:39:48] 10ops-eqiad, 06DC-Ops, 10decommission-hardware, 06Traffic: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12203755 (10ssingh) [15:40:18] (03PS1) 10Scott French: ingress: Copy istio 1.2.0 to 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324342 [15:40:19] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-master1004.eqiad.wmnet with OS bookworm [15:40:22] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-master1004.eqiad.wmnet with OS bookworm [15:41:24] !log dancy@deploy1003 Finished scap sync-world: Testing (duration: 06m 28s) [15:44:11] (03Abandoned) 10Aqu: Add airflow.job_stats event stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319089 (https://phabricator.wikimedia.org/T405360) (owner: 10Aqu) [15:45:02] (03PS1) 10Reedy: WmfConfig: Type hints and modernisation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324350 (https://phabricator.wikimedia.org/T434568) [15:47:50] (03CR) 10Scott French: "Thank you both in advance for the review!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 (owner: 10Scott French) [15:48:06] btullis@cumin1003 reimage (PID 2338137) is awaiting input [15:54:04] !log jhancock@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS trixie [15:54:07] (03PS2) 10Scott French: ingress: Copy istio 1.2.0 to 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324342 (https://phabricator.wikimedia.org/T427666) [15:54:10] (03PS3) 10Scott French: ingress: Introduce istio 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 (https://phabricator.wikimedia.org/T427666) [15:54:11] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [15:54:20] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12203842 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-ctrl2006.codfw.... [15:54:43] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [15:55:12] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db1157: Pool db1157.eqiad.wmnet in after cloning [15:56:26] (03CR) 10Arnaudb: [C:03+1] "looks good to me!" [puppet] - 10https://gerrit.wikimedia.org/r/1324278 (https://phabricator.wikimedia.org/T403560) (owner: 10Phedenskog) [15:59:13] (03CR) 10Reedy: [C:03+2] InitialiseSettings: Enable 2FA enforcement on various private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324335 (https://phabricator.wikimedia.org/T428103) (owner: 10Reedy) [16:00:05] jhathaway and rzl: gettimeofday() says it's time for Puppet request window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1600) [16:00:05] No Gerrit patches in the queue for this window AFAICS. [16:00:08] (03Merged) 10jenkins-bot: InitialiseSettings: Enable 2FA enforcement on various private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324335 (https://phabricator.wikimedia.org/T428103) (owner: 10Reedy) [16:00:15] (03PS1) 10Muehlenhoff: Remove obsolete ms-be.cfg Partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/1324354 (https://phabricator.wikimedia.org/T156955) [16:01:22] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [16:02:22] !log reedy@deploy1003 Started scap sync-world: Backport for [[gerrit:1324335|InitialiseSettings: Enable 2FA enforcement on various private wikis (T428103)]] [16:02:26] T428103: Enforce 2FA for all users on private wikis in WMF production - https://phabricator.wikimedia.org/T428103 [16:02:57] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7001.magru.wmnet [16:03:53] 10ops-codfw, 06SRE, 06collaboration-services, 06DC-Ops: lists2001 has multiple bus errors - https://phabricator.wikimedia.org/T423159#12203886 (10LSobanski) 05Open→03Resolved a:03LSobanski The host has been rebooted since (uptime is 31 days), @Jhancock.wm could you confirm is the warning light is... [16:03:55] PROBLEM - NTP peers and stratum check on dns4003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/NTP [16:04:36] !log reedy@deploy1003 reedy: Backport for [[gerrit:1324335|InitialiseSettings: Enable 2FA enforcement on various private wikis (T428103)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:04:58] !log reedy@deploy1003 reedy: Continuing with deployment [16:05:07] 06SRE, 06collaboration-services, 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Seen): Upgrade CI Jenkins ssh key to ecdsa - https://phabricator.wikimedia.org/T177826#12203907 (10LSobanski) @hashar any updates on this one? [16:05:32] jhancock@cumin1003 reimage (PID 2399041) is awaiting input [16:06:02] jmm@cumin2003 drain-node (PID 770863) is awaiting input [16:06:10] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2204: Security update [16:07:55] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:08:05] !log jhancock@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie [16:08:19] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12203921 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-ctrl2006.co... [16:09:03] !log reedy@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324335|InitialiseSettings: Enable 2FA enforcement on various private wikis (T428103)]] (duration: 06m 40s) [16:09:06] T428103: Enforce 2FA for all users on private wikis in WMF production - https://phabricator.wikimedia.org/T428103 [16:09:50] (03PS1) 10Ssingh: magru: cache-(text|upload): switch to LE certs [puppet] - 10https://gerrit.wikimedia.org/r/1324355 [16:11:08] (03CR) 10Ssingh: [V:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9197/co" [puppet] - 10https://gerrit.wikimedia.org/r/1324355 (owner: 10Ssingh) [16:11:44] (03CR) 10BBlack: [C:03+1] magru: cache-(text|upload): switch to LE certs [puppet] - 10https://gerrit.wikimedia.org/r/1324355 (owner: 10Ssingh) [16:11:55] PROBLEM - NTP peers and stratum check on dns7001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/NTP [16:12:13] (03PS2) 10Ssingh: magru: cache-(text|upload): switch to LE certs [puppet] - 10https://gerrit.wikimedia.org/r/1324355 [16:12:33] uh [16:13:15] !log restart ntpsec.serviceon dns7001 [16:13:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:13:41] jhancock@cumin1003 reimage (PID 2399041) is awaiting input [16:14:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:15:18] (03PS3) 10Reedy: InitialiseSettings: Enable 2FA warnings on more private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324338 (https://phabricator.wikimedia.org/T428103) [16:15:26] (03CR) 10CI reject: [V:04-1] InitialiseSettings: Enable 2FA warnings on more private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324338 (https://phabricator.wikimedia.org/T428103) (owner: 10Reedy) [16:15:35] (03CR) 10Ssingh: [V:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9198/co" [puppet] - 10https://gerrit.wikimedia.org/r/1324355 (owner: 10Ssingh) [16:15:54] (03PS4) 10Reedy: InitialiseSettings: Enable 2FA warnings on more private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324338 (https://phabricator.wikimedia.org/T428103) [16:15:59] (03PS5) 10Reedy: InitialiseSettings: Enable 2FA warnings on more private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324338 (https://phabricator.wikimedia.org/T428103) [16:19:41] 06SRE, 06collaboration-services, 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Seen): Upgrade CI Jenkins ssh key to ecdsa - https://phabricator.wikimedia.org/T177826#12203976 (10hashar) I am not working on it, last time I acted on that front was a couple years ago at T37... [16:21:26] (03PS1) 10Reedy: InitialiseSettings: Fix wgOATHAuthEnforce2FAForAll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324356 [16:22:42] (03CR) 10Reedy: [C:03+2] InitialiseSettings: Fix wgOATHAuthEnforce2FAForAll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324356 (owner: 10Reedy) [16:23:42] (03Merged) 10jenkins-bot: InitialiseSettings: Fix wgOATHAuthEnforce2FAForAll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324356 (owner: 10Reedy) [16:24:23] !log reedy@deploy1003 Started scap sync-world: Backport for [[gerrit:1324356|InitialiseSettings: Fix wgOATHAuthEnforce2FAForAll]] [16:24:39] RECOVERY - NTP peers and stratum check on dns7001 is OK: NTP OK: Offset -7.6063e-05 secs, stratum=1 https://wikitech.wikimedia.org/wiki/NTP [16:24:42] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12204035 (10Jhancock.wm) okay so this is what i've found. the provision script needed another go cause the PXE got moved again. I ra... [16:24:49] RECOVERY - NTP peers and stratum check on dns4003 is OK: NTP OK: Offset -0.002742301 secs, stratum=2 https://wikitech.wikimedia.org/wiki/NTP [16:26:27] !log reedy@deploy1003 reedy: Backport for [[gerrit:1324356|InitialiseSettings: Fix wgOATHAuthEnforce2FAForAll]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:27:13] !log reedy@deploy1003 reedy: Continuing with deployment [16:28:13] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-master1004.eqiad.wmnet with OS bookworm [16:30:37] !log jhancock@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS trixie [16:30:56] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12204059 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-ctrl2006.codfw.... [16:31:15] !log reedy@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324356|InitialiseSettings: Fix wgOATHAuthEnforce2FAForAll]] (duration: 06m 52s) [16:33:18] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12204095 (10KFrancis) Hi all, I'm waiting on legal counsel. I'll ping him again.... [16:37:55] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12204120 (10KFrancis) Hi all, the NDA is complete! Thanks! [16:38:19] (03CR) 10Blake: [C:03+1] ingress: Introduce istio 1.3.0 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [16:38:59] !log jhancock@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie [16:39:18] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12204121 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-ctrl2006.co... [16:40:13] (03CR) 10Ladsgroup: [C:03+1] "Thank you and sorry for the trouble!" [puppet] - 10https://gerrit.wikimedia.org/r/1324302 (https://phabricator.wikimedia.org/T434502) (owner: 10Tiziano Fogli) [16:40:25] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Pool db1157.eqiad.wmnet in after cloning [16:40:38] 10SRE-swift-storage, 10Thumbor: Gradually drop all thumbnails as a one-off clean up - https://phabricator.wikimedia.org/T379942#12204130 (10Ladsgroup) [16:41:40] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1157.eqiad.wmnet onto db1272.eqiad.wmnet [16:43:58] (03PS2) 10Hnowlan: debian: guard against missing systemd package [debs/pint] - 10https://gerrit.wikimedia.org/r/1319039 [16:44:01] jhancock@cumin1003 reimage (PID 2424350) is awaiting input [16:44:24] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-master1004.eqiad.wmnet with reason: host reimage [16:44:44] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [16:45:53] (03CR) 10Hnowlan: "I think we keep our base images as base as is possible so I think adding the packages is not the way to go. I've changed to use the `[ -x " [debs/pint] - 10https://gerrit.wikimedia.org/r/1319039 (owner: 10Hnowlan) [16:46:30] 10ops-codfw, 06SRE, 10Data-Persistence-Backup, 10database-backups, and 2 others: db2201's both mysql instances crashed in the last few days - https://phabricator.wikimedia.org/T434532#12204177 (10Jhancock.wm) SR230212724 [16:48:28] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-master1004.eqiad.wmnet with reason: host reimage [16:51:12] !log sukhe@dns1004 START - running authdns-update [16:52:23] (03CR) 10BCornwall: [C:03+1] magru: cache-(text|upload): switch to LE certs [puppet] - 10https://gerrit.wikimedia.org/r/1324355 (owner: 10Ssingh) [16:53:00] !log sukhe@dns1004 FAIL - running authdns-update [16:53:06] uh [16:53:08] !log sukhe@dns1004 START - running authdns-update [16:54:11] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:54:58] !log sukhe@dns1004 END - running authdns-update [16:56:37] uh? [16:58:09] (03PS1) 10Andrew Bogott: aptrepo: Add k8s v1.33, remove k8s v1.31 [puppet] - 10https://gerrit.wikimedia.org/r/1324360 (https://phabricator.wikimedia.org/T408785) [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1700) [17:00:23] any deploys happening in this window? I want to push a couple of clean ups [17:00:26] (03PS4) 10Scott French: ingress: Introduce istio 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 (https://phabricator.wikimedia.org/T427666) [17:02:13] !log sudo cumin "A:cp-magru" "disable-puppet 'merging CR 1324355'" [17:02:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:02:15] (03CR) 10Scott French: "Thanks for the review!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324343 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [17:02:45] (03CR) 10Majavah: [C:03+1] aptrepo: Add k8s v1.33, remove k8s v1.31 [puppet] - 10https://gerrit.wikimedia.org/r/1324360 (https://phabricator.wikimedia.org/T408785) (owner: 10Andrew Bogott) [17:03:13] (03CR) 10Ssingh: [V:03+1 C:03+2] magru: cache-(text|upload): switch to LE certs [puppet] - 10https://gerrit.wikimedia.org/r/1324355 (owner: 10Ssingh) [17:07:30] (03CR) 10Scott French: [C:03+1] envoy-future: Update envoy-future to 1.39.0. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1324312 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [17:07:44] (03PS1) 10Ladsgroup: Move FR config from IS.php to a dedicated file [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324361 [17:09:30] 10ops-codfw, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Noisy alerts: investigate fstrim.service failure on dse-k8s-wdqs2001 (new Supermicro host) - https://phabricator.wikimedia.org/T434299#12204300 (10Jhancock.wm) I've never seen it before. I did check the version of the bios and the... [17:11:43] !log sukhe@cp7005:~$ sudo puppet agent -tv [17:11:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:11:59] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-master1004.eqiad.wmnet with OS bookworm [17:13:21] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp7005 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-09-04 20:06:43 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:14:44] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [17:15:52] (03CR) 10Ladsgroup: "CI says it's noop." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324361 (owner: 10Ladsgroup) [17:16:04] (03PS1) 10Ladsgroup: Remove $wmg = $wg hacks in CentralAuth [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324363 (https://phabricator.wikimedia.org/T119117) [17:17:16] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324361 (owner: 10Ladsgroup) [17:17:17] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324363 (https://phabricator.wikimedia.org/T119117) (owner: 10Ladsgroup) [17:18:22] (03Merged) 10jenkins-bot: Move FR config from IS.php to a dedicated file [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324361 (owner: 10Ladsgroup) [17:18:25] (03Merged) 10jenkins-bot: Remove $wmg = $wg hacks in CentralAuth [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324363 (https://phabricator.wikimedia.org/T119117) (owner: 10Ladsgroup) [17:18:42] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1324361|Move FR config from IS.php to a dedicated file]], [[gerrit:1324363|Remove $wmg = $wg hacks in CentralAuth (T119117)]] [17:18:46] T119117: Get rid of $wg = $wmg hack - https://phabricator.wikimedia.org/T119117 [17:19:32] !log sudo cumin "A:cp-magru" "run-puppet-agent --enable 'merging CR 1324355'" [17:19:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:21:21] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp7007 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-09-04 20:06:43 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:22:21] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp7006 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-09-04 20:06:43 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:22:21] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp7004 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-09-04 20:06:43 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:22:21] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp7003 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-09-04 20:06:43 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:22:21] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp7008 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-09-04 20:06:43 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:22:21] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp7002 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-09-04 20:06:43 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:22:22] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp7001 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-09-04 20:06:43 +0000 (expires in 24 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:22:28] cool [17:23:10] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1324361|Move FR config from IS.php to a dedicated file]], [[gerrit:1324363|Remove $wmg = $wg hacks in CentralAuth (T119117)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:23:28] (03PS2) 10Jdlrobson: Disable wgMFCustomSiteModules on English Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319804 (https://phabricator.wikimedia.org/T375538) [17:24:58] RESOLVED: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:26:58] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [17:29:53] (03PS1) 10Jforrester: [WikiLambda] Log the …Orchestrator and …AbstractClient channels too [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324368 [17:31:05] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324361|Move FR config from IS.php to a dedicated file]], [[gerrit:1324363|Remove $wmg = $wg hacks in CentralAuth (T119117)]] (duration: 12m 23s) [17:31:09] T119117: Get rid of $wg = $wmg hack - https://phabricator.wikimedia.org/T119117 [17:31:26] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Degraded RAID on an-presto1013 - https://phabricator.wikimedia.org/T433030#12204397 (10VRiley-WMF) 05Open→03In progress Referring to https://phabricator.wikimedia.org/T408065 that these drives can be replaced without asking. W... [17:34:43] !log sukhe@cumin1003 START - Cookbook sre.hosts.reimage for host dns6001.wikimedia.org with OS trixie [17:36:01] !log sukhe@cumin1003 START - Cookbook sre.hosts.reimage for host dns5003.wikimedia.org with OS trixie [17:38:35] PROBLEM - BFD status on asw1-b12-drmrs.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [17:39:06] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti7001.magru.wmnet [17:39:10] FIRING: [2x] BFDdown: BFD session down between asw1-b12-drmrs and 185.15.58.5 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b12-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:39:24] (03PS1) 10Ladsgroup: Enable desktop lazy loading on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324369 (https://phabricator.wikimedia.org/T148047) [17:40:08] BFD alerts expected [17:42:55] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@drmrs - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:43:27] expected, dns6001 reimage [17:43:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324369 (https://phabricator.wikimedia.org/T148047) (owner: 10Ladsgroup) [17:44:10] FIRING: [6x] BFDdown: BFD session down between asw1-b12-drmrs and 185.15.58.5 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:44:38] (03PS3) 10STran: Document cusi_case_property table [puppet] - 10https://gerrit.wikimedia.org/r/1324266 (https://phabricator.wikimedia.org/T433853) [17:44:44] FIRING: [5x] JobUnavailable: Reduced availability for job haproxy in ops@drmrs - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:44:48] (03CR) 10Ladsgroup: [V:03+2 C:03+2] Document cusi_case_property table [puppet] - 10https://gerrit.wikimedia.org/r/1324266 (https://phabricator.wikimedia.org/T433853) (owner: 10STran) [17:45:08] (03PS2) 10Ladsgroup: mariadb: Mark cx_corpora as private [puppet] - 10https://gerrit.wikimedia.org/r/1324309 (https://phabricator.wikimedia.org/T415219) [17:45:14] (03CR) 10Ladsgroup: [V:03+2 C:03+2] mariadb: Mark cx_corpora as private [puppet] - 10https://gerrit.wikimedia.org/r/1324309 (https://phabricator.wikimedia.org/T415219) (owner: 10Ladsgroup) [17:45:14] (03Merged) 10jenkins-bot: Enable desktop lazy loading on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324369 (https://phabricator.wikimedia.org/T148047) (owner: 10Ladsgroup) [17:45:31] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1324369|Enable desktop lazy loading on group0 (T148047)]] [17:45:36] T148047: Enable native image lazy loading on desktop - https://phabricator.wikimedia.org/T148047 [17:45:56] (03PS4) 10Daimona Eaytoy: tables-catalog: Add ce_invitation_list_articles [puppet] - 10https://gerrit.wikimedia.org/r/1290814 (https://phabricator.wikimedia.org/T426102) [17:46:02] (03CR) 10Ladsgroup: [V:03+2 C:03+2] tables-catalog: Add ce_invitation_list_articles [puppet] - 10https://gerrit.wikimedia.org/r/1290814 (https://phabricator.wikimedia.org/T426102) (owner: 10Daimona Eaytoy) [17:47:33] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1324369|Enable desktop lazy loading on group0 (T148047)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:48:56] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [17:49:16] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7001.magru.wmnet [17:49:21] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7001.magru.wmnet [17:49:30] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Degraded RAID on an-presto1013 - https://phabricator.wikimedia.org/T433030#12204510 (10VRiley-WMF) 05In progress→03Resolved Drive has been replaced and currently is rebuilding. [17:49:44] FIRING: [5x] JobUnavailable: Reduced availability for job haproxy in ops@drmrs - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:49:46] !log ladsgroup@cumin1003 START - Cookbook sre.mysql.sanitarium_restart [17:49:46] !log ladsgroup@cumin1003 END (FAIL) - Cookbook sre.mysql.sanitarium_restart (exit_code=99) [17:49:56] !log ladsgroup@cumin1003 START - Cookbook sre.mysql.sanitarium_restart [17:52:33] !log sukhe@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on dns6001.wikimedia.org with reason: host reimage [17:53:02] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324369|Enable desktop lazy loading on group0 (T148047)]] (duration: 07m 31s) [17:53:07] T148047: Enable native image lazy loading on desktop - https://phabricator.wikimedia.org/T148047 [17:58:15] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns6001.wikimedia.org with reason: host reimage [17:59:48] (03PS2) 10Aleksandar Mastilovic: Update the blunderbuss chart to the latest Docker image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322970 (https://phabricator.wikimedia.org/T434344) [18:00:05] brennen and jnuche: That opportune time for a MediaWiki train - Utc-7+Utc-0 Version deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T1800). [18:00:17] (03PS1) 10Jdlrobson: Phase 4 of legal footer deployment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324370 (https://phabricator.wikimedia.org/T432796) [18:00:17] o/ [18:01:21] !log ladsgroup@cumin1003 END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0) [18:02:31] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Degraded RAID on an-presto1013 - https://phabricator.wikimedia.org/T433030#12204558 (10RobH) 05Resolved→03Open The raid is still in degraded status, we should keep this task open until its successful. Related: failed drive on... [18:04:11] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Degraded RAID on an-presto1013 - https://phabricator.wikimedia.org/T433030#12204563 (10RobH) Actually, I'm torn on this. R > aw Size: 3.638 TB [0x1d1c0beb0 Sectors] > Non Coerced Size: 3.637 TB [0x1d1b0beb0 Sectors] > Coerced Si... [18:05:28] !log import trafficserver-9.2.15~deb13+wmf1 into trixie-wikimedia (T434478) [18:05:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:05:32] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1007 - https://phabricator.wikimedia.org/T434505#12204573 (10RobH) an-presto1007 is from the exact same order batch of an-presto1013, which recently had a disk die via T433030, and a replacement disk ordered via T433625. An order task will be linked in... [18:05:32] !log sukhe@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on dns5003.wikimedia.org with reason: host reimage [18:06:35] !log 1.47.0-wmf.15 train status (T430834) - no current blockers, rolling to group0 [18:06:38] !log brett@cumin2002 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp7001.magru.wmnet} and A:cp - 9.2.15 Upgrade () [18:06:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:06:39] T430834: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834 [18:06:52] (03PS1) 10TrainBranchBot: group0 to 1.47.0-wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324371 (https://phabricator.wikimedia.org/T430834) [18:06:55] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by brennen@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324371 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [18:06:56] PROBLEM - Recursive DNS on 185.15.58.5 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [18:06:56] PROBLEM - Recursive DNS on 2a02:ec80:600:1:185:15:58:5 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [18:07:50] (03Merged) 10jenkins-bot: group0 to 1.47.0-wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324371 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [18:07:55] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@eqsin - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:08:03] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1007 - https://phabricator.wikimedia.org/T434505#12204624 (10RobH) [18:09:21] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns5003.wikimedia.org with reason: host reimage [18:10:34] (03CR) 10AOkoth: [C:03+2] miscweb: add pod_annotiations to deployment tpl [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320943 (https://phabricator.wikimedia.org/T433588) (owner: 10AOkoth) [18:12:03] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [18:12:43] !log brett@cumin2002 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp7001.magru.wmnet} and A:cp - 9.2.15 Upgrade () [18:13:45] (03Merged) 10jenkins-bot: miscweb: add pod_annotiations to deployment tpl [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320943 (https://phabricator.wikimedia.org/T433588) (owner: 10AOkoth) [18:13:55] !log brennen@deploy1003 rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.15 refs T430834 [18:13:59] T430834: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834 [18:14:19] !log brett@cumin2002 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp7009.magru.wmnet} and A:cp - 9.2.15 Upgrade () [18:16:01] PROBLEM - Recursive DNS on 103.102.166.10 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [18:17:40] (03PS1) 10Ssingh: dns5003/dns6001: use new pdns-rec config [puppet] - 10https://gerrit.wikimedia.org/r/1324374 (https://phabricator.wikimedia.org/T401832) [18:18:45] (03CR) 10Ssingh: [C:03+2] dns5003/dns6001: use new pdns-rec config [puppet] - 10https://gerrit.wikimedia.org/r/1324374 (https://phabricator.wikimedia.org/T401832) (owner: 10Ssingh) [18:19:51] !log brett@cumin2002 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp7009.magru.wmnet} and A:cp - 9.2.15 Upgrade () [18:22:55] FIRING: [4x] JobUnavailable: Reduced availability for job haproxy in ops@eqsin - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:26:01] PROBLEM - Recursive DNS on 2001:df2:e500:1:103:102:166:10 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [18:34:33] (03CR) 10Cwhite: [C:03+2] scap: Add optional logstash credentials file [puppet] - 10https://gerrit.wikimedia.org/r/1321653 (https://phabricator.wikimedia.org/T434114) (owner: 10Ahmon Dancy) [18:34:59] RECOVERY - Recursive DNS on 2001:df2:e500:1:103:102:166:10 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [18:34:59] RECOVERY - Recursive DNS on 103.102.166.10 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [18:41:19] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: pki1002 became unresponsive causing several hosts to alert on failed puppet runs. - https://phabricator.wikimedia.org/T434268#12204798 (10Jhancock.wm) 2026-08-06 23:43:22 CPU0704 CPU 1 machine check error detected. been seeing these a lot. th... [18:45:53] RECOVERY - Recursive DNS on 2a02:ec80:600:1:185:15:58:5 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [18:45:53] RECOVERY - Recursive DNS on 185.15.58.5 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [18:46:37] RECOVERY - BFD status on asw1-b12-drmrs.mgmt is OK: UP: 7 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [18:47:55] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:49:10] FIRING: [6x] BFDdown: BFD session down between asw1-b12-drmrs and 185.15.58.5 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:49:44] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:54:10] RESOLVED: [6x] BFDdown: BFD session down between asw1-b12-drmrs and 185.15.58.5 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:55:07] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6001.wikimedia.org with OS trixie [18:59:29] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns5003.wikimedia.org with OS trixie [19:01:50] going to roll back train for a blocker. [19:02:07] (03PS1) 10TrainBranchBot: testwikis to 1.47.0-wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324378 (https://phabricator.wikimedia.org/T430834) [19:02:10] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by brennen@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324378 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [19:03:04] (03Merged) 10jenkins-bot: testwikis to 1.47.0-wmf.15 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324378 (https://phabricator.wikimedia.org/T430834) (owner: 10TrainBranchBot) [19:03:26] !log brennen@deploy1003 Started scap sync-world: testwikis to 1.47.0-wmf.15 refs T430834 [19:03:31] T430834: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834 [19:04:07] (03PS5) 10Scott French: trafficserver: Support testwiki pretrain routing in XWD [puppet] - 10https://gerrit.wikimedia.org/r/1304189 (https://phabricator.wikimedia.org/T427666) [19:04:14] !log sukhe@cumin1003 START - Cookbook sre.hosts.remove-downtime for dns5003.wikimedia.org [19:04:15] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns5003.wikimedia.org [19:04:41] !log sukhe@cumin1003 START - Cookbook sre.hosts.remove-downtime for dns6001.wikimedia.org [19:04:42] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns6001.wikimedia.org [19:09:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 17.4% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:10:07] !log brennen@deploy1003 Finished scap sync-world: testwikis to 1.47.0-wmf.15 refs T430834 (duration: 06m 41s) [19:10:12] T430834: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834 [19:14:16] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.18s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [19:19:15] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.082s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [19:19:45] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 923.1ms - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [19:24:31] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 857.6ms - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [19:25:38] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:42:58] (03PS1) 10BBlack: Un-OCSP for Traffic puppetization [puppet] - 10https://gerrit.wikimedia.org/r/1324380 [19:47:34] jhancock@cumin1003 reimage (PID 2424350) is awaiting input [19:54:01] !log jhancock@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage [19:59:04] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: It is that lovely time of the day again! You are hereby commanded to deploy UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T2000). [20:00:05] No Gerrit patches in the queue for this window AFAICS. [20:06:29] (03PS1) 10DLynch: Editcheck: ecenable causing errors when editCheckTagging is active [extensions/VisualEditor] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1324384 (https://phabricator.wikimedia.org/T434403) [20:06:44] (03PS1) 10DLynch: Editcheck: ecenable causing errors when editCheckTagging is active [extensions/VisualEditor] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324385 (https://phabricator.wikimedia.org/T434403) [20:06:55] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, August 11 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [extensions/VisualEditor] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1324384 (https://phabricator.wikimedia.org/T434403) (owner: 10DLynch) [20:07:01] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, August 11 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [extensions/VisualEditor] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324385 (https://phabricator.wikimedia.org/T434403) (owner: 10DLynch) [20:07:20] Now there's two patches. I can get them myself. [20:07:44] FIRING: KubernetesDeploymentUnavailableReplicas: ... [20:07:44] Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ... [20:07:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [20:08:07] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c3-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T434547#12205276 (10VRiley-WMF) a:03VRiley-WMF [20:08:16] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c3-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T434547#12205277 (10VRiley-WMF) 05Open→03Resolved rebalanced power [20:09:15] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 823.3ms - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [20:09:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/VisualEditor] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1324384 (https://phabricator.wikimedia.org/T434403) (owner: 10DLynch) [20:09:28] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/VisualEditor] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324385 (https://phabricator.wikimedia.org/T434403) (owner: 10DLynch) [20:11:34] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1178.eqiad.wmnet - https://phabricator.wikimedia.org/T433471#12205282 (10VRiley-WMF) a:03VRiley-WMF [20:11:57] FIRING: ProbeDown: Service chart-renderer:30443 has failed probes (http_chart-renderer_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#chart-renderer:30443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:12:06] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1178.eqiad.wmnet - https://phabricator.wikimedia.org/T433471#12205284 (10VRiley-WMF) [20:12:25] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1178.eqiad.wmnet - https://phabricator.wikimedia.org/T433471#12205285 (10VRiley-WMF) 05Open→03Resolved This is completed [20:14:15] FIRING: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 20.43% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:14:15] FIRING: [2x] MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-int releases routed via main (k8s) 852.5ms - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [20:14:45] !log jhancock@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" [20:15:32] !log jhancock@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003" [20:15:33] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl2006.codfw.wmnet with OS trixie [20:15:46] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12205306 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-ctrl2006.codfw.... [20:15:56] ...is it impossible to backport stuff onto .14 without finding and backporting the composer.json `block-insecure` issue as well? [20:16:57] RESOLVED: ProbeDown: Service chart-renderer:30443 has failed probes (http_chart-renderer_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#chart-renderer:30443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:18:03] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, and 2 others: decommission db1177.eqiad.wmnet - https://phabricator.wikimedia.org/T433474#12205314 (10VRiley-WMF) a:03VRiley-WMF [20:18:28] !log jhancock@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie [20:18:39] (03Merged) 10jenkins-bot: Editcheck: ecenable causing errors when editCheckTagging is active [extensions/VisualEditor] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1324384 (https://phabricator.wikimedia.org/T434403) (owner: 10DLynch) [20:18:42] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12205336 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-ctrl2006.co... [20:18:45] (03CR) 10CI reject: [V:04-1] Editcheck: ecenable causing errors when editCheckTagging is active [extensions/VisualEditor] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324385 (https://phabricator.wikimedia.org/T434403) (owner: 10DLynch) [20:18:47] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, and 2 others: decommission db1177.eqiad.wmnet - https://phabricator.wikimedia.org/T433474#12205337 (10VRiley-WMF) [20:19:15] FIRING: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 20.43% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:19:15] RESOLVED: [2x] MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-int releases routed via main (k8s) 830.4ms - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [20:19:45] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, and 2 others: decommission db1177.eqiad.wmnet - https://phabricator.wikimedia.org/T433474#12205381 (10VRiley-WMF) 05Open→03Resolved this is completed [20:20:40] (03PS1) 10DLynch: build: Updating js-yaml to 3.15.1, 4.3.1 [extensions/VisualEditor] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324386 [20:26:02] (03PS2) 10DLynch: Editcheck: ecenable causing errors when editCheckTagging is active [extensions/VisualEditor] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324385 (https://phabricator.wikimedia.org/T434403) [20:27:49] (03CR) 10DLynch: "recheck" [extensions/VisualEditor] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324385 (https://phabricator.wikimedia.org/T434403) (owner: 10DLynch) [20:29:02] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12205416 (10Jhancock.wm) it's definitely the NIC not initializing in time to catch the reboot. checking to see if there's a firmware... [20:29:15] RESOLVED: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 21.84% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:31:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/VisualEditor] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324386 (owner: 10DLynch) [20:32:17] hi Jdlrobson :D [20:32:44] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [20:32:44] Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ... [20:32:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [20:33:37] (03Merged) 10jenkins-bot: build: Updating js-yaml to 3.15.1, 4.3.1 [extensions/VisualEditor] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324386 (owner: 10DLynch) [20:35:01] !log kemayo@deploy1003 Started scap sync-world: Backport for [[gerrit:1324386|build: Updating js-yaml to 3.15.1, 4.3.1]] [20:37:03] !log kemayo@deploy1003 kemayo: Backport for [[gerrit:1324386|build: Updating js-yaml to 3.15.1, 4.3.1]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:37:47] !log jhancock@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage [20:38:32] !log kemayo@deploy1003 kemayo: Continuing with deployment [20:42:00] (03Merged) 10jenkins-bot: Editcheck: ecenable causing errors when editCheckTagging is active [extensions/VisualEditor] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1324385 (https://phabricator.wikimedia.org/T434403) (owner: 10DLynch) [20:42:37] !log kemayo@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324386|build: Updating js-yaml to 3.15.1, 4.3.1]] (duration: 07m 36s) [20:42:57] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage [20:43:21] !log kemayo@deploy1003 Started scap sync-world: Backport for [[gerrit:1324384|Editcheck: ecenable causing errors when editCheckTagging is active (T434403)]], [[gerrit:1324385|Editcheck: ecenable causing errors when editCheckTagging is active (T434403)]] [20:43:26] T434403: VE slowed down in Chrome by "Recalculate Style" - https://phabricator.wikimedia.org/T434403 [20:45:25] !log kemayo@deploy1003 kemayo: Backport for [[gerrit:1324384|Editcheck: ecenable causing errors when editCheckTagging is active (T434403)]], [[gerrit:1324385|Editcheck: ecenable causing errors when editCheckTagging is active (T434403)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:46:17] !log kemayo@deploy1003 kemayo: Continuing with deployment [20:50:20] !log kemayo@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324384|Editcheck: ecenable causing errors when editCheckTagging is active (T434403)]], [[gerrit:1324385|Editcheck: ecenable causing errors when editCheckTagging is active (T434403)]] (duration: 06m 58s) [20:50:24] T434403: VE slowed down in Chrome by "Recalculate Style" - https://phabricator.wikimedia.org/T434403 [20:52:11] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1007 - https://phabricator.wikimedia.org/T434505#12205490 (10VRiley-WMF) a:03VRiley-WMF [20:52:21] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1007 - https://phabricator.wikimedia.org/T434505#12205491 (10VRiley-WMF) Will await on HDD [20:53:24] 06SRE, 10Wikimedia-Mailing-lists: Mailing list for Wikimedia North America Hub - https://phabricator.wikimedia.org/T434601#12205492 (10Ladsgroup) 05Open→03Resolved Went with discussion mailing list, if you want announce-only, you can change the settings https://lists.wikimedia.org/postorius/lists/wikim... [20:53:27] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: pki1002 became unresponsive causing several hosts to alert on failed puppet runs. - https://phabricator.wikimedia.org/T434268#12205495 (10VRiley-WMF) @andrea.denisse is there an estimated day that we would be able to this? [20:59:13] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl2006.codfw.wmnet with OS trixie [20:59:27] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12205514 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-ctrl2006.codfw.... [21:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260811T2100) [21:01:22] o/ [21:02:00] \o [21:02:11] ok! getting started with the deploy [21:02:15] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324370 (https://phabricator.wikimedia.org/T432796) (owner: 10Jdlrobson) [21:02:16] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319804 (https://phabricator.wikimedia.org/T375538) (owner: 10Jdlrobson) [21:02:29] (I have another config change i'll put out at the same time) [21:03:20] (03Merged) 10jenkins-bot: Phase 4 of legal footer deployment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324370 (https://phabricator.wikimedia.org/T432796) (owner: 10Jdlrobson) [21:03:24] (03Merged) 10jenkins-bot: Disable wgMFCustomSiteModules on English Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319804 (https://phabricator.wikimedia.org/T375538) (owner: 10Jdlrobson) [21:03:43] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1324370|Phase 4 of legal footer deployment (T432796)]], [[gerrit:1319804|Disable wgMFCustomSiteModules on English Wikipedia (T375538)]] [21:03:49] T432796: [Legal footer] Phase 4 deployment - https://phabricator.wikimedia.org/T432796 [21:03:49] T375538: Set wgMFCustomSiteModules to false for enwiki - https://phabricator.wikimedia.org/T375538 [21:05:57] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1324370|Phase 4 of legal footer deployment (T432796)]], [[gerrit:1319804|Disable wgMFCustomSiteModules on English Wikipedia (T375538)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:07:10] ok @Izno I'm testing in debug mode now to see how it behaves. [21:07:45] it's working for me in same! [21:07:57] Izno: it looks like https://en.wikipedia.org/w/load.php?modules=site.styles&only=styles&skin=minerva gives different CSS in debug mode so that's promising [21:08:21] and the relevant CSS shows up in console in [[World of Warcraft]] [21:09:07] does withJS work [21:09:23] have to find a page for that heh [21:11:29] yes it does [21:11:35] https://en.wikipedia.org/wiki/Wikipedia:Get_my_IP_address?withJS=MediaWiki:Get-my-ip.js&useformat=mobile [21:11:41] anon page view CSS goes from 23.6kb to 26kb [21:11:52] is that expected until we do some cleanup? [21:11:53] which is about what you had estimated iirc [21:12:07] or is that going to be the new norm? [21:12:15] Minerva will have some removals [21:12:43] so it will decrease some [21:12:50] ok will we kick those cleanups off today? I can monitor real world data post change to see if this causes any visible user impact [21:12:53] should i go ahead and sync? [21:12:58] yes, I'll be doing Minerva today [21:13:10] and yes, I think syncing makes sense [21:13:25] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:13:54] there seems like a lot of CSS relating to references but I'm not seeing any difference in the output [21:14:02] e.g. `.mw-parser-output a.external.autonumber::after{word-wrap:break-word}}` [21:14:22] don't think that's ours [21:14:31] oh wait.. that's the print stylesheet [21:14:44] I guess we are including https://en.wikipedia.org/wiki/MediaWiki:Print.css now? [21:14:44] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [21:14:53] ah yes [21:14:55] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12205686 (10VRiley-WMF) This is very strange. I just checked out this unit and it seems like it's up and active, however the iDRAC is now lighting up at all but nothing was moved or changed.... [21:15:02] ok proceeding [21:15:05] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [21:16:06] Hey all - I have a security patch I’d like to deploy, if there aren’t any objections. Late backport and web deploy windows didn’t seem busy… [21:19:09] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324370|Phase 4 of legal footer deployment (T432796)]], [[gerrit:1319804|Disable wgMFCustomSiteModules on English Wikipedia (T375538)]] (duration: 15m 26s) [21:19:16] T432796: [Legal footer] Phase 4 deployment - https://phabricator.wikimedia.org/T432796 [21:19:17] T375538: Set wgMFCustomSiteModules to false for enwiki - https://phabricator.wikimedia.org/T375538 [21:20:44] ok all done. @Izno is all looking good your end without the debug extension? [21:20:52] (if so I'll hand off to sbassett ) [21:21:00] (03CR) 10Bking: [C:03+2] Update the blunderbuss chart to the latest Docker image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322970 (https://phabricator.wikimedia.org/T434344) (owner: 10Aleksandar Mastilovic) [21:21:47] Jdlrobson: yup, looks good [21:21:59] sbassett: okay all yours! Izno we can switch back to discord :) [21:22:07] o/ [21:22:43] thanks [21:29:17] !log Deployed security fix for T434521 (wmf.14) [21:29:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:37:08] !log Deployed security fix for T434521 (wmf.15) [21:37:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:46:36] (03PS6) 10Tsevener: Point Test Wiki to new docroot [puppet] - 10https://gerrit.wikimedia.org/r/1315141 (https://phabricator.wikimedia.org/T432412) [21:53:09] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12205815 (10VRiley-WMF) @jcrespo Okay, so I just took a look at it, and there was a overheating incident with CPU 2 that the server was complaining about. I was able to find some thermal past... [21:54:22] (03CR) 10Lerickson: [C:03+1] "much better, thanks for the fix!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1323289 (https://phabricator.wikimedia.org/T434321) (owner: 10Trueg) [21:55:10] FIRING: BFDdown: BFD session down between cr2-esams and fe80::ee38:7300:17e8:9c56 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [21:55:55] (03CR) 10Lerickson: [C:03+2] WDQS V2: add http_proxy environment variable to the qlever container. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322799 (https://phabricator.wikimedia.org/T434321) (owner: 10Lerickson) [22:00:10] RESOLVED: BFDdown: BFD session down between cr2-esams and fe80::ee38:7300:17e8:9c56 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [22:03:10] 10ops-eqiad, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T433348#12205845 (10VRiley-WMF) @bking I don't know if there has been an update on this? Or if there is someone else to reach out to, I'd be more than ha... [22:11:30] (03PS8) 10Tsevener: Point Test Wiki to new docroot [puppet] - 10https://gerrit.wikimedia.org/r/1315141 (https://phabricator.wikimedia.org/T432412) [22:19:03] (03CR) 10Tsevener: "Thanks @cgoubert@wikimedia.org. I have fixed the line length and added some httpbb tests. I modified en.wikipedia.org to also recognize th" [puppet] - 10https://gerrit.wikimedia.org/r/1315141 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [22:39:53] (03PS1) 10Ladsgroup: Remove $wmg = $wg hacks in Collection [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324409 (https://phabricator.wikimedia.org/T119117) [22:49:44] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:25:38] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:41:17] (03PS3) 10Eevans: restbase: Set storage compatibility to UPGRADING [puppet] - 10https://gerrit.wikimedia.org/r/1315185 (https://phabricator.wikimedia.org/T433028) [23:41:17] (03PS3) 10Eevans: restbase: Set storage compatibility to NONE [puppet] - 10https://gerrit.wikimedia.org/r/1315186 (https://phabricator.wikimedia.org/T433028) [23:41:33] 06SRE, 10ChangeProp, 10MW-on-K8s, 06serviceops-deprecated, 10WMF-JobQueue: Alter changeprop chart to use the service mesh - https://phabricator.wikimedia.org/T360625#12206060 (10Scott_French) Connecting some historical dots: This is likely happen, in order to support diversion of testwiki traffic to... [23:42:25] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1324421 [23:42:25] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1324421 (owner: 10TrainBranchBot) [23:43:28] (03CR) 10Eevans: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1315185 (https://phabricator.wikimedia.org/T433028) (owner: 10Eevans) [23:43:34] (03CR) 10Eevans: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1315186 (https://phabricator.wikimedia.org/T433028) (owner: 10Eevans) [23:47:54] (03PS4) 10Eevans: restbase: Set storage compatibility to UPGRADING [puppet] - 10https://gerrit.wikimedia.org/r/1315185 (https://phabricator.wikimedia.org/T433028) [23:47:54] (03PS4) 10Eevans: restbase: Set storage compatibility to NONE [puppet] - 10https://gerrit.wikimedia.org/r/1315186 (https://phabricator.wikimedia.org/T433028) [23:48:08] (03CR) 10Eevans: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1315185 (https://phabricator.wikimedia.org/T433028) (owner: 10Eevans) [23:48:13] (03CR) 10Eevans: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1315186 (https://phabricator.wikimedia.org/T433028) (owner: 10Eevans) [23:54:12] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1324421 (owner: 10TrainBranchBot) [23:54:16] (03Abandoned) 10Jforrester: usage: Don't try to record Function usage for a target that isn't a ZID [extensions/WikiLambda] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1322752 (https://phabricator.wikimedia.org/T434194) (owner: 10Jforrester)