[00:07:45] (03PS6) 10Andrea Denisse: grafana: Add tamirsuliman-weathermap-panel to sandbox [puppet] - 10https://gerrit.wikimedia.org/r/1338056 (https://phabricator.wikimedia.org/T436075) [00:08:03] (03PS1) 10Dzahn: delete deprecated .m. records for arbcom and sysop special wikis [dns] - 10https://gerrit.wikimedia.org/r/1338313 [00:08:23] (03CR) 10Dzahn: [C:03+2] create wikipedia-ar-arbcom.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn) [00:08:51] !log dzahn@dns1004 START - running authdns-update [00:08:58] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1345.eqiad.wmnet with OS trixie [00:09:26] (03CR) 10Dzahn: create wikipedia-ar-arbcom.wikimedia.org (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1338267 (https://phabricator.wikimedia.org/T437403) (owner: 10Dzahn) [00:11:05] !log dzahn@dns1004 END - running authdns-update [00:12:08] (03CR) 10Andrea Denisse: [C:03+2] grafana: Add tamirsuliman-weathermap-panel to sandbox [puppet] - 10https://gerrit.wikimedia.org/r/1338056 (https://phabricator.wikimedia.org/T436075) (owner: 10Andrea Denisse) [00:12:56] (03PS2) 10Dzahn: delete deprecated .m. records for arbcom and sysop special wikis [dns] - 10https://gerrit.wikimedia.org/r/1338313 (https://phabricator.wikimedia.org/T214998) [00:13:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:14:52] (03PS3) 10Dzahn: delete deprecated .m. records for arbcom and sysop special wikis [dns] - 10https://gerrit.wikimedia.org/r/1338313 (https://phabricator.wikimedia.org/T214998) [00:17:15] (03CR) 10Andrea Denisse: [C:03+2] kubernetes: Switch to GitLab origin for LibreNMS [puppet] - 10https://gerrit.wikimedia.org/r/1338078 (https://phabricator.wikimedia.org/T436578) (owner: 10Andrea Denisse) [00:20:30] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1345.eqiad.wmnet [00:20:32] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1345.eqiad.wmnet [00:20:34] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1345.eqiad.wmnet [01:11:30] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1338344 [01:11:30] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1338344 (owner: 10TrainBranchBot) [01:20:49] (03PS1) 10Andrea Denisse: grafana: Add grafana-plugins to grafana-server suscription [puppet] - 10https://gerrit.wikimedia.org/r/1338350 (https://phabricator.wikimedia.org/T436056) [01:20:50] (03CR) 10Andrea Denisse: [V:03+1] "PCC results: https://puppet-compiler.wmflabs.org/output/1338350/9399/" [puppet] - 10https://gerrit.wikimedia.org/r/1338350 (https://phabricator.wikimedia.org/T436056) (owner: 10Andrea Denisse) [01:21:21] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1338344 (owner: 10TrainBranchBot) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T0200) [02:00:30] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:08:09] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 38s) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:56] (03PS1) 10RLazarus: Drop support for Python 3.9 [software/httpbb] - 10https://gerrit.wikimedia.org/r/1338398 (https://phabricator.wikimedia.org/T434337) [02:19:32] (03CR) 10RLazarus: "I won't bother cutting a new release for this, since there's nothing user-perceptible in it." [software/httpbb] - 10https://gerrit.wikimedia.org/r/1338398 (https://phabricator.wikimedia.org/T434337) (owner: 10RLazarus) [02:38:54] (03CR) 10Bhsd: [C:04-1] "Could you confirm that browsers not supporting CSS highlighting will fall back to the old behavior?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [03:05:49] (03PS1) 10DErenrich: Enable discord preview extension code on testwiki (tk 2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338439 (https://phabricator.wikimedia.org/T437344) [03:06:43] (03CR) 10CI reject: [V:04-1] Enable discord preview extension code on testwiki (tk 2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338439 (https://phabricator.wikimedia.org/T437344) (owner: 10DErenrich) [03:08:13] (03PS2) 10DErenrich: Enable discord preview extension code on testwiki (tk 2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338439 (https://phabricator.wikimedia.org/T437344) [03:09:06] (03CR) 10CI reject: [V:04-1] Enable discord preview extension code on testwiki (tk 2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338439 (https://phabricator.wikimedia.org/T437344) (owner: 10DErenrich) [03:10:26] (03CR) 10MusikAnimal: "I have no means to test it, but I presume they will no longer get highlighting at all. That is OK, I think. My impression is the Highlight" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [03:28:37] FIRING: [2x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [04:13:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:03:40] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tool-server' for release 'main' . [05:04:30] !log kevinbazira@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tool-server' for release 'main' . [05:27:56] RECOVERY - Backup freshness on backup1014 is OK: Fresh: 143 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T0600) [06:00:05] marostegui, cezmunsta, and federico3: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Primary database switchover deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T0600). [06:05:30] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12306378 (10BTullis) [06:05:31] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T437170#12306377 (10BTullis) [06:07:56] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T437170#12306392 (10BTullis) This one keeps dropping disks. I've added the parent task where I'm doing a larger investigation into what seems like a batch of serve... [06:37:51] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [06:38:49] !log filippo@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1075 [06:39:27] !log filippo@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1075 [06:42:06] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1075 cloud-private - filippo@cumin1003" [06:42:09] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1075 cloud-private - filippo@cumin1003" [06:42:09] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [06:43:29] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1065.eqiad.wmnet with OS trixie [06:59:26] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage [07:00:05] Amir1, urbanecm, and awight: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T0700). [07:00:05] No Gerrit patches in the queue for this window AFAICS. [07:03:00] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage [07:10:24] (03PS1) 10Slyngshede: CAS 7.3.8.3 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1338659 [07:11:54] (03CR) 10Slyngshede: [V:03+2 C:03+2] CAS 7.3.8.3 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1338659 (owner: 10Slyngshede) [07:28:20] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [07:28:37] FIRING: [2x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [07:29:09] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [07:32:11] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-dev: apply [07:32:14] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-dev: apply [07:32:27] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-analytics-product: apply [07:32:31] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-analytics-product: apply [07:32:34] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-analytics-test: apply [07:32:38] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-analytics-test: apply [07:32:42] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply [07:32:46] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply [07:33:01] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-experiment-platform: apply [07:33:05] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-experiment-platform: apply [07:33:10] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-fr-tech: apply [07:33:14] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-fr-tech: apply [07:33:17] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply [07:33:21] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply [07:33:24] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-ml: apply [07:33:28] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-ml: apply [07:33:31] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-platform-eng: apply [07:33:35] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-platform-eng: apply [07:33:38] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-research: apply [07:33:42] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-research: apply [07:33:45] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-search: apply [07:33:49] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-search: apply [07:33:52] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-sre: apply [07:33:56] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-sre: apply [07:34:00] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply [07:34:04] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply [07:34:07] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-wikidata: apply [07:34:11] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-wikidata: apply [07:34:15] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-wmde: apply [07:34:19] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-wmde: apply [07:34:22] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply [07:34:25] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply [07:34:29] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply [07:34:33] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply [07:34:37] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply [07:34:40] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply [07:39:53] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1065.eqiad.wmnet with OS trixie [07:47:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:53:55] (03CR) 10MusikAnimal: "That said we could put it in the enwiki-only guard… I will wait for David's advice. The production VE config is hard to parse at glance, g" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [07:56:53] !log netflow2005:~$ sudo dpkg -i gnmic_0.48.0_Linux_x86_64.deb [07:56:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:58:27] !log netflow1004:~$ sudo dpkg -i gnmic_0.48.0_Linux_x86_64.deb [07:58:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:07:17] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1066.eqiad.wmnet with OS trixie [08:07:38] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1067.eqiad.wmnet with OS trixie [08:09:14] (03PS1) 10Mszwarc: UIC: Fix getOpenContext when UserCardButton.vue is clicked [extensions/CheckUser] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338705 (https://phabricator.wikimedia.org/T435299) [08:09:55] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1072.eqiad.wmnet with OS trixie [08:10:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:11:11] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1073.eqiad.wmnet with OS trixie [08:11:28] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12306593 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt1073.eqiad.wmnet with OS trixie [08:11:54] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1074.eqiad.wmnet with OS trixie [08:13:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:15:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:18:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:19:14] Unless anyone minds, I'm going to backport https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/1338705 in a few minutes [08:19:16] 06SRE, 10LDAP-Access-Requests: Grant Access to <500> for - https://phabricator.wikimedia.org/T437488#12306615 (10Aklapper) @MMilenkovic-WMF: Please also [link your LDAP account to your Phabricator account](https://phabricator.wikimedia.org/settings/panel/external/), so your 'LDAP User' account... [08:21:21] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mszwarc@deploy1003 using scap backport" [extensions/CheckUser] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338705 (https://phabricator.wikimedia.org/T435299) (owner: 10Mszwarc) [08:23:12] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1066.eqiad.wmnet with reason: host reimage [08:23:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:23:15] (03PS1) 10Filippo Giunchedi: Revert "site: temp decom for cloudvirt1075 ahead of rack move" [puppet] - 10https://gerrit.wikimedia.org/r/1338710 (https://phabricator.wikimedia.org/T435923) [08:23:37] (03PS1) 10Marostegui: check_private_data_report: Remove email [puppet] - 10https://gerrit.wikimedia.org/r/1338711 [08:23:42] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1067.eqiad.wmnet with reason: host reimage [08:23:51] (03Merged) 10jenkins-bot: UIC: Fix getOpenContext when UserCardButton.vue is clicked [extensions/CheckUser] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338705 (https://phabricator.wikimedia.org/T435299) (owner: 10Mszwarc) [08:23:55] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [08:23:58] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [08:24:14] (03CR) 10Volans: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1338710 (https://phabricator.wikimedia.org/T435923) (owner: 10Filippo Giunchedi) [08:24:32] (03CR) 10Filippo Giunchedi: [C:03+2] Revert "site: temp decom for cloudvirt1075 ahead of rack move" [puppet] - 10https://gerrit.wikimedia.org/r/1338710 (https://phabricator.wikimedia.org/T435923) (owner: 10Filippo Giunchedi) [08:24:51] !log mszwarc@deploy1003 Started scap sync-world: Backport for [[gerrit:1338705|UIC: Fix getOpenContext when UserCardButton.vue is clicked (T435299)]] [08:24:54] T435299: Enable UserInfoCard on Special:NewPagesFeed - https://phabricator.wikimedia.org/T435299 [08:25:57] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1072.eqiad.wmnet with reason: host reimage [08:26:00] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1338711 (owner: 10Marostegui) [08:26:19] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Remove email [puppet] - 10https://gerrit.wikimedia.org/r/1338711 (owner: 10Marostegui) [08:27:27] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1073.eqiad.wmnet with reason: host reimage [08:27:50] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1075.eqiad.wmnet with OS trixie [08:28:02] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1074.eqiad.wmnet with reason: host reimage [08:29:44] !log mszwarc@deploy1003 mszwarc: Backport for [[gerrit:1338705|UIC: Fix getOpenContext when UserCardButton.vue is clicked (T435299)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:29:48] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12306648 (10Marostegui) I will follow up thanks! [08:29:48] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1066.eqiad.wmnet with reason: host reimage [08:30:16] !log mszwarc@deploy1003 mszwarc: Continuing with deployment [08:31:46] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12306681 (10Marostegui) 05Resolved→03Open @VRiley-WMF this host is unreachable: ` [08:30:46] marostegui@cumin1003:~$ ping db1245 PING db1245.eqiad.wmnet (10.64.32.59) 56(84) bytes of data... [08:31:54] (03CR) 10Phuedx: [C:03+1] test-kitchen-next: Disable validate_deployment_windows [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338310 (https://phabricator.wikimedia.org/T433733) (owner: 10Santiago Faci) [08:31:55] (03CR) 10Elukey: [C:03+2] profile::docker_registry: improve the blackbox exporter's config [puppet] - 10https://gerrit.wikimedia.org/r/1338246 (owner: 10Elukey) [08:32:24] 10ops-eqiad, 06DC-Ops: Unresponsive management for db1245.mgmt:22 - https://phabricator.wikimedia.org/T437565 (10phaultfinder) 03NEW [08:32:27] (03PS1) 10Btullis: cloudnative-pg: alert on WAL archiving failures rather than staleness [alerts] - 10https://gerrit.wikimedia.org/r/1338719 (https://phabricator.wikimedia.org/T432104) [08:32:28] (03CR) 10Phuedx: [C:03+1] Test Kitchen UI: Deploy v1.5.4 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338098 (https://phabricator.wikimedia.org/T421813) (owner: 10Santiago Faci) [08:33:14] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1072.eqiad.wmnet with reason: host reimage [08:34:12] (03PS2) 10Btullis: cloudnative-pg: alert on WAL archiving failures rather than staleness [alerts] - 10https://gerrit.wikimedia.org/r/1338719 (https://phabricator.wikimedia.org/T432104) [08:34:48] !log mszwarc@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338705|UIC: Fix getOpenContext when UserCardButton.vue is clicked (T435299)]] (duration: 09m 56s) [08:34:51] T435299: Enable UserInfoCard on Special:NewPagesFeed - https://phabricator.wikimedia.org/T435299 [08:35:38] Finished deploying [08:37:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:37:19] (03CR) 10CI reject: [V:04-1] cloudnative-pg: alert on WAL archiving failures rather than staleness [alerts] - 10https://gerrit.wikimedia.org/r/1338719 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [08:37:20] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1074.eqiad.wmnet with reason: host reimage [08:40:16] (03CR) 10Santiago Faci: [C:03+2] test-kitchen-next: Disable validate_deployment_windows [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338310 (https://phabricator.wikimedia.org/T433733) (owner: 10Santiago Faci) [08:41:35] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1073.eqiad.wmnet with reason: host reimage [08:42:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.59% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:42:43] (03Merged) 10jenkins-bot: test-kitchen-next: Disable validate_deployment_windows [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338310 (https://phabricator.wikimedia.org/T433733) (owner: 10Santiago Faci) [08:43:44] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1075.eqiad.wmnet with reason: host reimage [08:44:40] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1155.eqiad.wmnet with reason: Cloning x4 [08:44:47] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [08:45:01] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [08:46:28] (03CR) 10Santiago Faci: [C:03+2] Test Kitchen UI: Deploy v1.5.4 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338098 (https://phabricator.wikimedia.org/T421813) (owner: 10Santiago Faci) [08:46:56] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1067.eqiad.wmnet with reason: host reimage [08:47:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:49:00] (03Merged) 10jenkins-bot: Test Kitchen UI: Deploy v1.5.4 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338098 (https://phabricator.wikimedia.org/T421813) (owner: 10Santiago Faci) [08:51:16] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1075.eqiad.wmnet with reason: host reimage [08:52:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:55:57] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12306764 (10elukey) @Andrew I tried to dig a bit more into it and these are the disks and their size as seen by d-i: ` sda: rota=0 s... [08:58:23] FIRING: [3x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [09:03:22] FIRING: [4x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [09:04:58] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1067.eqiad.wmnet with OS trixie [09:06:02] (03PS9) 10Blake: mediawiki: stop using lamp.deployment. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338227 (https://phabricator.wikimedia.org/T417800) [09:07:03] (03Abandoned) 10Blake: mediawiki: stop using lamp.deployment in cronjob. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338254 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [09:08:33] (03PS5) 10Slyngshede: P:tofurkey enable Tofurkey for MAGRU [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) [09:09:30] (03PS10) 10Blake: mediawiki: stop using lamp.deployment. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338227 (https://phabricator.wikimedia.org/T417800) [09:22:46] PROBLEM - Host wikikube-worker1067 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 2761.54 ms [09:22:52] RECOVERY - Host wikikube-worker1067 is UP: PING OK - Packet loss = 0%, RTA = 423.03 ms [09:24:13] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply [09:24:24] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply [09:32:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:37:10] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1074.eqiad.wmnet with OS trixie [09:37:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:38:36] (03CR) 10Klausman: [C:03+1] "LGTM. I will see to rolling it out today (coordinating with Traffic team)" [dns] - 10https://gerrit.wikimedia.org/r/1337904 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [09:39:04] (03CR) 10Volans: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1338235 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [09:40:16] (03PS1) 10Santiago Faci: Revert "Test Kitchen UI: Deploy v1.5.4 release to production" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338747 [09:41:37] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1066.eqiad.wmnet with OS trixie [09:45:49] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1072.eqiad.wmnet with OS trixie [09:53:21] !log marostegui@cumin1003 conftool action : set/pooled=no; selector: name=clouddb1024.eqiad.wmnet [09:56:13] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1075.eqiad.wmnet with OS trixie [09:58:16] (03PS1) 10Ladsgroup: Reduce the webp threshold to 70 [puppet] - 10https://gerrit.wikimedia.org/r/1338760 (https://phabricator.wikimedia.org/T431150) [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1000) [10:00:18] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1073.eqiad.wmnet with OS trixie [10:00:33] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12306979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1073.eqiad.wmnet with OS trixie completed:... [10:00:35] 06SRE, 10ServiceOps-Upgrades-Hardware, 06ServiceOps (Next quarter): mc20[56-73] implementation tracking - https://phabricator.wikimedia.org/T436273#12306981 (10MLechvien-WMF) [10:06:49] (03PS2) 10Ladsgroup: Reduce the webp threshold to 60 [puppet] - 10https://gerrit.wikimedia.org/r/1338760 (https://phabricator.wikimedia.org/T431150) [10:06:54] (03PS1) 10Marostegui: clouddb1024: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1338768 (https://phabricator.wikimedia.org/T409557) [10:06:55] (03PS3) 10Ladsgroup: Reduce the webp threshold to 60 [puppet] - 10https://gerrit.wikimedia.org/r/1338760 (https://phabricator.wikimedia.org/T431150) [10:07:42] (03CR) 10Ladsgroup: [V:03+2 C:03+2] Reduce the webp threshold to 60 [puppet] - 10https://gerrit.wikimedia.org/r/1338760 (https://phabricator.wikimedia.org/T431150) (owner: 10Ladsgroup) [10:08:18] (03CR) 10Marostegui: [C:03+2] clouddb1024: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1338768 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [10:10:35] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb1024.eqiad.wmnet with reason: Cloning x4 [10:12:26] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12307040 (10elukey) @Andrew I took a look with Filippo today and these are our findings: * older cloudcephosd have two SSDs and a so... [10:20:13] (03CR) 10Slyngshede: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [10:23:36] (03CR) 10Phuedx: [C:03+1] Revert "Test Kitchen UI: Deploy v1.5.4 release to production" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338747 (owner: 10Santiago Faci) [10:32:23] (03CR) 10Hnowlan: [C:03+1] grafana: Add grafana-plugins to grafana-server suscription [puppet] - 10https://gerrit.wikimedia.org/r/1338350 (https://phabricator.wikimedia.org/T436056) (owner: 10Andrea Denisse) [10:34:02] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12307091 (10Ladsgroup) Please take a look at https://lists.wikimedia.org/hyperkitty/list/wikimedia-l@lists.wikimedia.org/message/IEYQ2HS3LZF2P3DAYMNZYQDGHWPVMTPY/ and as we said here multiple t... [10:37:23] (03PS6) 10Slyngshede: P:tofurkey enable Tofurkey for MAGRU [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) [10:37:41] (03PS3) 10Ladsgroup: [WIP] Run the Swift result storage read off the event loop [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337312 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup-claude) [10:37:45] (03CR) 10Slyngshede: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [10:38:27] (03PS4) 10Ladsgroup: [WIP] Run the Swift result storage read off the event loop [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337312 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup-claude) [10:39:36] (03PS5) 10Ladsgroup: Run the Swift result storage read off the event loop [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337312 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup-claude) [10:40:46] (03CR) 10Slyngshede: P:tofurkey enable Tofurkey for MAGRU (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [10:42:57] !log btullis@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-worker1199.eqiad.wmnet with reason: Upgrading RAID firmware [10:43:03] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12307109 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d454625a-4a29-4d4e-b8d8-7f2a38386afc) set by btullis@cumin1004 for 4:00:00 on 1 ho... [10:44:32] (03CR) 10Santiago Faci: [C:03+2] Revert "Test Kitchen UI: Deploy v1.5.4 release to production" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338747 (owner: 10Santiago Faci) [10:47:15] (03Merged) 10jenkins-bot: Revert "Test Kitchen UI: Deploy v1.5.4 release to production" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338747 (owner: 10Santiago Faci) [10:47:47] (03CR) 10Hnowlan: sre.opensearch.roll-restart-reboot: include checklist items (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (https://phabricator.wikimedia.org/T435265) (owner: 10Herron) [10:49:21] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9400/console" [puppet] - 10https://gerrit.wikimedia.org/r/1338216 (owner: 10Elukey) [10:50:32] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply [10:50:55] (03CR) 10Majavah: [V:03+1 C:04-1] "one style thing, otherwise LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1338216 (owner: 10Elukey) [10:50:58] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply [10:59:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:01:38] (03CR) 10Clément Goubert: "Enabling it everywhere but not using it, and then incremental toggling sounds perfect." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335285 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [11:03:58] !log btullis@cumin1004 START - Cookbook sre.hosts.remove-downtime for an-worker1199.eqiad.wmnet [11:03:59] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for an-worker1199.eqiad.wmnet [11:04:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:04:22] !log btullis@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-worker1200.eqiad.wmnet with reason: Upgrading RAID firmware [11:04:29] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12307139 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0523d7f5-2f58-4e9c-ab91-bf0b8b0fc7a9) set by btullis@cumin1004 for 4:00:00 on 1 ho... [11:16:04] !log btullis@cumin1004 START - Cookbook sre.hosts.remove-downtime for an-worker1200.eqiad.wmnet [11:16:05] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for an-worker1200.eqiad.wmnet [11:18:23] FIRING: [5x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [11:19:34] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12307169 (10Geertivp) I have sent a request earlier this month to `wikimediabe-l-owner@lists.wikimedia.org` to add me as an admin, but @Romaine seems to have lost admin rights as well... Isn't... [11:20:23] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T437170#12307176 (10BTullis) I think that this is is another one of those times when the disk hasn't actually failed, but it was disconnected by the RAID controlle... [11:20:34] RECOVERY - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1200 is OK: communication: 0 OK : controller: 0 OK : physical_disk: 0 OK : virtual_disk: 0 OK : bbu: 0 OK : enclosure: 0 OK https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [11:24:14] !log btullis@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-worker1204.eqiad.wmnet with reason: Upgrading RAID firmware [11:24:21] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12307181 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0289eb2c-f285-422f-ab6e-28151b64a5c5) set by btullis@cumin1004 for 4:00:00 on 1 ho... [11:25:55] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2004.codfw.wmnet [11:30:07] (03CR) 10Btullis: [C:03+2] ceph: Correct four faults in verify-cephx-keys [puppet] - 10https://gerrit.wikimedia.org/r/1338131 (https://phabricator.wikimedia.org/T437233) (owner: 10Btullis) [11:31:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2004.codfw.wmnet [11:31:45] (03PS1) 10Marostegui: check_private_data_report: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1338830 (https://phabricator.wikimedia.org/T409557) [11:32:38] (03PS1) 10JMeybohm: kubelet: Add missing evictionHard imagefs.inodesFree [puppet] - 10https://gerrit.wikimedia.org/r/1338831 (https://phabricator.wikimedia.org/T437539) [11:32:41] (03PS1) 10JMeybohm: kubelet: Start image garbage collection at 80% imagefs usage [puppet] - 10https://gerrit.wikimedia.org/r/1338832 (https://phabricator.wikimedia.org/T437539) [11:33:56] 06SRE, 06Data-Engineering, 10Dumps-Generation, 06serviceops-deprecated, and 2 others: conf* hosts ran out of disk space due to log spam - https://phabricator.wikimedia.org/T322360#12307345 (10MLechvien-WMF) [11:34:35] 06SRE, 14SRE-OnFire (FY2021/2022-Q4), 06DBA, 10GlobalBlocking, and 2 others: 2022-05-05 Wikimedia full site outage - https://phabricator.wikimedia.org/T307647#12307366 (10MLechvien-WMF) [11:36:31] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic, 07Wikimedia-Incident: Wikimedia's eqsin datacenter (Asia Pacific) had network connectivity issues - https://phabricator.wikimedia.org/T284986#12307394 (10MLechvien-WMF) [11:36:41] 06SRE, 06serviceops-deprecated, 10Wikidata, 05Release, and 3 others: June 2021: appservers accumulating active php-fpm workers, requiring rolling restarts to avoid user-visible latency impact - https://phabricator.wikimedia.org/T285634#12307392 (10MLechvien-WMF) [11:37:20] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 07Wikimedia-Incident: db1138 (s4 master) crashed due to memory issues - https://phabricator.wikimedia.org/T253808#12307414 (10MLechvien-WMF) [11:37:41] 06SRE, 06MediaWiki-Core-Platform-Team, 10MediaWiki-libs-BagOStuff, 10Page Content Service, and 2 others: cache_text cluster consistently backlogged on purge requests - https://phabricator.wikimedia.org/T249325#12307418 (10MLechvien-WMF) [11:37:50] 06SRE, 06MediaWiki-Core-Platform-Team, 10MediaWiki-ResourceLoader, 06Traffic, 07Wikimedia-Incident: load.php?modules=startup miss rate tripled on 2020-02-05 - https://phabricator.wikimedia.org/T247020#12307430 (10MLechvien-WMF) [11:38:09] 06SRE, 06Infrastructure-Foundations, 10netops, 07Wikimedia-Incident: Improve resiliency of the eqsin transport link - https://phabricator.wikimedia.org/T236878#12307437 (10MLechvien-WMF) [11:38:31] 06SRE, 06Infrastructure-Foundations, 10netops, 07Wikimedia-Incident: asw2-d2-eqiad crash - https://phabricator.wikimedia.org/T233645#12307447 (10MLechvien-WMF) [11:38:34] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 07Wikimedia-Incident: db1075 (s3 master) crashed - BBU failure - https://phabricator.wikimedia.org/T233534#12307449 (10MLechvien-WMF) [11:39:18] 06SRE, 06Infrastructure-Foundations, 10netops, 07Wikimedia-Incident: asw2-a-eqiad FPC5 gets disconnected every 10 minutes - https://phabricator.wikimedia.org/T201145#12307473 (10MLechvien-WMF) [11:39:27] 06SRE, 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10WMF-JobQueue, and 2 others: kafka eqiad cluster keeps crashing - https://phabricator.wikimedia.org/T199353#12307475 (10MLechvien-WMF) [11:39:30] 06SRE, 06Data-Engineering, 10Wikidata, 10Wikidata-Query-Service, and 3 others: Kafka eqiad.mediawiki.page-delete topic is empty - https://phabricator.wikimedia.org/T210451#12307471 (10MLechvien-WMF) [11:47:37] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:48:28] (03PS1) 10Slyngshede: site.pp move cp7010 from upload to text [puppet] - 10https://gerrit.wikimedia.org/r/1338843 (https://phabricator.wikimedia.org/T436363) [11:48:39] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1338830 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [11:49:19] (03CR) 10Clément Goubert: [C:03+2] editcheck-headless: Add ingress discovery records [dns] - 10https://gerrit.wikimedia.org/r/1338080 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [11:49:53] !log cgoubert@dns1004 START - running authdns-update [11:52:09] !log cgoubert@dns1004 END - running authdns-update [11:52:24] (03PS3) 10Btullis: cloudnative-pg: alert on WAL archiving failures rather than staleness [alerts] - 10https://gerrit.wikimedia.org/r/1338719 (https://phabricator.wikimedia.org/T432104) [11:53:01] (03PS1) 10Marostegui: orchestrator.conf.json.erb: Remove access from jynus [puppet] - 10https://gerrit.wikimedia.org/r/1338847 [11:54:40] !log btullis@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-worker1144.eqiad.wmnet with reason: Upgrading RAID firmware [11:54:46] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12307551 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=6fb6afd4-6ddb-4cff-9a70-30b3867baf56) set by btullis@cumin1004 for 4:00:00 on 1 ho... [11:55:52] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2005.codfw.wmnet [11:59:26] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for db1245.mgmt:22 - https://phabricator.wikimedia.org/T437565#12307577 (10Jclark-ctr) a:03Jclark-ctr [12:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1200) [12:01:10] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2005.codfw.wmnet [12:03:28] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1338847 (owner: 10Marostegui) [12:03:36] (03CR) 10Marostegui: [C:03+2] orchestrator.conf.json.erb: Remove access from jynus [puppet] - 10https://gerrit.wikimedia.org/r/1338847 (owner: 10Marostegui) [12:05:05] (03CR) 10Slyngshede: "Note that we're skipping cp7009 until Tofurkey CR is merged: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324689" [puppet] - 10https://gerrit.wikimedia.org/r/1338843 (https://phabricator.wikimedia.org/T436363) (owner: 10Slyngshede) [12:05:35] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb1025.eqiad.wmnet with reason: Cloning x4 [12:05:46] 06SRE, 06Traffic: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12307605 (10MLechvien-WMF) [12:05:50] (03PS1) 10Marostegui: clouddb1025: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1338868 (https://phabricator.wikimedia.org/T409557) [12:06:30] (03CR) 10Marostegui: [C:03+2] clouddb1025: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1338868 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [12:12:13] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for db1245.mgmt:22 - https://phabricator.wikimedia.org/T437565#12307632 (10Jclark-ctr) Merging with T431115 on going troubleshooting [12:12:20] (03PS1) 10Filippo Giunchedi: installserver: use cephosd recipe for cloudcephosd105[3-6] [puppet] - 10https://gerrit.wikimedia.org/r/1338879 (https://phabricator.wikimedia.org/T419892) [12:13:22] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for db1245.mgmt:22 - https://phabricator.wikimedia.org/T437565#12307641 (10Jclark-ctr) →14Duplicate dup:03T431115 [12:13:23] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12307638 (10Jclark-ctr) [12:13:36] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12307642 (10fgiunchedi) >>! In T419892#12307040, @elukey wrote: > @Andrew I took a look with Filippo today and these are our findings... [12:13:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:16:53] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12307653 (10fgiunchedi) 05Resolved→03Open a:05VRiley-WMF→03fgiunchedi I'll take this until 1075 is fully back in service (next week) [12:30:03] (03CR) 10Kevin Bazira: "great. thank you for targeting today. because a deployment tomorrow (Friday) may be tricky : )" [dns] - 10https://gerrit.wikimedia.org/r/1337904 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [12:32:22] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:32:37] 10ops-eqiad, 06DC-Ops: Unresponsive management for db1245.mgmt:22 - https://phabricator.wikimedia.org/T437585 (10phaultfinder) 03NEW [12:33:03] !log dbrant@deploy1003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [12:33:47] !log dbrant@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [12:34:15] !log dbrant@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [12:34:42] !log dbrant@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [12:35:25] !log dbrant@deploy1003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [12:35:34] 10ops-eqiad, 06DC-Ops: Unresponsive management for db1245.mgmt:22 - https://phabricator.wikimedia.org/T437585#12307753 (10Jclark-ctr) a:03Jclark-ctr [12:35:54] !log dbrant@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [12:38:23] FIRING: [4x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [12:41:26] (03CR) 10Klausman: [V:03+2 C:03+2] Add tool-server CNAMEs to k8s-ingress-ml-serve [dns] - 10https://gerrit.wikimedia.org/r/1337904 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [12:42:03] (03PS6) 10CDanis: New swift_per_container_stats_objects metrics [puppet] - 10https://gerrit.wikimedia.org/r/1329558 (https://phabricator.wikimedia.org/T431597) (owner: 10Cparle) [12:42:07] (03CR) 10Ladsgroup: [V:03+2 C:03+2] New swift_per_container_stats_objects metrics [puppet] - 10https://gerrit.wikimedia.org/r/1329558 (https://phabricator.wikimedia.org/T431597) (owner: 10Cparle) [12:42:26] (03CR) 10Brouberol: [C:03+1] "Beautiful, thanks!" [alerts] - 10https://gerrit.wikimedia.org/r/1338719 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [12:43:26] (03PS4) 10Elukey: Move docker::registry to toolforge::registry [puppet] - 10https://gerrit.wikimedia.org/r/1338216 [12:43:26] (03PS2) 10Elukey: Rename docker::registry hiera key to docker_registry_endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1338224 [12:43:27] (03PS7) 10Elukey: docker_registry: remove the Swift support [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) [12:44:54] (03PS1) 10Tryvix1509: afwiki: Create Draft and Draft talk namespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) [12:45:20] (03PS5) 10Elukey: Move docker::registry to toolforge::registry [puppet] - 10https://gerrit.wikimedia.org/r/1338216 [12:45:20] (03PS3) 10Elukey: Rename docker::registry hiera key to docker_registry_endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1338224 [12:45:20] (03PS8) 10Elukey: docker_registry: remove the Swift support [puppet] - 10https://gerrit.wikimedia.org/r/1338136 (https://phabricator.wikimedia.org/T427175) [12:45:37] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338216 (owner: 10Elukey) [12:45:46] (03CR) 10Elukey: [C:03+2] Assign a role to pki[12]003 [puppet] - 10https://gerrit.wikimedia.org/r/1338235 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [12:46:44] !log klausman@dns1004 START - running authdns-update [12:48:57] !log klausman@dns1004 END - running authdns-update [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: May I have your attention please! UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1300) [13:00:05] No Gerrit patches in the queue for this window AFAICS. [13:00:32] (03CR) 10Klausman: [V:03+2 C:03+2] tool-server: Add envoy services-proxy listener [puppet] - 10https://gerrit.wikimedia.org/r/1337895 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [13:02:08] (03CR) 10Klausman: [C:03+1] "LGTM from my POV, adding Traffic reviewers Sukhbir and Simon" [puppet] - 10https://gerrit.wikimedia.org/r/1337895 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [13:02:57] (03CR) 10Klausman: [V:03+2 C:03+2] "My bad, wrong change." [puppet] - 10https://gerrit.wikimedia.org/r/1337895 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [13:03:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:03:28] (03CR) 10Klausman: [C:03+1] "LGTM to me, adding Sukhbir and Simon from Traffic for review" [puppet] - 10https://gerrit.wikimedia.org/r/1337900 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [13:03:32] (03CR) 10Majavah: [C:03+1] "thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1338216 (owner: 10Elukey) [13:03:57] (03PS1) 10Slyngshede: data.yaml Offboarding jynus [puppet] - 10https://gerrit.wikimedia.org/r/1338911 [13:04:42] (03CR) 10Klausman: [V:03+2 C:03+2] service::catalog: Add tool-server [puppet] - 10https://gerrit.wikimedia.org/r/1337900 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [13:05:28] (03CR) 10Klausman: [C:03+1] service::catalog: Add tool-server [puppet] - 10https://gerrit.wikimedia.org/r/1337900 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [13:05:43] (03CR) 10Klausman: [V:03+2 C:03+1] tool-server: Add envoy services-proxy listener [puppet] - 10https://gerrit.wikimedia.org/r/1337895 (https://phabricator.wikimedia.org/T436892) (owner: 10Kevin Bazira) [13:06:31] (03CR) 10Slyngshede: [C:03+2] data.yaml Offboarding jynus [puppet] - 10https://gerrit.wikimedia.org/r/1338911 (owner: 10Slyngshede) [13:11:12] (03PS1) 10Slyngshede: C:nagios_common::contactgroups offboarding jynus [puppet] - 10https://gerrit.wikimedia.org/r/1338916 [13:11:44] (03CR) 10Elukey: [C:03+2] Move docker::registry to toolforge::registry [puppet] - 10https://gerrit.wikimedia.org/r/1338216 (owner: 10Elukey) [13:13:07] (03CR) 10Slyngshede: [C:03+2] C:nagios_common::contactgroups offboarding jynus [puppet] - 10https://gerrit.wikimedia.org/r/1338916 (owner: 10Slyngshede) [13:14:44] (03PS1) 10Tiziano Fogli: logstash/mediawiki: add 10 days to retention window [puppet] - 10https://gerrit.wikimedia.org/r/1338921 [13:15:27] (03PS3) 10Elukey: Add conftool-data for the pki[12]003 VMs [puppet] - 10https://gerrit.wikimedia.org/r/1338236 (https://phabricator.wikimedia.org/T436809) [13:17:37] (03PS2) 10Tiziano Fogli: logstash/mediawiki: add 10 days to retention window [puppet] - 10https://gerrit.wikimedia.org/r/1338921 [13:23:19] 10ops-eqiad, 06DC-Ops: Unresponsive management for db1245.mgmt:22 - https://phabricator.wikimedia.org/T437585#12307978 (10Jclark-ctr) 05Open→03Resolved [13:24:42] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1338044 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [13:25:19] (03PS4) 10Jelto: fix helm command in all NOTES.txt [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333699 (https://phabricator.wikimedia.org/T433589) [13:25:51] (03CR) 10Jelto: fix helm command in all NOTES.txt (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333699 (https://phabricator.wikimedia.org/T433589) (owner: 10Jelto) [13:29:29] (03CR) 10Krinkle: "I'm neutral on these specific wikis, but I disagree with the commit message as stated, which presumed and implies precedent to remove thes" [dns] - 10https://gerrit.wikimedia.org/r/1338313 (https://phabricator.wikimedia.org/T214998) (owner: 10Dzahn) [13:29:47] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in profile lists [puppet] - 10https://gerrit.wikimedia.org/r/1338044 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [13:31:05] (03PS1) 10Slyngshede: Icinga: offboarding Jcrespo [puppet] - 10https://gerrit.wikimedia.org/r/1338935 [13:31:45] (03PS4) 10Elukey: Add conftool-data for the pki[12]003 VMs [puppet] - 10https://gerrit.wikimedia.org/r/1338236 (https://phabricator.wikimedia.org/T436809) [13:31:45] (03PS1) 10Elukey: role::pki: add lvs configurations [puppet] - 10https://gerrit.wikimedia.org/r/1338936 (https://phabricator.wikimedia.org/T436809) [13:31:47] (03PS1) 10Elukey: Move the pki service entry to lvs_setup [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809) [13:32:00] (03CR) 10CI reject: [V:04-1] Icinga: offboarding Jcrespo [puppet] - 10https://gerrit.wikimedia.org/r/1338935 (owner: 10Slyngshede) [13:33:07] (03CR) 10Elukey: Move the pki service entry to lvs_setup (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [13:35:05] (03CR) 10Slyngshede: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338935 (owner: 10Slyngshede) [13:35:44] (03PS1) 10Andrew Bogott: Rename cloudvps-project-usage.py to openstack-project-usage.py [puppet] - 10https://gerrit.wikimedia.org/r/1338938 (https://phabricator.wikimedia.org/T436275) [13:36:01] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip for mobileapps, proton, push-notifications codfw [puppet] - 10https://gerrit.wikimedia.org/r/1337923 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [13:36:36] (03CR) 10Slyngshede: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1338935 (owner: 10Slyngshede) [13:37:05] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw [13:38:23] FIRING: [2x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [13:38:36] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338938 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [13:40:41] (03CR) 10Ssingh: [C:03+1] site.pp move cp7010 from upload to text [puppet] - 10https://gerrit.wikimedia.org/r/1338843 (https://phabricator.wikimedia.org/T436363) (owner: 10Slyngshede) [13:41:03] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [13:41:59] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [13:41:59] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw [13:42:54] (03PS2) 10Jelto: service::catalog: Set ipip for mobileapps, proton, push-notifications eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1337924 (https://phabricator.wikimedia.org/T420436) [13:46:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:47:51] (03CR) 10Ssingh: [C:03+1] P:tofurkey enable Tofurkey for MAGRU [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [13:51:15] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip for mobileapps, proton, push-notifications eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1337924 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [13:51:36] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad [13:51:38] (03CR) 10Andrew Bogott: [C:03+2] Rename cloudvps-project-usage.py to openstack-project-usage.py [puppet] - 10https://gerrit.wikimedia.org/r/1338938 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [13:55:17] !log jelto@cumin1003 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [13:56:32] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [13:56:32] !log jelto@cumin1003 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad [13:57:24] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bookworm [13:57:24] (03CR) 10Krinkle: "Afaik that's not the reason this task exists. The UI does not parse this list in any way, and the order does not imply that one is current" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319818 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [14:01:24] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:02:36] (03CR) 10Ssingh: [C:03+1] "Seems fine from the CDN as long as you have verified the backends. Let us know when you want us to roll it out." [puppet] - 10https://gerrit.wikimedia.org/r/1338134 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [14:05:52] (03PS1) 10Majavah: hieradata: Route x4 traffic to the s4 replicas for now [puppet] - 10https://gerrit.wikimedia.org/r/1338942 [14:06:55] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9401/co" [puppet] - 10https://gerrit.wikimedia.org/r/1338942 (owner: 10Majavah) [14:07:34] !log bking@cumin2003 START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. [14:07:46] (03CR) 10Majavah: [V:03+1 C:03+2] hieradata: Route x4 traffic to the s4 replicas for now [puppet] - 10https://gerrit.wikimedia.org/r/1338942 (owner: 10Majavah) [14:12:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.15% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:13:41] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on clouddumps1001.wikimedia.org with reason: host reimage [14:17:56] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddumps1001.wikimedia.org with reason: host reimage [14:18:25] (03PS1) 10CDanis: Code changes [software/hiddenparma/deploy] - 10https://gerrit.wikimedia.org/r/1338944 [14:18:47] (03PS2) 10CDanis: Etcd fanout fix & details UX [software/hiddenparma/deploy] - 10https://gerrit.wikimedia.org/r/1338944 [14:19:24] (03CR) 10CDanis: [V:03+2 C:03+2] Etcd fanout fix & details UX [software/hiddenparma/deploy] - 10https://gerrit.wikimedia.org/r/1338944 (owner: 10CDanis) [14:20:10] !log cdanis@cumin1004 START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "etcd fanout fix & details UX - cdanis@cumin1004" [14:20:11] !log cdanis@cumin1004 START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: etcd fanout fix & details UX - cdanis@cumin1004 [14:21:02] !log cdanis@cumin1004 END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: etcd fanout fix & details UX - cdanis@cumin1004 [14:21:03] !log cdanis@cumin1004 END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "etcd fanout fix & details UX - cdanis@cumin1004" [14:22:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.55% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:24:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.55% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:25:12] (03Abandoned) 10Tiziano Fogli: logstash/mediawiki: add 10 days to retention window [puppet] - 10https://gerrit.wikimedia.org/r/1338921 (owner: 10Tiziano Fogli) [14:25:24] FIRING: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:26:38] (03PS2) 10Hnowlan: logstash: increase mediawiki retention [puppet] - 10https://gerrit.wikimedia.org/r/1338923 (https://phabricator.wikimedia.org/T437597) [14:29:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.55% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1430) [14:36:50] (03PS1) 10Majavah: hieradata: cloudlb: Remove temporary x4 section_port override [puppet] - 10https://gerrit.wikimedia.org/r/1338949 (https://phabricator.wikimedia.org/T437208) [14:38:24] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9403/co" [puppet] - 10https://gerrit.wikimedia.org/r/1338949 (https://phabricator.wikimedia.org/T437208) (owner: 10Majavah) [14:38:55] (03PS1) 10TheDJ: WebP: apply coalesce before resizing [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1338951 (https://phabricator.wikimedia.org/T290345) [14:40:16] !log bking@cumin2003 END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. [14:40:26] (03CR) 10Ssingh: "Yeah testing it on clouddumps is fine. How did we install python3-iptables on clouddumps1002 though? Was it done manually? It seems this i" [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins) [14:43:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:46:20] (03CR) 10Herron: [C:03+1] logstash: increase mediawiki retention [puppet] - 10https://gerrit.wikimedia.org/r/1338923 (https://phabricator.wikimedia.org/T437597) (owner: 10Hnowlan) [14:50:21] (03CR) 10Hnowlan: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338923 (https://phabricator.wikimedia.org/T437597) (owner: 10Hnowlan) [14:51:26] jouncebot: nowandnext [14:51:26] For the next 0 hour(s) and 8 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1430) [14:51:26] In 1 hour(s) and 8 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1600) [14:53:00] (03CR) 10JMeybohm: [C:03+1] fix helm command in all NOTES.txt [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333699 (https://phabricator.wikimedia.org/T433589) (owner: 10Jelto) [14:53:02] (03PS4) 10Samtar: IS: enable wgEnableWatchstarPopover on test.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335032 (https://phabricator.wikimedia.org/T436955) [14:53:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:54:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335032 (https://phabricator.wikimedia.org/T436955) (owner: 10Samtar) [14:55:51] (03Merged) 10jenkins-bot: IS: enable wgEnableWatchstarPopover on test.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335032 (https://phabricator.wikimedia.org/T436955) (owner: 10Samtar) [14:56:17] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1335032|IS: enable wgEnableWatchstarPopover on test.wikipedia (T436955)]] [14:56:20] T436955: Enable WatchstarPopover on test.wikipedia - https://phabricator.wikimedia.org/T436955 [14:58:27] RESOLVED: [2x] GnmiInterfaceCountersDrop: fasw1-f5a-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:58:39] (03PS3) 10Hnowlan: logstash: increase mediawiki retention [puppet] - 10https://gerrit.wikimedia.org/r/1338923 [14:58:44] !log samtar@deploy1003 samtar: Backport for [[gerrit:1335032|IS: enable wgEnableWatchstarPopover on test.wikipedia (T436955)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:58:56] (03CR) 10Hnowlan: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338923 (owner: 10Hnowlan) [14:59:46] !log samtar@deploy1003 samtar: Continuing with deployment [14:59:48] (03Abandoned) 10Elukey: registry: expand the HTTP Accept headers and drop v1 compatibility [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1078345 (owner: 10Elukey) [15:00:04] (03Abandoned) 10Elukey: python-webapp: add external-services support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1169162 (https://phabricator.wikimedia.org/T398640) (owner: 10Elukey) [15:00:50] (03Abandoned) 10Elukey: WIP: test reimage workaround for sretest2010 [cookbooks] - 10https://gerrit.wikimedia.org/r/1287829 (owner: 10Elukey) [15:02:27] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddumps1001.wikimedia.org with OS bookworm [15:03:41] (03CR) 10Mr. Starfleet Command: "It seems likely that there are users out there who use the 2017 editor, have already enabled syntax highlighting, and use a browser that d" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [15:04:25] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1335032|IS: enable wgEnableWatchstarPopover on test.wikipedia (T436955)]] (duration: 08m 08s) [15:04:29] T436955: Enable WatchstarPopover on test.wikipedia - https://phabricator.wikimedia.org/T436955 [15:05:35] (03PS4) 10Hnowlan: logstash: increase mediawiki retention [puppet] - 10https://gerrit.wikimedia.org/r/1338923 [15:06:06] (03CR) 10Hnowlan: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338923 (owner: 10Hnowlan) [15:11:02] (03CR) 10Hnowlan: [C:03+2] logstash: increase mediawiki retention [puppet] - 10https://gerrit.wikimedia.org/r/1338923 (owner: 10Hnowlan) [15:14:32] 06SRE, 10SRE-Access-Requests: Requesting no server access - https://phabricator.wikimedia.org/T437611 (10AKanji-WMF) 03NEW [15:16:09] 06SRE, 10SRE-Access-Requests: Requesting no server access - https://phabricator.wikimedia.org/T437611#12308617 (10AKanji-WMF) [15:17:27] 06SRE, 10SRE-Access-Requests: Requesting no server access - https://phabricator.wikimedia.org/T437611#12308622 (10AKanji-WMF) [15:20:07] 06SRE-OnFire, 06ServiceOps, 10ServiceOps-Services-Oids, 06Release-Engineering-Team (Radar), 07Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162#12308680 (10Dzahn) After thinking about this some more: That last comment (only) makes sense if you lo... [15:21:37] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12308698 (10Andrew) > cloudcephosd[1048-52] have a single nvme disk (no hw raid of any kind) Those hosts have a BOSS card which is a... [15:24:06] PROBLEM - Host an-worker1204 is DOWN: PING CRITICAL - Packet loss = 100% [15:25:25] RESOLVED: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:33:15] !log marostegui@cumin1003 conftool action : set/pooled=no; selector: name=clouddb1025.eqiad.wmnet,service=s4 [15:44:06] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338975 (https://phabricator.wikimedia.org/T128546) [15:48:38] (03Abandoned) 10Ladsgroup: [WIP] Give thumbor's engine thread pool a worker [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337316 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup-claude) [15:54:56] PROBLEM - SSH on an-worker1144 is CRITICAL: connect to address 10.64.142.3 and port 22: Connection refused https://wikitech.wikimedia.org/wiki/SSH/monitoring [15:58:40] (03PS1) 10Btullis: mediawiki-dumps-legacy: Update the SSH host key for clouddumps1001 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338981 (https://phabricator.wikimedia.org/T403154) [15:59:51] (03PS1) 10Urbanecm: refactor: Provider: Add IConfigurationProvider::invalidateCache() [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338983 (https://phabricator.wikimedia.org/T437588) [16:00:05] jhathaway and rzl: It is that lovely time of the day again! You are hereby commanded to deploy Puppet request window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1600). [16:00:05] (03PS1) 10Urbanecm: refactor: Provider: Add IConfigurationProvider::invalidateCache() [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338984 (https://phabricator.wikimedia.org/T437588) [16:00:05] No Gerrit patches in the queue for this window AFAICS. [16:02:47] (03PS1) 10Urbanecm: Provider: Cache the valid configuration in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338986 (https://phabricator.wikimedia.org/T437588) [16:07:48] (03PS5) 10Milazg: Remove mode from RestModuleOverrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328215 (https://phabricator.wikimedia.org/T434267) [16:08:05] (03PS6) 10Milazg: Remove mode from RestModuleOverrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328215 (https://phabricator.wikimedia.org/T434267) [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:11:42] (03PS1) 10Urbanecm: Provider: Cache the valid configuration in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338990 (https://phabricator.wikimedia.org/T437588) [16:12:22] jouncebot: nowandnext [16:12:22] For the next 0 hour(s) and 47 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1600) [16:12:22] In 0 hour(s) and 47 minute(s): Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1700) [16:12:22] In 0 hour(s) and 47 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1700) [16:13:03] is the puppet window used for something? i have an UBN (T437588) to fix [16:13:03] T437588: Wikimedia\RequestTimeout\RequestTimeoutException: The maximum execution time of {limit} seconds was exceeded - https://phabricator.wikimedia.org/T437588 [16:14:16] (03PS7) 10Milazg: Remove mode from RestModuleOverrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328215 (https://phabricator.wikimedia.org/T434267) [16:14:56] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12309022 (10RobH) IRC Sync update: * All switches except the lsw and ssw in A8 are racked, connected to SCS * I've added columns for the switch fiber cable... [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:58] (03CR) 10EggRoll97: [C:03+1] afwiki: Create Draft and Draft talk namespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) (owner: 10Tryvix1509) [16:18:36] rzl: jhathaway: before i make use of the window, double checking (i have an UBN to fix, and nothing seems to be happening) [16:19:06] please go ahead urbanecm [16:19:09] urbanecm: all yours [16:19:16] ty [16:19:27] (03CR) 10Urbanecm: [C:03+2] refactor: Provider: Add IConfigurationProvider::invalidateCache() [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338984 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [16:19:33] (03CR) 10Urbanecm: [C:03+2] Provider: Cache the valid configuration in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338990 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [16:19:43] (03CR) 10Urbanecm: [C:03+2] refactor: Provider: Add IConfigurationProvider::invalidateCache() [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338983 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [16:19:43] (03CR) 10Urbanecm: [C:03+2] Provider: Cache the valid configuration in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338986 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [16:27:02] (03Merged) 10jenkins-bot: refactor: Provider: Add IConfigurationProvider::invalidateCache() [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338984 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [16:27:04] (03Merged) 10jenkins-bot: Provider: Cache the valid configuration in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1338990 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [16:27:07] (03Merged) 10jenkins-bot: refactor: Provider: Add IConfigurationProvider::invalidateCache() [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338983 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [16:27:09] (03Merged) 10jenkins-bot: Provider: Cache the valid configuration in the process [extensions/CommunityConfiguration] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338986 (https://phabricator.wikimedia.org/T437588) (owner: 10Urbanecm) [16:28:44] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1338984|refactor: Provider: Add IConfigurationProvider::invalidateCache() (T437588)]], [[gerrit:1338990|Provider: Cache the valid configuration in the process (T437588)]], [[gerrit:1338983|refactor: Provider: Add IConfigurationProvider::invalidateCache() (T437588)]], [[gerrit:1338986|Provider: Cache the valid configuration in the process (T437588)]] [16:28:48] T437588: Wikimedia\RequestTimeout\RequestTimeoutException: The maximum execution time of {limit} seconds was exceeded - https://phabricator.wikimedia.org/T437588 [16:33:08] !log urbanecm@deploy1003 urbanecm: Backport for [[gerrit:1338984|refactor: Provider: Add IConfigurationProvider::invalidateCache() (T437588)]], [[gerrit:1338990|Provider: Cache the valid configuration in the process (T437588)]], [[gerrit:1338983|refactor: Provider: Add IConfigurationProvider::invalidateCache() (T437588)]], [[gerrit:1338986|Provider: Cache the valid configuration in the process (T437588)]] synced to the te [16:33:08] stservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:33:53] (03PS3) 10Jdlrobson: Enable ReadingLists on mediawiki and wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337984 (https://phabricator.wikimedia.org/T437113) [16:34:51] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12309116 (10cmooney) >>! In T418012#12309022, @RobH wrote: > * All switches except the lsw and ssw in A8 are racked, connected to SCS Can we confirm what po... [16:35:42] (03PS4) 10Jdlrobson: Enable ReadingLists on mediawiki and wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337984 (https://phabricator.wikimedia.org/T437113) [16:35:43] !log urbanecm@deploy1003 urbanecm: Continuing with deployment [16:39:28] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12309140 (10RobH) >>! In T418012#12309116, @cmooney wrote: >>>! In T418012#12309022, @RobH wrote: >> * All switches except the lsw and ssw in A8 are racked,... [16:40:12] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338984|refactor: Provider: Add IConfigurationProvider::invalidateCache() (T437588)]], [[gerrit:1338990|Provider: Cache the valid configuration in the process (T437588)]], [[gerrit:1338983|refactor: Provider: Add IConfigurationProvider::invalidateCache() (T437588)]], [[gerrit:1338986|Provider: Cache the valid configuration in the process (T437588)]] [16:40:12] (duration: 11m 28s) [16:40:16] T437588: Wikimedia\RequestTimeout\RequestTimeoutException: The maximum execution time of {limit} seconds was exceeded - https://phabricator.wikimedia.org/T437588 [16:41:53] (03CR) 10Bking: [C:03+1] mediawiki-dumps-legacy: Update the SSH host key for clouddumps1001 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338981 (https://phabricator.wikimedia.org/T403154) (owner: 10Btullis) [16:42:17] (03CR) 10Dzahn: [C:03+2] spdx: ignore `.rspec` file [puppet] - 10https://gerrit.wikimedia.org/r/1329227 (owner: 10Hashar) [16:44:17] 06SRE-OnFire, 06ServiceOps, 10ServiceOps-Services-Oids, 06Release-Engineering-Team (Radar), 07Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162#12309163 (10Raine) 05Resolved→03Open >>! In T309162#12308680, @Dzahn wrote: > After thinking about thi... [16:45:02] * urbanecm done [16:45:09] Thanks! [16:46:20] Dreamy_Jazz: I'd appreciate second eyes on the "profiling after" though ;). See task. [16:46:42] 06SRE-OnFire, 06ServiceOps, 10ServiceOps-Services-Oids, 06Release-Engineering-Team (Radar), 07Sustainability: deployment servers should have identical repo data (was: Remove old scap repositories from deploy1002) - https://phabricator.wikimedia.org/T309162#12309170 (10Dzahn) [16:47:03] * Dreamy_Jazz looks [16:47:51] 06SRE, 10SRE-Access-Requests: Requesting access to production for derenrich - https://phabricator.wikimedia.org/T436965#12309185 (10ssingh) 05Open→03Resolved a:03ssingh @derenrich: Boldly resolving as part of clinic duty; please re-open if you see any issues. [16:47:53] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Attempt to use a VM as Kerberos replica - https://phabricator.wikimedia.org/T435873#12309188 (10bking) Theory #1 from above has been confirmed. I've roll-restarted the `an-presto` hosts and [[ https://grafana.wikimedia.org/goto/s4... [16:49:14] (03CR) 10Dzahn: [C:03+2] zookeeper: add spec tests [puppet] - 10https://gerrit.wikimedia.org/r/1329228 (https://phabricator.wikimedia.org/T435503) (owner: 10Hashar) [16:53:22] (03CR) 10Herron: "thanks for the pointers!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (https://phabricator.wikimedia.org/T435265) (owner: 10Herron) [17:00:05] bd808: That opportune time for a Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1700). [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1700) [17:00:36] (03CR) 10DLynch: [C:03+1] "First, MusikAnimal is right, as-is they'll not get highlighting. I think they'll get some non-highlighting features like line numbers etc," [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [17:04:58] (03PS1) 10Ladsgroup: thumbor: Actually enable asyncio [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339015 (https://phabricator.wikimedia.org/T431767) [17:33:51] (03PS5) 10Atsuko: airflow: pluggable authmanager for K8s tokens [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338217 (https://phabricator.wikimedia.org/T437236) [17:38:15] (03PS5) 10MusikAnimal: [CodeMirror] enable for new and logged-out users by default on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) [17:39:02] (03CR) 10Bking: [C:03+1] airflow: pluggable authmanager for K8s tokens [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338217 (https://phabricator.wikimedia.org/T437236) (owner: 10Atsuko) [17:39:29] (03PS6) 10MusikAnimal: [CodeMirror] enable for new and logged-out users by default on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) [17:39:54] (03CR) 10MusikAnimal: "Alright, well, with David's word I say we move forward and see how she flies. I will however keep the config changes separate so it's easi" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [17:40:55] (03CR) 10MusikAnimal: [C:03+1] "Done" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1322175 (https://phabricator.wikimedia.org/T432558) (owner: 10DLynch) [17:42:12] 10ops-eqiad, 06SRE, 06DC-Ops, 06tools-infrastructure-team: Physical move - https://phabricator.wikimedia.org/T437631 (10VRiley-WMF) 03NEW [17:42:38] 10ops-eqiad, 06SRE, 06DC-Ops, 06tools-infrastructure-team: Physical move - https://phabricator.wikimedia.org/T437631#12309372 (10VRiley-WMF) 05Open→03Resolved This has been completed. [17:51:33] (03CR) 10Thilio: [C:03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) (owner: 10Tryvix1509) [17:52:52] (03CR) 10Btullis: [C:03+2] mediawiki-dumps-legacy: Update the SSH host key for clouddumps1001 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338981 (https://phabricator.wikimedia.org/T403154) (owner: 10Btullis) [17:54:58] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339034 [17:55:13] (03Merged) 10jenkins-bot: mediawiki-dumps-legacy: Update the SSH host key for clouddumps1001 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338981 (https://phabricator.wikimedia.org/T403154) (owner: 10Btullis) [17:57:42] (03CR) 10Dbrant: [C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339034 (owner: 10PipelineBot) [18:00:00] (03Merged) 10jenkins-bot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339034 (owner: 10PipelineBot) [18:00:05] dduvall and dancy: gettimeofday() says it's time for MediaWiki train - Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1800) [18:00:14] o/ [18:00:53] (03PS1) 10TrainBranchBot: group2 to 1.47.0-wmf.19 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339035 (https://phabricator.wikimedia.org/T430838) [18:00:56] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dancy@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339035 (https://phabricator.wikimedia.org/T430838) (owner: 10TrainBranchBot) [18:01:39] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:01:59] (03Merged) 10jenkins-bot: group2 to 1.47.0-wmf.19 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339035 (https://phabricator.wikimedia.org/T430838) (owner: 10TrainBranchBot) [18:06:42] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [18:08:39] (03PS1) 10Santiago Faci: Test Kitchen UI: Deploy v1.5.5. release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339043 (https://phabricator.wikimedia.org/T429524) [18:08:51] !log dbrant@deploy1003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [18:09:13] !log dbrant@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [18:09:54] (03PS2) 10Santiago Faci: Test Kitchen UI: Deploy v1.5.5. release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339043 (https://phabricator.wikimedia.org/T429524) [18:10:40] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 10 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328215 (https://phabricator.wikimedia.org/T434267) (owner: 10Milazg) [18:10:53] !log dzahn@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gerrit1003.wikimedia.org with reason: maintenance upgrade [18:11:05] !log dbrant@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [18:11:11] !log dancy@deploy1003 rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.19 refs T430838 [18:11:14] T430838: 1.47.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T430838 [18:11:20] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 10 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328215 (https://phabricator.wikimedia.org/T434267) (owner: 10Milazg) [18:11:27] !log dzahn@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gerrit2002.wikimedia.org with reason: maintenance upgrade [18:11:31] !log dbrant@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [18:11:46] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for VolkerE - https://phabricator.wikimedia.org/T437634 (10Volker_E) 03NEW [18:12:02] !log dbrant@deploy1003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [18:13:00] !log dbrant@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [18:13:08] !log otto@deploy1003 Started deploy [analytics/refinery@92b5c04] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@92b5c04e] [18:13:48] !log otto@deploy1003 Finished deploy [analytics/refinery@92b5c04] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@92b5c04e] (duration: 00m 39s) [18:14:29] !log otto@deploy1003 Started deploy [analytics/refinery@92b5c04]: Regular analytics weekly train [analytics/refinery@92b5c04e] [18:14:36] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [18:19:42] !log otto@deploy1003 Finished deploy [analytics/refinery@92b5c04]: Regular analytics weekly train [analytics/refinery@92b5c04e] (duration: 05m 13s) [18:20:37] !log otto@deploy1003 Started deploy [analytics/refinery@92b5c04] (thin): Regular analytics weekly train THIN [analytics/refinery@92b5c04e] [18:21:36] !log otto@deploy1003 Finished deploy [analytics/refinery@92b5c04] (thin): Regular analytics weekly train THIN [analytics/refinery@92b5c04e] (duration: 00m 59s) [18:26:44] (03PS3) 10Eric Gardner: Enable discord preview extension code on testwiki (tk 2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338439 (https://phabricator.wikimedia.org/T437344) (owner: 10DErenrich) [18:28:06] (03PS4) 10Eric Gardner: Enable discord preview extension code on testwiki (tk 2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338439 (https://phabricator.wikimedia.org/T437344) (owner: 10DErenrich) [18:29:36] 06SRE, 10LDAP-Access-Requests: Request for LDAP NDA Access for CentralNotice Metrics for Yahya - https://phabricator.wikimedia.org/T436294#12309515 (10greg) >>! In T436294#12284882, @Volans wrote: > Gentle reminder for @FRomeo_WMF and @greg to express their vote on this request. Approved from me (adding Yahya... [18:35:26] (03CR) 10RLazarus: [C:03+1] kubelet: Add missing evictionHard imagefs.inodesFree [puppet] - 10https://gerrit.wikimedia.org/r/1338831 (https://phabricator.wikimedia.org/T437539) (owner: 10JMeybohm) [18:40:20] (03PS1) 10JHathaway: icinga: remove elasticsearch settings checks [puppet] - 10https://gerrit.wikimedia.org/r/1339061 [18:40:34] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1339061 (owner: 10JHathaway) [18:41:59] (03PS2) 10JHathaway: icinga: remove elasticsearch settings checks [puppet] - 10https://gerrit.wikimedia.org/r/1339061 [18:42:03] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1339061 (owner: 10JHathaway) [18:47:05] (03CR) 10RLazarus: [C:03+1] "Thanks, let's give this a try!" [puppet] - 10https://gerrit.wikimedia.org/r/1338832 (https://phabricator.wikimedia.org/T437539) (owner: 10JMeybohm) [18:48:26] (03PS1) 10Ladsgroup-claude: [WIP] Set JPGs to be thumbnailed via VIPS if md5 starts with a [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1339064 (https://phabricator.wikimedia.org/T220171) [18:49:45] (03CR) 10Bking: [C:03+1] icinga: remove elasticsearch settings checks [puppet] - 10https://gerrit.wikimedia.org/r/1339061 (owner: 10JHathaway) [18:50:01] (03CR) 10Ladsgroup: "Overall, looks good but I need to test it locally before putting in front of the world." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1339064 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [18:50:05] (03CR) 10JHathaway: [C:03+2] icinga: remove elasticsearch settings checks [puppet] - 10https://gerrit.wikimedia.org/r/1339061 (owner: 10JHathaway) [18:51:05] (03PS5) 10Eric Gardner: Enable discord preview extension code on testwiki (tk 2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338439 (https://phabricator.wikimedia.org/T437344) (owner: 10DErenrich) [18:51:39] (03CR) 10Eric Gardner: [C:03+1] "LGTM for deployment today" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338439 (https://phabricator.wikimedia.org/T437344) (owner: 10DErenrich) [18:52:58] (03PS2) 10Slyngshede: Icinga: offboarding Jcrespo [puppet] - 10https://gerrit.wikimedia.org/r/1338935 [18:56:04] 06SRE, 10SRE-Access-Requests: Requesting access to growthbook-customelevatedaccess - https://phabricator.wikimedia.org/T437611#12309592 (10ssingh) [18:56:30] 06SRE, 10SRE-Access-Requests: Requesting access to growthbook-customelevatedaccess - https://phabricator.wikimedia.org/T437611#12309597 (10ssingh) @KReid-WMF: This requires your approval, thanks! [18:58:40] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for VolkerE - https://phabricator.wikimedia.org/T437634#12309620 (10ssingh) [19:01:05] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for VolkerE - https://phabricator.wikimedia.org/T437634#12309629 (10ssingh) @Jdrewniak: This requires your approval, thanks. Note: @Volker_E already has shell access, so we will just them to analytics-privatedata. Growthbook permi... [19:04:11] 06SRE, 10LDAP-Access-Requests: Request for LDAP NDA Access for CentralNotice Metrics for Yahya - https://phabricator.wikimedia.org/T436294#12309648 (10ssingh) [We are waiting on the NDA since already have Greg's approval.] [19:04:44] !log bking@cumin2003 START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [19:11:49] (03PS1) 10Jdlrobson: Instrument donor ID dialog: hooks defined [extensions/WikimediaEvents] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1339076 (https://phabricator.wikimedia.org/T435565) [19:12:19] (03CR) 10JHathaway: [C:03+2] Icinga: offboarding Jcrespo [puppet] - 10https://gerrit.wikimedia.org/r/1338935 (owner: 10Slyngshede) [19:16:22] !log dzahn@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on gerrit2003.wikimedia.org with reason: maintenance upgrade [19:17:04] !log dzahn@cumin1003 DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:30:00 on gerrit.wikimedia.org with reason: maintenance upgrade [19:17:06] !log bking@cumin2003 END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [19:17:31] jouncebot: nowandnext [19:17:31] For the next 0 hour(s) and 42 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T1800) [19:17:31] In 0 hour(s) and 42 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T2000) [19:17:48] !log bking@cumin2003 START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade. [19:17:57] !log Gerrit downtime incoming for upgrade [19:17:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:21:14] !log bking@cumin2003 END (FAIL) - Cookbook sre.hadoop.roll-restart-workers (exit_code=99) restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade. [19:32:18] !log rzl@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1346.eqiad.wmnet [19:32:22] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1346.eqiad.wmnet [19:32:58] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1346.eqiad.wmnet [19:33:13] !log rzl@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1346.eqiad.wmnet with OS trixie [19:33:32] !log rzl@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1346 [19:33:52] !log rzl@cumin2003 START - Cookbook sre.dns.netbox [19:37:59] !log rzl@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1346 - rzl@cumin2003" [19:38:03] !log rzl@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1346 - rzl@cumin2003" [19:38:04] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [19:38:04] !log rzl@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1346.eqiad.wmnet 196.48.64.10.in-addr.arpa 6.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [19:38:07] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1346.eqiad.wmnet 196.48.64.10.in-addr.arpa 6.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [19:38:08] !log rzl@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1346 [19:38:35] !log rzl@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1346 [19:38:35] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1346 [19:45:52] 06SRE, 10SRE-Access-Requests: Requesting access to growthbook-customelevatedaccess - https://phabricator.wikimedia.org/T437611#12309824 (10KReid-WMF) I approve the growthbook-customelevatedaccess access. Note that this group (growthbook-customelevatedaccess) can be requested via: https://idm.wikimedia.org/pe... [19:50:40] !log rzl@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1346.eqiad.wmnet with reason: host reimage [19:53:43] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1346.eqiad.wmnet with reason: host reimage [19:55:59] (03CR) 10Dzahn: "I guess in this case we should just rely on institutional knowledge to not add new ones and ignore old ones." [dns] - 10https://gerrit.wikimedia.org/r/1338313 (https://phabricator.wikimedia.org/T214998) (owner: 10Dzahn) [19:57:04] (03Abandoned) 10Dzahn: delete deprecated .m. records for arbcom and sysop special wikis [dns] - 10https://gerrit.wikimedia.org/r/1338313 (https://phabricator.wikimedia.org/T214998) (owner: 10Dzahn) [19:58:32] (03PS15) 10Hashar: zookeeper: fix log4j addition when tls is enabled [puppet] - 10https://gerrit.wikimedia.org/r/1327569 (https://phabricator.wikimedia.org/T435503) [19:58:46] (03CR) 10JHathaway: [C:03+1] sre.hosts.provision: avoid user locked out for Supermicro BMCs [cookbooks] - 10https://gerrit.wikimedia.org/r/1334894 (https://phabricator.wikimedia.org/T419892) (owner: 10Elukey) [19:59:52] (03PS1) 10Bking: an-worker1144: move to insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1339115 [20:00:04] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: May I have your attention please! UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T2000) [20:00:04] jan_drewniak and obvious-dust: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:06] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1339115 (owner: 10Bking) [20:00:51] I'll start the backport window with my patch [20:01:39] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 10 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1330446 (https://phabricator.wikimedia.org/T419684) (owner: 10Arendpieter) [20:01:46] !log eevans@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore2005.codfw.wmnet with reason: Firmware upgrades — T437516 [20:01:49] T437516: sessionstore: Debian Bookworm upgrades - https://phabricator.wikimedia.org/T437516 [20:02:17] !log eevans@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sessionstore2005.codfw.wmnet [20:02:45] (03CR) 10Dzahn: [C:03+2] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1326812 (https://phabricator.wikimedia.org/T435186) (owner: 10Hashar) [20:03:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdrewniak@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338975 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [20:04:11] (03CR) 10Dzahn: "yea, in eqiad there was a test what happens with 2 main nodes - more or less. to decide how to move forward (1 main node vs 2 main nodes i" [puppet] - 10https://gerrit.wikimedia.org/r/1326812 (https://phabricator.wikimedia.org/T435186) (owner: 10Hashar) [20:05:34] (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338975 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [20:05:48] !log jdrewniak@deploy1003 Started scap sync-world: Backport for [[gerrit:1338975|Bumping portals to master (T128546)]] [20:05:51] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [20:07:03] !log jdrewniak@deploy1003 sync-world failed: Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.18,1.47.0-wmf.19,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted [20:07:04] /mediawiki-multiversion --singleversion-image-basename docker-registry.discovery.wmnet/restricted/mediawiki-singleversion --webserver-image-name docker-registry.discovery.wmnet/restricted/mediawiki-webserver --latest-tag latest --label vnd.wikimedia.builder.name=scap --label vnd.wikimedia.builder.version=4.289.0 --label vnd.wikimedia.scap.stage_dir=/srv/mediawiki-staging --label vnd.wikimedia.scap.build_state_dir=/srv/med [20:07:04] iawiki-staging/scap/image-build' returned non-zero exit status 1. (scap version: 4.289.0) (duration: 01m 16s) [20:08:03] hmmm [20:08:05] (03PS2) 10Bking: an-worker1144: move to insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1339115 [20:08:18] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1339115 (owner: 10Bking) [20:10:08] jan_drewniak_: Try again please [20:10:59] !log jdrewniak@deploy1003 Started scap sync-world: Backport for [[gerrit:1338975|Bumping portals to master (T128546)]] [20:11:02] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [20:11:11] dancy: ah, didn't notice the error thanks! [20:12:59] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1346.eqiad.wmnet with OS trixie [20:13:20] (03CR) 10JHathaway: [C:03+1] Rename puppetmaster::puppetdb to puppetdb [puppet] - 10https://gerrit.wikimedia.org/r/1331470 (owner: 10Muehlenhoff) [20:13:47] (03CR) 10JHathaway: [C:03+1] Remove puppet/config-master records pointing to puppetmaster1001 [dns] - 10https://gerrit.wikimedia.org/r/1337819 (https://phabricator.wikimedia.org/T416606) (owner: 10Muehlenhoff) [20:14:49] !log jdrewniak@deploy1003 jdrewniak: Backport for [[gerrit:1338975|Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:15:03] I need a deployer for my patch [20:15:13] !log eevans@cumin1003 START - Cookbook sre.hosts.reboot-single for host sessionstore2005.codfw.wmnet [20:15:42] (03PS1) 10Dzahn: Revert "zuul: add firewall rules for Zookeeper" [puppet] - 10https://gerrit.wikimedia.org/r/1339123 [20:16:29] (03CR) 10Dzahn: "Error: /Stage[main]/Ferm/Service[ferm]: Failed to call refresh: Systemd restart for ferm failed!" [puppet] - 10https://gerrit.wikimedia.org/r/1339123 (owner: 10Dzahn) [20:16:58] (03CR) 10Dzahn: [C:03+2] Revert "zuul: add firewall rules for Zookeeper" [puppet] - 10https://gerrit.wikimedia.org/r/1339123 (owner: 10Dzahn) [20:17:47] (03CR) 10Dzahn: [C:03+2] "other pings requiring real time action meant I had no time to deal with this but needs follow-up. see "expects variable name" etc" [puppet] - 10https://gerrit.wikimedia.org/r/1339123 (owner: 10Dzahn) [20:17:51] rzl@cumin2003 renumber-node (PID 930166) is awaiting input [20:17:54] !log jdrewniak@deploy1003 jdrewniak: Continuing with deployment [20:18:40] FIRING: [2x] ProbeDown: Service sessionstore2005-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:18:51] obvious-dust: I can deploy your patches [20:19:37] thank you 🙏 [20:19:37] 06SRE, 10SRE-Access-Requests: Requesting access to growthbook-customelevatedaccess - https://phabricator.wikimedia.org/T437611#12309911 (10ssingh) >>! In T437611#12309824, @KReid-WMF wrote: > I approve the growthbook-customelevatedaccess access. > > Note that this group (growthbook-customelevatedaccess) can... [20:20:21] 06SRE, 10SRE-Access-Requests: Requesting access to growthbook-customelevatedaccess - https://phabricator.wikimedia.org/T437611#12309914 (10ssingh) [20:20:33] (03CR) 10JHathaway: "@hashar@free.fr I think this is ready to merge in?" [puppet] - 10https://gerrit.wikimedia.org/r/1329284 (https://phabricator.wikimedia.org/T435917) (owner: 10Hashar) [20:22:23] !log jdrewniak@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338975|Bumping portals to master (T128546)]] (duration: 11m 24s) [20:22:26] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [20:23:19] (03PS1) 10Ssingh: admin: move anil to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) [20:24:11] (03CR) 10CI reject: [V:04-1] admin: move anil to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) (owner: 10Ssingh) [20:24:54] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [20:25:33] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [20:25:34] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [20:25:37] (03CR) 10Dzahn: "it's because they already exist in the "ldap_only" section in this file. they need to be removed there." [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) (owner: 10Ssingh) [20:26:10] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [20:26:31] (03CR) 10Ssingh: "Yeah, I see that but now I am confused because I see https://ldap.toolforge.org/user/anil as well but also https://ldap.toolforge.org/user" [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) (owner: 10Ssingh) [20:26:32] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdrewniak@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328215 (https://phabricator.wikimedia.org/T434267) (owner: 10Milazg) [20:27:09] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore2005.codfw.wmnet [20:27:11] !log eevans@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts sessionstore2005.codfw.wmnet [20:27:51] !log eevans@cumin1003 START - Cookbook sre.hosts.provision for host sessionstore2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [20:28:08] (03Merged) 10jenkins-bot: Remove mode from RestModuleOverrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328215 (https://phabricator.wikimedia.org/T434267) (owner: 10Milazg) [20:28:12] rzl@cumin2003 renumber-node (PID 930166) is awaiting input [20:28:21] !log jdrewniak@deploy1003 Started scap sync-world: Backport for [[gerrit:1328215|Remove mode from RestModuleOverrides (T434267)]] [20:28:25] T434267: Add $wgRestModuleOverrides config for fragments module - https://phabricator.wikimedia.org/T434267 [20:28:30] logmsgbot: yeah yeah I know, I'm waiting for something to finish running [20:28:30] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [20:28:40] RESOLVED: [2x] ProbeDown: Service sessionstore2005-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:28:44] (03CR) 10Dzahn: "I check with this: [ldap-maint1001:~] $ ldapsearch -x mail=akanji*" [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) (owner: 10Ssingh) [20:29:17] (03CR) 10Ssingh: "Yeah. I am going to ask them I guess; let me know if you have any suggestions." [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) (owner: 10Ssingh) [20:29:43] (03CR) 10Dzahn: "While it's somewhat normal to have 2 users, one work and one volunteer, they should not both be linked to the @wikimedia.org email; but th" [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) (owner: 10Ssingh) [20:30:22] (03CR) 10Ssingh: "Thanks for the help. I will do that." [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) (owner: 10Ssingh) [20:30:56] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host sessionstore2005.codfw.wmnet with OS bookworm [20:31:19] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to growthbook-customelevatedaccess - https://phabricator.wikimedia.org/T437611#12309943 (10ssingh) Hi @AKanji-WMF. We see two accounts under your name, https://ldap.toolforge.org/user/anil and https://ldap.toolforge.org/user/anilk, both with... [20:31:27] (03CR) 10Dzahn: "the second, newer one, with the higher UID is "better" because that is linked to a Phabricator ID and a Global Wiki ID." [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) (owner: 10Ssingh) [20:32:43] !log jdrewniak@deploy1003 jdrewniak, milazg: Backport for [[gerrit:1328215|Remove mode from RestModuleOverrides (T434267)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:33:37] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to growthbook-customelevatedaccess - https://phabricator.wikimedia.org/T437611#12309957 (10Dzahn) The newer, anilk, account seems a bit better because that is already linked to a Phabricator ID and a global Wiki user, which is the WMF user.... [20:34:55] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1346.eqiad.wmnet [20:34:57] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1346.eqiad.wmnet [20:34:59] !log rzl@cumin2003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1346.eqiad.wmnet [20:35:07] FIRING: ProbeDown: Service sessionstore2005-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#sessionstore2005-a:9042 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:36:55] 06SRE, 10SRE-Access-Requests: Requesting access to deployment and restricted for WRai-WMF - https://phabricator.wikimedia.org/T437652 (10WRai-WMF) 03NEW [20:38:50] (03CR) 10JHathaway: [C:03+1] Add pki svc IP A and PTR records [dns] - 10https://gerrit.wikimedia.org/r/1338234 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [20:40:07] FIRING: [2x] ProbeDown: Service sessionstore2005-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:44:19] (03CR) 10JHathaway: [C:03+1] Add conftool-data for the pki[12]003 VMs [puppet] - 10https://gerrit.wikimedia.org/r/1338236 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [20:45:26] (03CR) 10JHathaway: [C:03+1] role::pki: add lvs configurations [puppet] - 10https://gerrit.wikimedia.org/r/1338936 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [20:47:39] !log jdrewniak@deploy1003 jdrewniak, milazg: Continuing with deployment [20:47:50] 06SRE, 10SRE-Access-Requests: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653 (10Cklimas) 03NEW [20:50:07] RESOLVED: [2x] ProbeDown: Service sessionstore2005-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:51:24] !log eevans@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore2005.codfw.wmnet with reason: host reimage [20:52:11] !log jdrewniak@deploy1003 Finished scap sync-world: Backport for [[gerrit:1328215|Remove mode from RestModuleOverrides (T434267)]] (duration: 23m 49s) [20:52:14] T434267: Add $wgRestModuleOverrides config for fragments module - https://phabricator.wikimedia.org/T434267 [20:54:33] tgr_: I'm done with my deploys [20:54:57] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore2005.codfw.wmnet with reason: host reimage [20:55:19] (03PS1) 10Jdlrobson: Instrument the donor account dialog [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1339075 (https://phabricator.wikimedia.org/T435565) [20:55:23] (03PS1) 10Jdlrobson: DonorIdentification: Adjust experiment behavior [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338301 (https://phabricator.wikimedia.org/T435534) [20:55:41] jan_drewniak_: thx! [20:56:04] jan_drewniak_: can i spiderpig at the same time? [20:56:23] (I'm in our hangout if you want to join early) [20:56:40] Jdlrobson: do you want to add another patch? [20:56:52] tgr_: oh i have a window in 4m time [20:57:03] i was gonna start early if you are done and that's okay. [20:57:17] I was just about to start [20:57:27] but I can come back later [20:57:36] oh... [20:57:50] did i read calendar wrong? [20:58:09] or maybe not, but I can do it another time, it's not urgent [20:58:32] ok i'll try and get wrapped up in 30m. It's for an experiment launch on Monday and I need to set up Elena for testing tomorrow. [20:58:55] ack [20:59:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1339076 (https://phabricator.wikimedia.org/T435565) (owner: 10Jdlrobson) [20:59:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1339075 (https://phabricator.wikimedia.org/T435565) (owner: 10Jdlrobson) [20:59:08] mine is not time-sensitive so don't worry about it [20:59:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338301 (https://phabricator.wikimedia.org/T435534) (owner: 10Jdlrobson) [20:59:09] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338287 (https://phabricator.wikimedia.org/T436572) (owner: 10Jdlrobson) [20:59:32] (03PS3) 10Bking: an-worker1144: move to insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1339115 (https://phabricator.wikimedia.org/T435873) [21:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260910T2100) [21:00:28] i'm also going to want to merge something but i can go whenever [21:00:55] tgr_: are you still going to want to merge in this window? [21:01:21] (03Merged) 10jenkins-bot: Instrument donor ID dialog: hooks defined [extensions/WikimediaEvents] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1339076 (https://phabricator.wikimedia.org/T435565) (owner: 10Jdlrobson) [21:02:22] derenrich: no, I'll do it next week [21:02:28] ok thanks [21:02:47] (03Merged) 10jenkins-bot: Allow donor consent workflow to work on non-Minerva skins [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338287 (https://phabricator.wikimedia.org/T436572) (owner: 10Jdlrobson) [21:02:49] (03Merged) 10jenkins-bot: DonorIdentification: Adjust experiment behavior [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1338301 (https://phabricator.wikimedia.org/T435534) (owner: 10Jdlrobson) [21:02:52] (03Merged) 10jenkins-bot: Instrument the donor account dialog [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1339075 (https://phabricator.wikimedia.org/T435565) (owner: 10Jdlrobson) [21:03:12] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1339076|Instrument donor ID dialog: hooks defined (T435565)]], [[gerrit:1339075|Instrument the donor account dialog (T435565)]], [[gerrit:1338301|DonorIdentification: Adjust experiment behavior (T435534)]], [[gerrit:1338287|Allow donor consent workflow to work on non-Minerva skins (T436572)]] [21:03:19] T435565: Add instrumentation to the donor consent experiment - https://phabricator.wikimedia.org/T435565 [21:03:20] T435534: DE5.4.1: Wire up buttons in consent overlay - https://phabricator.wikimedia.org/T435534 [21:03:20] T436572: Donor consent dialog needs to work in other skins - https://phabricator.wikimedia.org/T436572 [21:06:07] (03CR) 10DLynch: "A comparison occurred to me that might be helpful: the patch trades an inconsistent no-highlighting state (the article you're editing is l" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [21:06:41] (03CR) 10Ryan Kemper: [C:03+1] an-worker1144: move to insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1339115 (https://phabricator.wikimedia.org/T435873) (owner: 10Bking) [21:07:31] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1339076|Instrument donor ID dialog: hooks defined (T435565)]], [[gerrit:1339075|Instrument the donor account dialog (T435565)]], [[gerrit:1338301|DonorIdentification: Adjust experiment behavior (T435534)]], [[gerrit:1338287|Allow donor consent workflow to work on non-Minerva skins (T436572)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdeb [21:07:31] ug). Changes can now be verified there. [21:09:25] Hey all - have a couple of sec patches I’d like to get out, if possible. Any deployments or Readers Web stuff going on rn? [21:12:25] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [21:13:34] sbassett: yes deploying at moment and then @derenrich and @tgr_ are after [21:13:47] 👋 [21:13:51] well, ok, these might just have to wait until monday then [21:14:43] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore2005.codfw.wmnet with OS bookworm [21:17:06] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1339076|Instrument donor ID dialog: hooks defined (T435565)]], [[gerrit:1339075|Instrument the donor account dialog (T435565)]], [[gerrit:1338301|DonorIdentification: Adjust experiment behavior (T435534)]], [[gerrit:1338287|Allow donor consent workflow to work on non-Minerva skins (T436572)]] (duration: 13m 54s) [21:17:13] T435565: Add instrumentation to the donor consent experiment - https://phabricator.wikimedia.org/T435565 [21:17:13] T435534: DE5.4.1: Wire up buttons in consent overlay - https://phabricator.wikimedia.org/T435534 [21:17:13] T436572: Donor consent dialog needs to work in other skins - https://phabricator.wikimedia.org/T436572 [21:19:17] !log eevans@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore2006.codfw.wmnet with reason: Firmware upgrades — T437516 [21:19:20] T437516: sessionstore: Debian Bookworm upgrades - https://phabricator.wikimedia.org/T437516 [21:19:36] !log eevans@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sessionstore2006.codfw.wmnet [21:19:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337984 (https://phabricator.wikimedia.org/T437113) (owner: 10Jdlrobson) [21:20:38] (03CR) 10Andrea Denisse: [V:03+1 C:03+2] grafana: Add grafana-plugins to grafana-server suscription [puppet] - 10https://gerrit.wikimedia.org/r/1338350 (https://phabricator.wikimedia.org/T436056) (owner: 10Andrea Denisse) [21:21:31] (03Merged) 10jenkins-bot: Enable ReadingLists on mediawiki and wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337984 (https://phabricator.wikimedia.org/T437113) (owner: 10Jdlrobson) [21:21:46] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1337984|Enable ReadingLists on mediawiki and wikitech (T437113)]] [21:21:50] T437113: Release reading list & bookmark to wikitech and mediawiki - https://phabricator.wikimedia.org/T437113 [21:23:53] (03PS4) 10Southparkfan: profile::mediawiki::scap_client: allow configuring betacluster_udp2log_host [puppet] - 10https://gerrit.wikimedia.org/r/1338305 (https://phabricator.wikimedia.org/T436469) [21:26:21] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1337984|Enable ReadingLists on mediawiki and wikitech (T437113)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:27:01] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [21:27:40] ok derenrich all done all yours [21:27:46] (it's just syncing now) [21:27:51] thanks [21:30:22] (03PS5) 10Southparkfan: profile::mediawiki::scap_client: allow configuring betacluster_udp2log_host [puppet] - 10https://gerrit.wikimedia.org/r/1338305 (https://phabricator.wikimedia.org/T436469) [21:31:31] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337984|Enable ReadingLists on mediawiki and wikitech (T437113)]] (duration: 09m 45s) [21:31:35] T437113: Release reading list & bookmark to wikitech and mediawiki - https://phabricator.wikimedia.org/T437113 [21:31:45] !log eevans@cumin1003 START - Cookbook sre.hosts.reboot-single for host sessionstore2006.codfw.wmnet [21:32:16] ok starting backport [21:32:27] !log rzl@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1348.eqiad.wmnet [21:32:31] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1348.eqiad.wmnet [21:32:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by derenrich@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338439 (https://phabricator.wikimedia.org/T437344) (owner: 10DErenrich) [21:33:26] (03Merged) 10jenkins-bot: Enable discord preview extension code on testwiki (tk 2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338439 (https://phabricator.wikimedia.org/T437344) (owner: 10DErenrich) [21:33:41] !log derenrich@deploy1003 Started scap sync-world: Backport for [[gerrit:1338439|Enable discord preview extension code on testwiki (tk 2) (T437344 T431352)]] [21:33:42] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1348.eqiad.wmnet [21:33:46] T437344: Enable WMC Discord REST Endpoint - https://phabricator.wikimedia.org/T437344 [21:33:46] T431352: QuickSurveys: Show a survey to readers arriving from Discord via wprov link parameter - https://phabricator.wikimedia.org/T431352 [21:33:56] !log rzl@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1348.eqiad.wmnet with OS trixie [21:34:26] !log rzl@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1348 [21:35:07] FIRING: ProbeDown: Service sessionstore2006-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#sessionstore2006-a:9042 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:35:21] !log rzl@cumin2003 START - Cookbook sre.dns.netbox [21:37:52] !log derenrich@deploy1003 derenrich: Backport for [[gerrit:1338439|Enable discord preview extension code on testwiki (tk 2) (T437344 T431352)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:39:21] !log rzl@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1348 - rzl@cumin2003" [21:39:25] !log rzl@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1348 - rzl@cumin2003" [21:39:26] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [21:39:26] !log rzl@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1348.eqiad.wmnet 197.48.64.10.in-addr.arpa 7.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [21:39:29] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1348.eqiad.wmnet 197.48.64.10.in-addr.arpa 7.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [21:39:30] !log rzl@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1348 [21:40:07] !log rzl@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1348 [21:40:07] FIRING: [2x] ProbeDown: Service sessionstore2006-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:40:07] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1348 [21:42:33] !log derenrich@deploy1003 derenrich: Continuing with deployment [21:44:14] (03PS1) 10Andrew Bogott: Export a few stats about the magnum capi worker cluster [puppet] - 10https://gerrit.wikimedia.org/r/1339182 (https://phabricator.wikimedia.org/T429557) [21:44:24] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore2006.codfw.wmnet [21:44:26] !log eevans@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts sessionstore2006.codfw.wmnet [21:44:52] (03CR) 10CI reject: [V:04-1] Export a few stats about the magnum capi worker cluster [puppet] - 10https://gerrit.wikimedia.org/r/1339182 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [21:45:12] RESOLVED: [2x] ProbeDown: Service sessionstore2006-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:46:09] !log eevans@cumin1003 START - Cookbook sre.hosts.provision for host sessionstore2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [21:46:30] (03PS2) 10Andrew Bogott: Export a few stats about the magnum capi worker cluster [puppet] - 10https://gerrit.wikimedia.org/r/1339182 (https://phabricator.wikimedia.org/T429557) [21:46:49] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [21:47:04] !log derenrich@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338439|Enable discord preview extension code on testwiki (tk 2) (T437344 T431352)]] (duration: 13m 23s) [21:47:08] T437344: Enable WMC Discord REST Endpoint - https://phabricator.wikimedia.org/T437344 [21:47:08] T431352: QuickSurveys: Show a survey to readers arriving from Discord via wprov link parameter - https://phabricator.wikimedia.org/T431352 [21:47:38] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host sessionstore2006.codfw.wmnet with OS bookworm [21:50:59] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1339182 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [21:52:14] !log rzl@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1348.eqiad.wmnet with reason: host reimage [21:56:06] PROBLEM - Host wikikube-worker1348 is DOWN: PING CRITICAL - Packet loss = 100% [21:56:10] FIRING: [2x] ProbeDown: Service sessionstore2006-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:56:12] (03PS1) 10Andrew Bogott: magnum: move path to the capi worker kubeconfig to hiera [puppet] - 10https://gerrit.wikimedia.org/r/1339193 [21:56:47] (03CR) 10CI reject: [V:04-1] magnum: move path to the capi worker kubeconfig to hiera [puppet] - 10https://gerrit.wikimedia.org/r/1339193 (owner: 10Andrew Bogott) [21:57:58] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1348.eqiad.wmnet with reason: host reimage [21:58:08] (03PS2) 10Andrew Bogott: magnum: move path to the capi worker kubeconfig to hiera [puppet] - 10https://gerrit.wikimedia.org/r/1339193 [21:58:38] (03PS3) 10Andrew Bogott: magnum: move path to the capi worker kubeconfig to hiera [puppet] - 10https://gerrit.wikimedia.org/r/1339193 [21:59:34] (03CR) 10TrainBranchBot: [C:03+2] "Approved by musikanimal@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1322175 (https://phabricator.wikimedia.org/T432558) (owner: 10DLynch) [21:59:34] (03CR) 10TrainBranchBot: [C:03+2] "Approved by musikanimal@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [21:59:42] RECOVERY - Host wikikube-worker1348 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [22:00:28] !log eevans@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore2006.codfw.wmnet with OS bookworm [22:00:41] (03Merged) 10jenkins-bot: CodeMirror: turn on the new 2017 editor integration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1322175 (https://phabricator.wikimedia.org/T432558) (owner: 10DLynch) [22:00:42] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host sessionstore2006.codfw.wmnet with OS bookworm [22:00:44] (03Merged) 10jenkins-bot: [CodeMirror] enable for new and logged-out users by default on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [22:01:00] !log musikanimal@deploy1003 Started scap sync-world: Backport for [[gerrit:1322175|CodeMirror: turn on the new 2017 editor integration (T432558)]], [[gerrit:1338308|[CodeMirror] enable for new and logged-out users by default on enwiki (T288161)]] [22:01:05] T432558: The 2017 wikitext editor performs very poorly when CM6 is enabled - https://phabricator.wikimedia.org/T432558 [22:01:06] T288161: Enable CodeMirror's Syntax Highlighting on English Wikipedia by default - https://phabricator.wikimedia.org/T288161 [22:01:40] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:05:12] (03PS4) 10Andrew Bogott: magnum: move capi worker kubeconfig path to hiera [puppet] - 10https://gerrit.wikimedia.org/r/1339193 [22:05:12] (03PS3) 10Andrew Bogott: Export a few stats about the magnum capi worker cluster [puppet] - 10https://gerrit.wikimedia.org/r/1339182 (https://phabricator.wikimedia.org/T429557) [22:05:45] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1339193 (owner: 10Andrew Bogott) [22:05:46] !log musikanimal@deploy1003 kemayo, musikanimal: Backport for [[gerrit:1322175|CodeMirror: turn on the new 2017 editor integration (T432558)]], [[gerrit:1338308|[CodeMirror] enable for new and logged-out users by default on enwiki (T288161)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:06:15] (03CR) 10CI reject: [V:04-1] Export a few stats about the magnum capi worker cluster [puppet] - 10https://gerrit.wikimedia.org/r/1339182 (https://phabricator.wikimedia.org/T429557) (owner: 10Andrew Bogott) [22:06:46] (03PS5) 10Southparkfan: profile::mediawiki::scap_client: allow configuring betacluster_udp2log_host [puppet] - 10https://gerrit.wikimedia.org/r/1338305 (https://phabricator.wikimedia.org/T436469) [22:06:56] !log musikanimal@deploy1003 kemayo, musikanimal: Rolling back deployment [22:09:57] (03PS1) 10MusikAnimal: Revert "[CodeMirror] enable for new and logged-out users by default on enwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339204 [22:10:43] (03CR) 10MusikAnimal: [C:03+2] Revert "[CodeMirror] enable for new and logged-out users by default on enwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339204 (owner: 10MusikAnimal) [22:11:45] (03Merged) 10jenkins-bot: Revert "[CodeMirror] enable for new and logged-out users by default on enwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339204 (owner: 10MusikAnimal) [22:12:00] !log musikanimal@deploy1003 Finished scap sync-world: Backport for [[gerrit:1322175|CodeMirror: turn on the new 2017 editor integration (T432558)]], [[gerrit:1338308|[CodeMirror] enable for new and logged-out users by default on enwiki (T288161)]] (duration: 10m 59s) [22:12:05] T432558: The 2017 wikitext editor performs very poorly when CM6 is enabled - https://phabricator.wikimedia.org/T432558 [22:12:05] T288161: Enable CodeMirror's Syntax Highlighting on English Wikipedia by default - https://phabricator.wikimedia.org/T288161 [22:13:17] (03PS1) 10MusikAnimal: Revert "CodeMirror: turn on the new 2017 editor integration" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339207 [22:13:27] (03CR) 10MusikAnimal: [C:03+2] Revert "CodeMirror: turn on the new 2017 editor integration" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339207 (owner: 10MusikAnimal) [22:14:17] (03Merged) 10jenkins-bot: Revert "CodeMirror: turn on the new 2017 editor integration" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339207 (owner: 10MusikAnimal) [22:15:17] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1348.eqiad.wmnet with OS trixie [22:18:38] (03CR) 10MusikAnimal: "Well, the config change didn't work anyway. Existing users with `usecodemirror=1` were not getting CM. No good!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338308 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [22:20:02] (03PS1) 10Bking: Druid: Point PSI alerts to the correct prometheus instance [alerts] - 10https://gerrit.wikimedia.org/r/1339209 (https://phabricator.wikimedia.org/T436968) [22:20:33] !log eevans@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore2006.codfw.wmnet with reason: host reimage [22:21:10] RESOLVED: [2x] ProbeDown: Service sessionstore2006-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:23:46] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore2006.codfw.wmnet with reason: host reimage [22:24:15] 06SRE, 10SRE-Access-Requests: Requesting access to deployment and restricted for cooltey - https://phabricator.wikimedia.org/T437658 (10cooltey) 03NEW [22:24:55] 06SRE, 10SRE-Access-Requests: Requesting access to deployment and restricted for cooltey - https://phabricator.wikimedia.org/T437658#12310319 (10Seddon) Approved as Manager. [22:25:17] 06SRE, 10SRE-Access-Requests: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653#12310320 (10Seddon) Approved as Manager [22:25:19] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1348.eqiad.wmnet [22:25:21] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1348.eqiad.wmnet [22:25:23] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1348.eqiad.wmnet [22:25:24] 06SRE, 10SRE-Access-Requests: Requesting access to deployment and restricted for WRai-WMF - https://phabricator.wikimedia.org/T437652#12310322 (10Seddon) Approved as Manager [22:26:33] !log rzl@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1349.eqiad.wmnet [22:26:37] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1349.eqiad.wmnet [22:27:04] (03PS4) 10Andrew Bogott: Export a few stats about the magnum capi worker cluster [puppet] - 10https://gerrit.wikimedia.org/r/1339182 (https://phabricator.wikimedia.org/T429557) [22:27:15] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1349.eqiad.wmnet [22:27:55] !log rzl@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1349.eqiad.wmnet with OS trixie [22:28:24] !log rzl@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1349 [22:28:42] !log rzl@cumin2003 START - Cookbook sre.dns.netbox [22:32:44] !log rzl@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1349 - rzl@cumin2003" [22:32:48] !log rzl@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1349 - rzl@cumin2003" [22:32:48] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [22:32:49] !log rzl@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1349.eqiad.wmnet 198.48.64.10.in-addr.arpa 8.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [22:32:52] !log rzl@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1349.eqiad.wmnet 198.48.64.10.in-addr.arpa 8.9.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [22:32:52] !log rzl@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1349 [22:33:46] !log rzl@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1349 [22:33:46] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1349 [22:44:45] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore2006.codfw.wmnet with OS bookworm [22:44:45] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) (owner: 10Tryvix1509) [22:46:02] !log rzl@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1349.eqiad.wmnet with reason: host reimage [22:49:07] FIRING: ProbeDown: Service sessionstore2006-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#sessionstore2006-a:9042 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:49:56] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1349.eqiad.wmnet with reason: host reimage [22:54:07] FIRING: [2x] ProbeDown: Service sessionstore2006-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:59:07] RESOLVED: [2x] ProbeDown: Service sessionstore2006-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:09:12] !log rzl@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1349.eqiad.wmnet with OS trixie [23:18:26] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662 (10LPetty-WMF) 03NEW [23:20:58] !log rzl@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1349.eqiad.wmnet [23:21:00] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1349.eqiad.wmnet [23:21:02] !log rzl@cumin2003 END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1349.eqiad.wmnet [23:29:55] Any concerns with me doing a beta cluster only deploy for someone in next 30 m? [23:30:51] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12310498 (10Seddon) Approved as manager [23:40:59] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1339232 [23:40:59] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1339232 (owner: 10TrainBranchBot) [23:48:54] (03PS1) 10Jdlrobson: Enable experiment on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339234 (https://phabricator.wikimedia.org/T437665) [23:51:09] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1339232 (owner: 10TrainBranchBot) [23:51:10] (03CR) 10VolkerE: [C:03+1] Enable experiment on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339234 (https://phabricator.wikimedia.org/T437665) (owner: 10Jdlrobson) [23:51:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339234 (https://phabricator.wikimedia.org/T437665) (owner: 10Jdlrobson) [23:52:18] (03Merged) 10jenkins-bot: Enable experiment on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339234 (https://phabricator.wikimedia.org/T437665) (owner: 10Jdlrobson) [23:54:40] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users level 3 for derenrich - https://phabricator.wikimedia.org/T437668 (10derenrich) 03NEW