[02:00:05] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T0200) [02:00:49] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:08:23] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 33s) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:37:27] PROBLEM - Host cloudnet2006-dev is DOWN: PING CRITICAL - Packet loss = 100% [02:40:13] RECOVERY - Host cloudnet2006-dev is UP: PING OK - Packet loss = 0%, RTA = 30.33 ms [02:49:15] (03PS1) 10Andrew Bogott: cloudnet codfw1dev: add clientpackages profile [puppet] - 10https://gerrit.wikimedia.org/r/1332849 [02:49:33] PROBLEM - SSH on an-druid1006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [02:50:12] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332849 (owner: 10Andrew Bogott) [02:52:22] (03CR) 10Andrew Bogott: [C:03+2] cloudnet codfw1dev: add clientpackages profile [puppet] - 10https://gerrit.wikimedia.org/r/1332849 (owner: 10Andrew Bogott) [03:00:05] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T0300) [03:01:35] PROBLEM - Host cloudnet2006-dev is DOWN: PING CRITICAL - Packet loss = 100% [03:02:02] (03PS1) 10TrainBranchBot: testwikis to 1.47.0-wmf.18 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332851 (https://phabricator.wikimedia.org/T430837) [03:02:04] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by mwpresync@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332851 (https://phabricator.wikimedia.org/T430837) (owner: 10TrainBranchBot) [03:02:59] (03Merged) 10jenkins-bot: testwikis to 1.47.0-wmf.18 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332851 (https://phabricator.wikimedia.org/T430837) (owner: 10TrainBranchBot) [03:03:23] !log mwpresync@deploy1003 Started scap sync-world: testwikis to 1.47.0-wmf.18 refs T430837 [03:03:26] T430837: 1.47.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T430837 [03:04:13] RECOVERY - Host cloudnet2006-dev is UP: PING OK - Packet loss = 0%, RTA = 33.04 ms [03:11:45] PROBLEM - Host cloudnet2005-dev is DOWN: PING CRITICAL - Packet loss = 100% [03:15:13] RECOVERY - Host cloudnet2005-dev is UP: PING OK - Packet loss = 0%, RTA = 30.31 ms [03:16:23] RECOVERY - SSH on an-druid1006 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [03:16:23] PROBLEM - Druid historical on an-druid1006 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [03:18:23] RECOVERY - Druid historical on an-druid1006 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [03:40:53] !log mwpresync@deploy1003 Finished scap sync-world: testwikis to 1.47.0-wmf.18 refs T430837 (duration: 37m 30s) [03:40:56] T430837: 1.47.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T430837 [03:50:18] !log denisse@deploy1003 Started deploy [librenms/librenms@9b0b502]: Upgrade LibreNMS to 26.8.2 [03:50:37] !log denisse@deploy1003 Finished deploy [librenms/librenms@9b0b502]: Upgrade LibreNMS to 26.8.2 (duration: 00m 19s) [04:00:05] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T0400) [04:02:31] !log mwpresync@deploy1003 Pruned MediaWiki: 1.47.0-wmf.15 (duration: 02m 25s) [04:24:32] FIRING: [2x] Traffic bill over quota: Alert for device cr1-drmrs.wikimedia.org - Traffic bill over quota Has worsened - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [04:25:59] PROBLEM - Backup freshness on backup1014 is CRITICAL: All failures: 3 (cumin1004, ...), Fresh: 142 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [04:38:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:43:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:44:32] RESOLVED: [2x] Traffic bill over quota: Alert for device cr1-drmrs.wikimedia.org - Traffic bill over quota Has worsened - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [05:06:52] FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [05:10:03] FIRING: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [05:25:38] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-d4-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T0600) [06:00:05] marostegui, cezmunsta, and federico3: Primary database switchover (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T0600). Please do the needful. [06:13:58] (03PS3) 10Giuseppe Lavagetto: aptrepo: add thirdparty/gvisor to newer distros as well [puppet] - 10https://gerrit.wikimedia.org/r/1328540 (https://phabricator.wikimedia.org/T435758) [06:13:58] (03PS2) 10Giuseppe Lavagetto: profile::containerd: format according to our style guide [puppet] - 10https://gerrit.wikimedia.org/r/1330128 [06:13:59] (03PS2) 10Giuseppe Lavagetto: profile::containerd: add gVisor support [puppet] - 10https://gerrit.wikimedia.org/r/1330129 (https://phabricator.wikimedia.org/T435796) [06:13:59] (03PS2) 10Giuseppe Lavagetto: containerd: use hiera parameters to detect if dragonfly is enabled [puppet] - 10https://gerrit.wikimedia.org/r/1330130 [06:14:00] (03PS2) 10Giuseppe Lavagetto: kubernetes::staging: make containerd support gVisor runtime [puppet] - 10https://gerrit.wikimedia.org/r/1330131 (https://phabricator.wikimedia.org/T435796) [06:14:01] (03PS2) 10Giuseppe Lavagetto: profile::kubernetes::node: Add gvisor labels [puppet] - 10https://gerrit.wikimedia.org/r/1330187 (https://phabricator.wikimedia.org/T436212) [06:17:20] (03CR) 10CI reject: [V:04-1] profile::kubernetes::node: Add gvisor labels [puppet] - 10https://gerrit.wikimedia.org/r/1330187 (https://phabricator.wikimedia.org/T436212) (owner: 10Giuseppe Lavagetto) [06:18:42] (03PS4) 10Giuseppe Lavagetto: admin: add support for gVisor RuntimeClass handlers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330344 (https://phabricator.wikimedia.org/T436212) [06:26:03] (03CR) 10Filippo Giunchedi: [C:03+1] P:toolforge::bastion: Remove temporary index migration script [puppet] - 10https://gerrit.wikimedia.org/r/1332769 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [06:26:12] (03CR) 10Filippo Giunchedi: [C:03+1] P:toolforge: Remove unused apt_pinning class [puppet] - 10https://gerrit.wikimedia.org/r/1332771 (owner: 10Majavah) [06:26:28] (03CR) 10Filippo Giunchedi: [C:03+1] O:wmcs::toolforge: Remove Elasticsearch related classes [puppet] - 10https://gerrit.wikimedia.org/r/1332770 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [06:29:35] !log installing Java 17 security updates [06:29:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:35:10] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12274035 (10MoritzMuehlenhoff) [06:38:11] (03CR) 10Filippo Giunchedi: [C:03+2] site: temp decom for cloudvirt10(6[567]|7[234]) ahead of rack move [puppet] - 10https://gerrit.wikimedia.org/r/1332754 (https://phabricator.wikimedia.org/T435921) (owner: 10Filippo Giunchedi) [06:38:16] (03CR) 10Filippo Giunchedi: [C:03+2] site: temp decom for cloudvirt1075 ahead of rack move [puppet] - 10https://gerrit.wikimedia.org/r/1332755 (https://phabricator.wikimedia.org/T435923) (owner: 10Filippo Giunchedi) [06:45:19] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1075.eqiad.wmnet [06:49:57] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [06:55:25] filippo@cumin1003 decommission (PID 1575938) is awaiting input [07:00:04] Amir1, urbanecm, and awight: Your horoscope predicts another UTC morning backport window deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T0700). [07:00:04] No Gerrit patches in the queue for this window AFAICS. [07:04:11] (03CR) 10Jasmine: "Thanks Simon!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [07:06:59] (03PS1) 10JMeybohm: validating-admission-policies: Update for k8s 1.34 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333038 (https://phabricator.wikimedia.org/T436380) [07:08:53] (03CR) 10Jasmine: mw-web: upsize for single-DC serving (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [07:13:53] (03CR) 10Jasmine: mw-web: upsize for single-DC serving (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [07:15:39] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1075.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:17:13] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1075.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:17:13] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [07:17:14] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1075.eqiad.wmnet [07:17:31] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12274066 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1075.eqiad.wmnet` - cloudvirt1075.eqiad.wmnet (**PASS**)... [07:18:26] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274069 (10fgiunchedi) [07:18:55] (03CR) 10Jasmine: mw-web: upsize for single-DC serving (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [07:21:33] filippo@cumin1003 decommission (PID 1580061) is awaiting input [07:23:10] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1065.eqiad.wmnet [07:23:46] (03CR) 10Majavah: [C:03+2] P:toolforge::bastion: Remove temporary index migration script [puppet] - 10https://gerrit.wikimedia.org/r/1332769 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [07:24:00] (03CR) 10Majavah: [C:03+2] O:wmcs::toolforge: Remove Elasticsearch related classes [puppet] - 10https://gerrit.wikimedia.org/r/1332770 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [07:24:16] (03CR) 10Majavah: [C:03+2] P:toolforge: Remove unused apt_pinning class [puppet] - 10https://gerrit.wikimedia.org/r/1332771 (owner: 10Majavah) [07:25:47] (03PS5) 10Tiziano Fogli: prometheus: add cardinality exporter (ops instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332549 (https://phabricator.wikimedia.org/T435334) [07:25:47] (03PS5) 10Tiziano Fogli: prometheus: add cardinality exporter (ext instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332550 (https://phabricator.wikimedia.org/T435334) [07:25:47] (03PS5) 10Tiziano Fogli: prometheus: add cardinality exporter (analytics instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332551 (https://phabricator.wikimedia.org/T435334) [07:25:48] (03PS5) 10Tiziano Fogli: prometheus: add cardinality exporter (cloud instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332552 (https://phabricator.wikimedia.org/T435334) [07:25:49] (03PS5) 10Tiziano Fogli: prometheus: add cardinality exporter (services instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332553 (https://phabricator.wikimedia.org/T435334) [07:25:51] (03PS5) 10Tiziano Fogli: prometheus: add cardinality exporter (k8s instances) [puppet] - 10https://gerrit.wikimedia.org/r/1332554 (https://phabricator.wikimedia.org/T435334) [07:25:55] (03PS1) 10Tiziano Fogli: prometheus: add cardinality exporter hiera values [puppet] - 10https://gerrit.wikimedia.org/r/1333040 (https://phabricator.wikimedia.org/T435334) [07:26:00] RECOVERY - Backup freshness on backup1014 is OK: Fresh: 145 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [07:27:47] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [07:28:46] (03CR) 10Filippo Giunchedi: [C:03+1] P:toolforge::prometheus: Remove Elasticsearch targets [puppet] - 10https://gerrit.wikimedia.org/r/1332777 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [07:29:02] (03PS2) 10Majavah: P:toolforge::prometheus: Remove Elasticsearch targets [puppet] - 10https://gerrit.wikimedia.org/r/1332777 (https://phabricator.wikimedia.org/T401818) [07:32:25] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1065.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:32:39] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1065.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:32:40] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [07:32:41] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1065.eqiad.wmnet [07:32:43] (03PS8) 10Tiziano Fogli: prometheus: add cardinality exporter [puppet] - 10https://gerrit.wikimedia.org/r/1331600 (https://phabricator.wikimedia.org/T435334) [07:32:43] (03PS2) 10Tiziano Fogli: prometheus: add cardinality exporter hiera values [puppet] - 10https://gerrit.wikimedia.org/r/1333040 (https://phabricator.wikimedia.org/T435334) [07:32:43] (03PS6) 10Tiziano Fogli: prometheus: add cardinality exporter (ops instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332549 (https://phabricator.wikimedia.org/T435334) [07:32:44] (03PS6) 10Tiziano Fogli: prometheus: add cardinality exporter (ext instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332550 (https://phabricator.wikimedia.org/T435334) [07:32:45] (03PS6) 10Tiziano Fogli: prometheus: add cardinality exporter (analytics instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332551 (https://phabricator.wikimedia.org/T435334) [07:32:46] (03PS6) 10Tiziano Fogli: prometheus: add cardinality exporter (cloud instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332552 (https://phabricator.wikimedia.org/T435334) [07:32:50] (03PS6) 10Tiziano Fogli: prometheus: add cardinality exporter (services instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332553 (https://phabricator.wikimedia.org/T435334) [07:32:54] (03PS6) 10Tiziano Fogli: prometheus: add cardinality exporter (k8s instances) [puppet] - 10https://gerrit.wikimedia.org/r/1332554 (https://phabricator.wikimedia.org/T435334) [07:33:00] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274093 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1065.eqiad.wmnet` - cloudvirt1065.eqiad.wmnet (... [07:34:27] (03CR) 10Majavah: [C:03+2] P:toolforge::prometheus: Remove Elasticsearch targets [puppet] - 10https://gerrit.wikimedia.org/r/1332777 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [07:34:54] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1066.eqiad.wmnet [07:37:57] (03PS9) 10Tiziano Fogli: prometheus: add cardinality exporter [puppet] - 10https://gerrit.wikimedia.org/r/1331600 (https://phabricator.wikimedia.org/T435334) [07:37:57] (03PS3) 10Tiziano Fogli: prometheus: add cardinality exporter hiera values [puppet] - 10https://gerrit.wikimedia.org/r/1333040 (https://phabricator.wikimedia.org/T435334) [07:37:57] (03PS7) 10Tiziano Fogli: prometheus: add cardinality exporter (ops instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332549 (https://phabricator.wikimedia.org/T435334) [07:37:58] (03PS7) 10Tiziano Fogli: prometheus: add cardinality exporter (ext instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332550 (https://phabricator.wikimedia.org/T435334) [07:37:59] (03PS7) 10Tiziano Fogli: prometheus: add cardinality exporter (analytics instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332551 (https://phabricator.wikimedia.org/T435334) [07:38:00] (03PS7) 10Tiziano Fogli: prometheus: add cardinality exporter (cloud instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332552 (https://phabricator.wikimedia.org/T435334) [07:38:04] (03PS7) 10Tiziano Fogli: prometheus: add cardinality exporter (services instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332553 (https://phabricator.wikimedia.org/T435334) [07:38:08] (03PS7) 10Tiziano Fogli: prometheus: add cardinality exporter (k8s instances) [puppet] - 10https://gerrit.wikimedia.org/r/1332554 (https://phabricator.wikimedia.org/T435334) [07:38:12] (03CR) 10CI reject: [V:04-1] prometheus: add cardinality exporter (services instance) [puppet] - 10https://gerrit.wikimedia.org/r/1332553 (https://phabricator.wikimedia.org/T435334) (owner: 10Tiziano Fogli) [07:38:17] (03CR) 10CI reject: [V:04-1] prometheus: add cardinality exporter (k8s instances) [puppet] - 10https://gerrit.wikimedia.org/r/1332554 (https://phabricator.wikimedia.org/T435334) (owner: 10Tiziano Fogli) [07:40:52] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] Use QLever example queries in query-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321920 (https://phabricator.wikimedia.org/T432638) (owner: 10Sadiya.mohammed13) [07:41:06] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [07:42:47] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie [07:42:57] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations: attempts to reimage cp5022 are failing - https://phabricator.wikimedia.org/T436386#12274100 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1003 for host cp5022.eqsin.wmnet with OS trixie [07:45:57] (03CR) 10Tiziano Fogli: [C:03+2] prometheus: add cardinality exporter (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1331600 (https://phabricator.wikimedia.org/T435334) (owner: 10Tiziano Fogli) [07:46:04] (03CR) 10Tiziano Fogli: [C:03+2] prometheus: add cardinality exporter hiera values [puppet] - 10https://gerrit.wikimedia.org/r/1333040 (https://phabricator.wikimedia.org/T435334) (owner: 10Tiziano Fogli) [07:46:24] (03CR) 10Tiziano Fogli: [C:03+2] prometheus: add cardinality exporter (ops instance) (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1332549 (https://phabricator.wikimedia.org/T435334) (owner: 10Tiziano Fogli) [07:46:56] (03CR) 10Tiziano Fogli: [C:03+2] prometheus: add cardinality exporter (ext instance) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1332550 (https://phabricator.wikimedia.org/T435334) (owner: 10Tiziano Fogli) [07:47:04] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1066.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:47:08] (03CR) 10Tiziano Fogli: [C:03+2] prometheus: add cardinality exporter (analytics instance) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1332551 (https://phabricator.wikimedia.org/T435334) (owner: 10Tiziano Fogli) [07:47:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.55% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:47:17] (03CR) 10Tiziano Fogli: [C:03+2] prometheus: add cardinality exporter (cloud instance) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1332552 (https://phabricator.wikimedia.org/T435334) (owner: 10Tiziano Fogli) [07:47:30] (03CR) 10Tiziano Fogli: [C:03+2] prometheus: add cardinality exporter (services instance) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1332553 (https://phabricator.wikimedia.org/T435334) (owner: 10Tiziano Fogli) [07:47:56] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1066.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:47:56] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [07:47:57] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1066.eqiad.wmnet [07:48:09] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274130 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1066.eqiad.wmnet` - cloudvirt1066.eqiad.wmnet (... [07:49:03] (03CR) 10Tiziano Fogli: [C:03+2] prometheus: add cardinality exporter (k8s instances) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1332554 (https://phabricator.wikimedia.org/T435334) (owner: 10Tiziano Fogli) [07:50:59] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1067.eqiad.wmnet [07:52:27] !log joal@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply [07:52:59] !log elukey@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie [07:53:06] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations: attempts to reimage cp5022 are failing - https://phabricator.wikimedia.org/T436386#12274160 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1003 for host cp5022.eqsin.wmnet with OS trixie executed with errors: - cp5022 (**FA... [07:53:06] !log joal@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply [07:53:13] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations: attempts to reimage cp5022 are failing - https://phabricator.wikimedia.org/T436386#12274163 (10elukey) @CDobbins o/ you can inspect live what happens during reimage connecting to the iDRAC's serial console: * ssh wmfroot@cp5022.mgmt.eqsin.wmnet * console com... [07:54:59] !log elukey@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp5022'] [07:56:07] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [07:57:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:00:08] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1067.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [08:02:18] (03CR) 10Brouberol: [C:04-1] "I've talked to @abran@wikimedia.org and I'm going to -1 this one. The code itself is fine, but I think we can change the way airflow/gitsy" [puppet] - 10https://gerrit.wikimedia.org/r/1332720 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [08:02:30] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1067.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [08:02:30] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:02:32] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1067.eqiad.wmnet [08:02:43] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274181 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1067.eqiad.wmnet` - cloudvirt1067.eqiad.wmnet (... [08:03:19] (03PS1) 10Muehlenhoff: Setup Homer sync config for cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1333067 (https://phabricator.wikimedia.org/T427897) [08:04:45] (03PS1) 10Joal: Add is_thumb_generated to webrequest_sampled turnilo conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333087 (https://phabricator.wikimedia.org/T435634) [08:05:39] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1072.eqiad.wmnet [08:07:03] (03PS1) 10Brouberol: gitlab: allow pods running in the dse clusters to egress to internal endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1333119 (https://phabricator.wikimedia.org/T435292) [08:07:06] (03PS1) 10Brouberol: global_config: register the internal gitlab endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1333120 (https://phabricator.wikimedia.org/T435292) [08:10:17] 06SRE, 10SRE-swift-storage, 06Infrastructure-Foundations: exim4.conf permissions mean xfs_scrub_all_fail.service cannot run - https://phabricator.wikimedia.org/T436615 (10MatthewVernon) 03NEW [08:10:27] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [08:10:32] (03PS1) 10Brouberol: airflow: allow gitsync to egress to the gitlab internal endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333124 (https://phabricator.wikimedia.org/T435292) [08:11:04] elukey@cumin1003 upgrade-firmware (PID 1587272) is awaiting input [08:13:10] (03CR) 10Elukey: [C:03+1] Setup Homer sync config for cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1333067 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [08:14:01] (03CR) 10Brouberol: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332512 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [08:14:41] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1072.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [08:15:04] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1072.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [08:15:04] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:15:05] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1072.eqiad.wmnet [08:15:06] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host urldownloader1003.wikimedia.org with OS bookworm [08:15:19] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274233 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1072.eqiad.wmnet` - cloudvirt1072.eqiad.wmnet (... [08:15:21] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12274234 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host urldownlo... [08:16:09] (03CR) 10Muehlenhoff: [C:03+2] Setup Homer sync config for cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1333067 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [08:16:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:16:28] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1073.eqiad.wmnet [08:19:13] (03CR) 10Trueg: [C:03+2] Allow up to 3 redirects for federated endpoints. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331787 (https://phabricator.wikimedia.org/T433133) (owner: 10Lerickson) [08:20:27] !log elukey@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp5022'] [08:21:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:21:37] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [08:21:54] (03Merged) 10jenkins-bot: Allow up to 3 redirects for federated endpoints. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331787 (https://phabricator.wikimedia.org/T433133) (owner: 10Lerickson) [08:22:43] 10ops-eqiad, 06SRE, 06DC-Ops: Lumen eqiad<->codfw transport outage (Aug 2026). - https://phabricator.wikimedia.org/T435810#12274271 (10ayounsi) OSPF was set to 5000 only on one side, I normalized it back to 500 now that the link has been stable. [08:22:48] 10ops-eqiad, 06SRE, 10Continuous-Integration-Infrastructure, 06DC-Ops, 06Release-Engineering-Team: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12274272 (10elukey) As FYI for Dells we set: ` 'EnergyPerformanceBias': 'BalancedPerformance', ` But afa... [08:24:16] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie [08:24:23] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations: attempts to reimage cp5022 are failing - https://phabricator.wikimedia.org/T436386#12274278 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1003 for host cp5022.eqsin.wmnet with OS trixie [08:24:53] 06SRE, 10Pywikibot, 06Traffic, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12274279 (10Xqt) >>! In T421642#12271821, @JeanFred wrote: > As I wrote back in T244030 (which was {T242081} back then): >>>! In T244030#5842377, @JeanFred w... [08:24:55] (03CR) 10Gergő Tisza: [C:03+1] "We only need ResourceLoader scripts to work on auth.wm.org and I assume if one of those work, all do. In any case, there aren't that many " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1330446 (https://phabricator.wikimedia.org/T419684) (owner: 10Arendpieter) [08:25:31] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1073.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [08:26:06] (03PS1) 10MVernon: exim4: use 0444 for exim4.conf not 0440 [puppet] - 10https://gerrit.wikimedia.org/r/1333127 (https://phabricator.wikimedia.org/T436615) [08:27:48] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1003.wikimedia.org with reason: host reimage [08:28:35] filippo@cumin1003 decommission (PID 1588817) is awaiting input [08:30:54] (03PS1) 10Muehlenhoff: Fix hostname in homer sync [puppet] - 10https://gerrit.wikimedia.org/r/1333128 [08:31:04] (03PS1) 10Marostegui: instances.yaml: Add db290[1-3] [puppet] - 10https://gerrit.wikimedia.org/r/1333129 (https://phabricator.wikimedia.org/T427059) [08:32:36] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1073.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [08:32:36] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:32:36] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db290[1-3] [puppet] - 10https://gerrit.wikimedia.org/r/1333129 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [08:32:37] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1073.eqiad.wmnet [08:32:49] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274301 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1073.eqiad.wmnet` - cloudvirt1073.eqiad.wmnet (... [08:32:51] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1074.eqiad.wmnet [08:33:06] (03PS1) 10Trueg: wdqs: Utilize the wdqs-proxy's readiness probe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333130 (https://phabricator.wikimedia.org/T432331) [08:34:25] (03CR) 10Elukey: [C:03+1] Fix hostname in homer sync [puppet] - 10https://gerrit.wikimedia.org/r/1333128 (owner: 10Muehlenhoff) [08:34:33] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db2901 to test-s4 T427059', diff saved to https://phabricator.wikimedia.org/P96283 and previous config saved to /var/cache/conftool/dbconfig/20260901-083432-marostegui.json [08:34:36] T427059: Discussion for adding support of test-s4 in dbctl - https://phabricator.wikimedia.org/T427059 [08:34:42] (03PS2) 10Muehlenhoff: Fix hostname in homer sync [puppet] - 10https://gerrit.wikimedia.org/r/1333128 [08:34:55] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1003.wikimedia.org with reason: host reimage [08:35:28] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db2902 to test-s4 T427059', diff saved to https://phabricator.wikimedia.org/P96284 and previous config saved to /var/cache/conftool/dbconfig/20260901-083527-marostegui.json [08:35:58] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db2903 to test-s4 T427059', diff saved to https://phabricator.wikimedia.org/P96285 and previous config saved to /var/cache/conftool/dbconfig/20260901-083557-marostegui.json [08:36:27] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db2902: test [08:37:11] !log marostegui@cumin1003 END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2902: test [08:37:24] (03CR) 10Muehlenhoff: [C:03+2] Fix hostname in homer sync [puppet] - 10https://gerrit.wikimedia.org/r/1333128 (owner: 10Muehlenhoff) [08:39:51] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [08:41:54] (03PS1) 10Marostegui: pool.py: Add support for test-s4 [cookbooks] - 10https://gerrit.wikimedia.org/r/1333132 (https://phabricator.wikimedia.org/T427059) [08:42:49] !log marostegui@cumin1003 dbctl commit (dc=all): 'Test depool db2902', diff saved to https://phabricator.wikimedia.org/P96287 and previous config saved to /var/cache/conftool/dbconfig/20260901-084249-marostegui.json [08:43:06] (03CR) 10Marostegui: "Doing it via dbctl, of course works fine:" [cookbooks] - 10https://gerrit.wikimedia.org/r/1333132 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [08:43:17] !log marostegui@cumin1003 dbctl commit (dc=all): 'Test repool db2902', diff saved to https://phabricator.wikimedia.org/P96288 and previous config saved to /var/cache/conftool/dbconfig/20260901-084317-marostegui.json [08:43:27] (03CR) 10Marostegui: "Same for repool:" [cookbooks] - 10https://gerrit.wikimedia.org/r/1333132 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [08:43:29] (03PS1) 10Blake: memcached: continue pki rollout [puppet] - 10https://gerrit.wikimedia.org/r/1333133 (https://phabricator.wikimedia.org/T353511) [08:44:16] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1333133 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [08:45:20] filippo@cumin1003 decommission (PID 1593625) is awaiting input [08:45:42] (03PS1) 10Slyngshede: Geo-maps: Update Meta mapping for september 2026 [dns] - 10https://gerrit.wikimedia.org/r/1333134 [08:46:53] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1074.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [08:47:10] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1074.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [08:47:10] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:47:12] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1074.eqiad.wmnet [08:47:23] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274331 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1074.eqiad.wmnet` - cloudvirt1074.eqiad.wmnet (... [08:47:26] (03PS2) 10Brouberol: airflow: allow gitsync to egress to the gitlab internal endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333124 (https://phabricator.wikimedia.org/T435292) [08:50:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:50:19] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12274333 (10fgiunchedi) a:05fgiunchedi→03VRiley-WMF Hosts have been temporarily decom'd and are ready for you @VRiley-WMF [08:50:35] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1003.wikimedia.org with OS bookworm [08:50:37] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12274336 (10fgiunchedi) a:05fgiunchedi→03VRiley-WMF Host has been temporarily decom'd and is ready for you @VRiley-WMF [08:50:50] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12274338 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host urldownloader... [08:52:05] (03CR) 10Brouberol: [C:03+1] Puppet 8: Replace unscoped legacy facts in module bigtop [puppet] - 10https://gerrit.wikimedia.org/r/1332512 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [08:52:09] (03CR) 10Brouberol: [V:03+1] Puppet 8: Replace unscoped legacy facts in module elasticsearch [puppet] - 10https://gerrit.wikimedia.org/r/1332513 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [08:52:13] (03CR) 10Brouberol: [C:03+1] Puppet 8: Replace unscoped legacy facts in profile analytics [puppet] - 10https://gerrit.wikimedia.org/r/1332515 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [08:53:22] (03CR) 10Blake: [C:03+2] memcached: continue pki rollout [puppet] - 10https://gerrit.wikimedia.org/r/1333133 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [08:54:53] 06SRE, 10SRE-swift-storage, 06Infrastructure-Foundations, 13Patch-For-Review: xfs_scrub_all_fail.service cannot send email - https://phabricator.wikimedia.org/T436615#12274356 (10MatthewVernon) [08:55:36] (03PS2) 10Muehlenhoff: cas: Remove idp::memcached [puppet] - 10https://gerrit.wikimedia.org/r/1208362 [08:55:48] (03CR) 10Muehlenhoff: "(blocked task is now resolved)" [puppet] - 10https://gerrit.wikimedia.org/r/1208362 (owner: 10Muehlenhoff) [08:56:21] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage [08:56:25] (03CR) 10Arnaudb: [C:03+1] global_config: register the internal gitlab endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1333120 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [08:56:27] (03CR) 10Arnaudb: [C:03+1] "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1333120 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [08:57:34] (03Abandoned) 10Muehlenhoff: cas: Remove idp::memcached [puppet] - 10https://gerrit.wikimedia.org/r/1208362 (owner: 10Muehlenhoff) [08:57:55] (03CR) 10Arnaudb: [C:03+2] gitlab: allow pods running in the dse clusters to egress to internal endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1333119 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [08:58:21] (03CR) 10Brouberol: [C:03+2] global_config: register the internal gitlab endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1333120 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [08:58:22] (03PS1) 10Marostegui: instances.yaml: Add db190[1-3] [puppet] - 10https://gerrit.wikimedia.org/r/1333137 (https://phabricator.wikimedia.org/T427059) [08:59:43] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db190[1-3] [puppet] - 10https://gerrit.wikimedia.org/r/1333137 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [09:00:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:00:16] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage [09:00:23] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Alert on packet loss to core sites from POPs - https://phabricator.wikimedia.org/T435592#12274367 (10ayounsi) Should that be abandoned in favor of RPM probes alerting ? As the RPM probes are more accurate, and generic services monitoring ar... [09:01:22] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1901 to test-s4 T427059', diff saved to https://phabricator.wikimedia.org/P96289 and previous config saved to /var/cache/conftool/dbconfig/20260901-090121-marostegui.json [09:01:25] T427059: Discussion for adding support of test-s4 in dbctl - https://phabricator.wikimedia.org/T427059 [09:01:41] 06SRE, 06Infrastructure-Foundations, 10netops, 10Observability-Metrics: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12274375 (10ayounsi) Nice work !! >>! In T435855#12253094, @cmooney wrote: > The stats for the magru HE circuits are w... [09:01:59] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1902 to test-s4 T427059', diff saved to https://phabricator.wikimedia.org/P96290 and previous config saved to /var/cache/conftool/dbconfig/20260901-090158-marostegui.json [09:02:34] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1903 to test-s4 T427059', diff saved to https://phabricator.wikimedia.org/P96291 and previous config saved to /var/cache/conftool/dbconfig/20260901-090233-marostegui.json [09:03:12] (03PS1) 10Muehlenhoff: Allow cumin1004 in SSH firewall rules as cumin master [puppet] - 10https://gerrit.wikimedia.org/r/1333138 (https://phabricator.wikimedia.org/T427897) [09:04:45] (03PS1) 10Giuseppe Lavagetto: shellbox: add gVisor support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333139 [09:06:36] (03PS1) 10Marostegui: check_private_data_report: Add db1270 [puppet] - 10https://gerrit.wikimedia.org/r/1333141 (https://phabricator.wikimedia.org/T407942) [09:06:37] (03CR) 10Jelto: "looks mostly good, two questions in line" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333038 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [09:06:39] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST events) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [09:06:41] (03CR) 10CI reject: [V:04-1] shellbox: add gVisor support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333139 (owner: 10Giuseppe Lavagetto) [09:06:52] FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [09:07:32] (03PS2) 10Marostegui: check_private_data_report: Add db1270 [puppet] - 10https://gerrit.wikimedia.org/r/1333141 (https://phabricator.wikimedia.org/T407942) [09:10:03] FIRING: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:10:12] (03CR) 10Fabfur: [C:03+2] external_cloud_vendors: set datetime from yesterday on ripe query [puppet] - 10https://gerrit.wikimedia.org/r/1332778 (owner: 10Fabfur) [09:10:27] (03PS4) 10Ayounsi: Add TransportLinksInUsageNoRedundancy alert [alerts] - 10https://gerrit.wikimedia.org/r/1321906 [09:12:36] (03CR) 10Ladsgroup: [C:03+1] Add is_thumb_generated to webrequest_sampled turnilo conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333087 (https://phabricator.wikimedia.org/T435634) (owner: 10Joal) [09:13:21] (03CR) 10Ayounsi: Add TransportLinksInUsageNoRedundancy alert (032 comments) [alerts] - 10https://gerrit.wikimedia.org/r/1321906 (owner: 10Ayounsi) [09:13:24] (03CR) 10CI reject: [V:04-1] Add TransportLinksInUsageNoRedundancy alert [alerts] - 10https://gerrit.wikimedia.org/r/1321906 (owner: 10Ayounsi) [09:18:05] (03PS1) 10JavierMonton: stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333142 (https://phabricator.wikimedia.org/T435291) [09:19:02] 06SRE, 10corto, 10Incident Tooling: Increase trusted volunteer's visibility into production incidents - https://phabricator.wikimedia.org/T426137#12274417 (10MLechvien-WMF) Thanks for surfacing that, it was not intended. We'll attempt to fix the access for acl*security users. [09:19:49] (03CR) 10Hnowlan: [C:03+2] sre/cdn: use traffic ratios in ATSBackendErrorsHigh [alerts] - 10https://gerrit.wikimedia.org/r/1326811 (https://phabricator.wikimedia.org/T400675) (owner: 10Hnowlan) [09:20:10] (03CR) 10Joal: [C:03+1] "LGTM" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333142 (https://phabricator.wikimedia.org/T435291) (owner: 10JavierMonton) [09:21:44] (03Merged) 10jenkins-bot: sre/cdn: use traffic ratios in ATSBackendErrorsHigh [alerts] - 10https://gerrit.wikimedia.org/r/1326811 (https://phabricator.wikimedia.org/T400675) (owner: 10Hnowlan) [09:22:08] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Add db1270 [puppet] - 10https://gerrit.wikimedia.org/r/1333141 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [09:22:35] 06SRE, 10SRE-swift-storage, 13Patch-For-Review: xfs_scrub_all_fail.service cannot send email (Debian #1116595) - https://phabricator.wikimedia.org/T436615#12274427 (10MatthewVernon) [09:22:42] (03CR) 10Jelto: "looks mostly good but I'd try to separate the changes into decom of phab1004 and setup of phab1006. Or is there a strict need to switch bo" [puppet] - 10https://gerrit.wikimedia.org/r/1332730 (https://phabricator.wikimedia.org/T377889) (owner: 10AOkoth) [09:24:18] (03CR) 10Hnowlan: [C:03+2] klaxon: add support for managers escalation [puppet] - 10https://gerrit.wikimedia.org/r/1332762 (owner: 10Hnowlan) [09:24:45] !log marostegui@cumin1003 dbctl commit (dc=all): 'Change db1176 and db2230's weight, test-s4 masters, to 0 to mimic the rest of production T427059', diff saved to https://phabricator.wikimedia.org/P96292 and previous config saved to /var/cache/conftool/dbconfig/20260901-092444-marostegui.json [09:24:48] T427059: Discussion for adding support of test-s4 in dbctl - https://phabricator.wikimedia.org/T427059 [09:25:38] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-d4-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:30:23] (03PS1) 10Elukey: sre.hosts.provision: add virtualization workload when virt is enabled [cookbooks] - 10https://gerrit.wikimedia.org/r/1333146 (https://phabricator.wikimedia.org/T435537) [09:31:44] (03PS1) 10Joal: Remove turnilo as it now runs in dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1333147 (https://phabricator.wikimedia.org/T416125) [09:32:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:33:03] (03CR) 10Joal: "The only code leftover I don't know how to handle is in:" [puppet] - 10https://gerrit.wikimedia.org/r/1333147 (https://phabricator.wikimedia.org/T416125) (owner: 10Joal) [09:33:58] (03CR) 10JavierMonton: [C:03+2] stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333142 (https://phabricator.wikimedia.org/T435291) (owner: 10JavierMonton) [09:36:12] (03Merged) 10jenkins-bot: stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333142 (https://phabricator.wikimedia.org/T435291) (owner: 10JavierMonton) [09:36:13] (03CR) 10Brouberol: [C:03+1] "I think it should be safe enough to remove, as we're now using an httpd-cas sidecar to auth requests to CAS." [puppet] - 10https://gerrit.wikimedia.org/r/1333147 (https://phabricator.wikimedia.org/T416125) (owner: 10Joal) [09:37:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:38:04] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [09:38:06] 06SRE, 10Pywikibot, 06Traffic, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12274488 (10Ladsgroup) It depends on what kind of load and what is causing the lag to go up. e.g. If it's a gigantic write, reducing reads on replicas won't... [09:38:38] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [09:39:43] (03PS1) 10Marostegui: mariadb: Change s8 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333148 (https://phabricator.wikimedia.org/T436078) [09:40:21] (03CR) 10Marostegui: "Made the change in dbctl too" [puppet] - 10https://gerrit.wikimedia.org/r/1333148 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [09:40:39] (03CR) 10Marostegui: [C:03+2] mariadb: Change s8 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333148 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [09:42:47] (03PS1) 10Marostegui: mariadb: Change s2 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333149 (https://phabricator.wikimedia.org/T436078) [09:44:07] (03CR) 10Marostegui: "Replaced in dbctl too" [puppet] - 10https://gerrit.wikimedia.org/r/1333149 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [09:44:10] (03CR) 10Marostegui: [C:03+2] mariadb: Change s2 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333149 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [09:44:12] (03PS5) 10Ayounsi: Add TransportLinksInUsageNoRedundancy alert [alerts] - 10https://gerrit.wikimedia.org/r/1321906 [09:46:45] (03CR) 10Brouberol: [C:03+2] airflow: allow gitsync to egress to the gitlab internal endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333124 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [09:47:15] (03CR) 10CI reject: [V:04-1] Add TransportLinksInUsageNoRedundancy alert [alerts] - 10https://gerrit.wikimedia.org/r/1321906 (owner: 10Ayounsi) [09:47:21] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations: attempts to reimage cp5022 are failing - https://phabricator.wikimedia.org/T436386#12274514 (10elukey) I was able to reimage the host, but now the check WaitRebootSince times out after 10s, so it can't verify that the host is indeed up. I checked with cumin a... [09:47:32] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [09:48:09] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [09:48:30] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [09:48:47] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [09:49:54] !log atsuko@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/echoserver: apply [09:50:03] !log atsuko@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/echoserver: apply [09:54:20] (03CR) 10Federico Ceratto: [C:03+1] "LGTM, I'll add a test later of if needed." [cookbooks] - 10https://gerrit.wikimedia.org/r/1333132 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [09:56:49] (03CR) 10Marostegui: [C:03+2] pool.py: Add support for test-s4 [cookbooks] - 10https://gerrit.wikimedia.org/r/1333132 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [09:57:05] (03CR) 10Marostegui: [C:03+2] "> LGTM, I'll add a test later of if needed." [cookbooks] - 10https://gerrit.wikimedia.org/r/1333132 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [09:58:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:58:35] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db2902: test [09:58:37] !log marostegui@cumin1003 END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db2902: test [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1000) [10:01:13] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db2902: test [10:01:40] !log marostegui@cumin1003 END (ERROR) - Cookbook sre.mysql.depool (exit_code=97) depool db2902: test [10:02:42] (03CR) 10Ladsgroup: [C:03+2] Supported animated webp thumbnailing [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1270150 (https://phabricator.wikimedia.org/T290345) (owner: 10TheDJ) [10:03:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:03:44] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db2902: test [10:03:47] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: test [10:03:58] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [10:04:19] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db2902: Pool back db2902 [10:04:22] !log marostegui@cumin1003 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2902: Pool back db2902 [10:04:26] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [10:04:33] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db2902: Pool back db2902 [10:06:07] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [10:06:43] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [10:08:13] (03Merged) 10jenkins-bot: Supported animated webp thumbnailing [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1270150 (https://phabricator.wikimedia.org/T290345) (owner: 10TheDJ) [10:09:28] (03PS1) 10Marostegui: mariadb: Add candidate masters in test-s4 [puppet] - 10https://gerrit.wikimedia.org/r/1333151 (https://phabricator.wikimedia.org/T427059) [10:10:19] (03CR) 10Marostegui: [C:03+2] mariadb: Add candidate masters in test-s4 [puppet] - 10https://gerrit.wikimedia.org/r/1333151 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [10:11:04] (03Abandoned) 10Arnaudb: kubernetes: allow pod egress to text-lb for gitlab [puppet] - 10https://gerrit.wikimedia.org/r/1332720 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [10:19:23] (03PS1) 10Marostegui: mariadb: Fix test-s4 candidate masters [puppet] - 10https://gerrit.wikimedia.org/r/1333152 (https://phabricator.wikimedia.org/T427059) [10:20:09] (03CR) 10Marostegui: [C:03+2] mariadb: Fix test-s4 candidate masters [puppet] - 10https://gerrit.wikimedia.org/r/1333152 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [10:20:29] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [10:20:41] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [10:26:14] (03PS1) 10Brouberol: blunderbuss: allow egress to the internal gitlab endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333153 (https://phabricator.wikimedia.org/T435292) [10:26:39] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST events) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [10:30:05] !log ayounsi@cumin1003 START - Cookbook sre.dns.netbox [10:32:41] (03PS1) 10Hnowlan: thumbor: use service mesh to talk to swift [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333155 [10:34:45] !log ayounsi@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp5022 - ayounsi@cumin1003" [10:34:49] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp5022 - ayounsi@cumin1003" [10:34:49] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:35:52] (03PS1) 10Marostegui: mariadb: Change s3 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333156 (https://phabricator.wikimedia.org/T436078) [10:36:09] !log ayounsi@cumin1003 START - Cookbook sre.dns.wipe-cache cp5022.eqsin.wmnet on all recursors [10:36:13] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp5022.eqsin.wmnet on all recursors [10:36:29] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5022.eqsin.wmnet with OS trixie [10:36:32] (03CR) 10Marostegui: [C:03+2] mariadb: Change s3 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333156 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [10:36:38] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations: attempts to reimage cp5022 are failing - https://phabricator.wikimedia.org/T436386#12274733 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1003 for host cp5022.eqsin.wmnet with OS trixie completed: - cp5022 (**PASS**) - R... [10:37:47] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host urldownloader1004.wikimedia.org with OS bookworm [10:38:06] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12274739 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host urldownlo... [10:39:17] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations: attempts to reimage cp5022 are failing - https://phabricator.wikimedia.org/T436386#12274741 (10elukey) Thanks to @ayounsi we fixed the main issue, namely that cp5022 was not allocated in the right rack in netbox, ending up to have the wrong auto-generated AAA... [10:39:37] (03PS1) 10Marostegui: mariadb: Change s5 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333157 (https://phabricator.wikimedia.org/T436078) [10:40:16] (03CR) 10Marostegui: [C:03+2] mariadb: Change s5 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333157 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [10:41:57] (03PS1) 10Brouberol: gitlab: allow traffic from our analytics subnet [puppet] - 10https://gerrit.wikimedia.org/r/1333158 (https://phabricator.wikimedia.org/T435292) [10:42:05] (03PS6) 10Ayounsi: Add TransportLinksInUsageNoRedundancy alert [alerts] - 10https://gerrit.wikimedia.org/r/1321906 [10:42:19] (03CR) 10Arnaudb: [C:03+2] gitlab: allow traffic from our analytics subnet [puppet] - 10https://gerrit.wikimedia.org/r/1333158 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [10:44:10] (03PS1) 10Marostegui: mariadb: Change x1 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333159 (https://phabricator.wikimedia.org/T436078) [10:44:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:44:59] (03CR) 10Marostegui: [C:03+2] mariadb: Change x1 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333159 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [10:45:32] !log installing Python 3.11 security updates [10:45:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:49:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:49:37] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2902: Pool back db2902 [10:53:00] (03PS1) 10Marostegui: mariadb: Change s4 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333161 (https://phabricator.wikimedia.org/T436078) [10:53:00] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1004.wikimedia.org with reason: host reimage [10:56:49] (03CR) 10Marostegui: [C:03+2] mariadb: Change s4 candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333161 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [10:59:49] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1004.wikimedia.org with reason: host reimage [11:11:18] (03PS2) 10Slyngshede: mw-web: upsize for single-DC serving [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) [11:12:45] (03CR) 10Slyngshede: "Done" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [11:12:51] !log jmm@cumin2003 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad [11:12:54] !log installing openjdk-21 security updates [11:12:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:14:06] (03PS1) 10Clément Goubert: zotero: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333167 (https://phabricator.wikimedia.org/T429175) [11:14:08] (03PS1) 10Clément Goubert: citoid: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333168 (https://phabricator.wikimedia.org/T429175) [11:14:10] (03PS1) 10Clément Goubert: toolhub: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333169 (https://phabricator.wikimedia.org/T429175) [11:14:13] (03PS1) 10Clément Goubert: push-notifications: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333170 (https://phabricator.wikimedia.org/T429175) [11:14:15] (03PS1) 10Clément Goubert: wmf-navigator: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333171 (https://phabricator.wikimedia.org/T429175) [11:14:17] (03PS1) 10Clément Goubert: cxserver: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333172 (https://phabricator.wikimedia.org/T429175) [11:14:20] (03PS1) 10Clément Goubert: airflow: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333173 (https://phabricator.wikimedia.org/T429175) [11:14:22] (03PS1) 10Clément Goubert: wdqs_common: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333174 (https://phabricator.wikimedia.org/T429175) [11:15:31] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1004.wikimedia.org with OS bookworm [11:15:50] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12274901 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host urldownloader... [11:26:16] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host urldownloader2003.wikimedia.org with OS bookworm [11:26:21] (03CR) 10Clément Goubert: mediawiki-vhost: Redirect /api/ to /w/rest.php (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1329607 (https://phabricator.wikimedia.org/T433547) (owner: 10Clément Goubert) [11:26:30] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12274953 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host urldownlo... [11:27:25] (03PS5) 10Clément Goubert: mediawiki-vhost: Redirect /api/ to /w/rest.php [puppet] - 10https://gerrit.wikimedia.org/r/1329607 (https://phabricator.wikimedia.org/T433547) [11:28:08] (03PS3) 10Clément Goubert: mediawiki: Redirect /api/ to /w/rest.php [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330505 (https://phabricator.wikimedia.org/T433547) [11:28:09] (03CR) 10Clément Goubert: mediawiki-vhost: Redirect /api/ to /w/rest.php (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1329607 (https://phabricator.wikimedia.org/T433547) (owner: 10Clément Goubert) [11:30:40] (03CR) 10Majavah: [C:03+1] openstack: stop using icinga for pdns auth checks [puppet] - 10https://gerrit.wikimedia.org/r/1332750 (https://phabricator.wikimedia.org/T328502) (owner: 10Filippo Giunchedi) [11:30:54] (03CR) 10Majavah: [C:03+1] openldap: stop using icinga in clouddev [puppet] - 10https://gerrit.wikimedia.org/r/1332751 (https://phabricator.wikimedia.org/T328502) (owner: 10Filippo Giunchedi) [11:31:44] (03PS3) 10Slyngshede: hieradata: update deployment_server to deploy2003 [puppet] - 10https://gerrit.wikimedia.org/r/1319826 (https://phabricator.wikimedia.org/T436630) [11:32:55] !log jmm@cumin2003 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-test-eqiad [11:32:57] (03CR) 10Filippo Giunchedi: [C:03+2] openldap: stop using icinga in clouddev [puppet] - 10https://gerrit.wikimedia.org/r/1332751 (https://phabricator.wikimedia.org/T328502) (owner: 10Filippo Giunchedi) [11:33:02] (03CR) 10Filippo Giunchedi: [C:03+2] openstack: stop using icinga for pdns auth checks [puppet] - 10https://gerrit.wikimedia.org/r/1332750 (https://phabricator.wikimedia.org/T328502) (owner: 10Filippo Giunchedi) [11:41:37] (03PS2) 10Slyngshede: wmnet: update deployment CNAME record to deploy2003 [dns] - 10https://gerrit.wikimedia.org/r/1319825 (https://phabricator.wikimedia.org/T436630) [11:43:31] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2003.wikimedia.org with reason: host reimage [11:49:27] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2003.wikimedia.org with reason: host reimage [11:51:47] 10ops-eqiad, 06DC-Ops, 10decommission-hardware: decommission phab1004 - https://phabricator.wikimedia.org/T436632 (10Arnoldokoth) 03NEW [11:52:07] (03CR) 10KartikMistry: [C:03+1] cxserver: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333172 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [11:52:36] 10ops-codfw, 06DC-Ops, 10decommission-hardware: decommission phab2002 - https://phabricator.wikimedia.org/T436632#12275101 (10Arnoldokoth) [11:53:33] (03CR) 10Clément Goubert: [C:03+2] cxserver: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333172 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [11:54:01] 10ops-codfw, 06DC-Ops, 10decommission-hardware: decommission phab2002 - https://phabricator.wikimedia.org/T436632#12275117 (10Arnoldokoth) I ran the decom cookbook quite some time back though I completely forgot about creating this tracking task. [11:54:06] (03PS1) 10Marostegui: mariadb: Change s5 codfw candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333179 (https://phabricator.wikimedia.org/T436078) [11:55:10] (03CR) 10Marostegui: [C:03+2] mariadb: Change s5 codfw candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1333179 (https://phabricator.wikimedia.org/T436078) (owner: 10Marostegui) [11:57:26] claime: should I deploy https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1333172 or will it be done with other similar changes? [11:57:41] kart_: no you can go ahead, I was about to do it, but feel free [11:57:55] (Once it's passed CI and merged) [11:58:03] claime: is it fine to do it tomorrow morning as well? [11:58:14] kart_: yeah nothing urgent [11:58:21] cool. Thanks! [12:00:04] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1200) [12:00:30] 10ops-eqiad, 06DC-Ops, 10decommission-hardware: decommission phab1004 - https://phabricator.wikimedia.org/T436635 (10Arnoldokoth) 03NEW [12:00:40] (03PS1) 10Muehlenhoff: proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333180 [12:00:56] (03PS3) 10AOkoth: phabricator: decom phab1004 [puppet] - 10https://gerrit.wikimedia.org/r/1332730 (https://phabricator.wikimedia.org/T436635) [12:05:39] (03CR) 10Mvolz: [C:03+1] "I can +2 and deploy tomorrow during the window unless you're planning to yourself?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333168 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [12:05:47] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2003.wikimedia.org with OS bookworm [12:06:01] (03CR) 10Clément Goubert: "SGTM, thanks." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333168 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [12:06:02] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275173 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host urldownloader... [12:06:11] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply [12:06:15] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply [12:06:21] !log atsuko@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply [12:06:29] !log atsuko@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply [12:07:07] (03CR) 10Gmodena: [C:03+1] wdqs_common: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333174 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [12:07:28] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [12:08:17] 06SRE, 06Traffic, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12275178 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF T436001 is now public with a report. Let's call... [12:08:27] 06SRE, 06Traffic, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12275184 (10Jdforrester-WMF) a:05Jdforrester-WMF→03None [12:09:01] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [12:09:31] !log atsuko@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [12:10:40] (03CR) 10Muehlenhoff: [C:03+2] proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333180 (owner: 10Muehlenhoff) [12:13:11] (03Merged) 10jenkins-bot: proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333180 (owner: 10Muehlenhoff) [12:14:11] !log jmm@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [12:14:59] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host urldownloader2004.wikimedia.org with OS bookworm [12:14:59] !log jmm@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [12:15:08] (03PS1) 10Ayounsi: Add netflow1004 to kafka ACL [puppet] - 10https://gerrit.wikimedia.org/r/1333181 (https://phabricator.wikimedia.org/T436517) [12:15:20] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275219 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host urldownlo... [12:16:51] !log ayounsi@cumin1003 START - Cookbook sre.ganeti.makevm for new host netflow1004.eqiad.wmnet [12:16:52] !log jmm@deploy1003 helmfile [codfw] START helmfile.d/services/proton: apply [12:16:53] !log ayounsi@cumin1003 START - Cookbook sre.dns.netbox [12:17:17] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1333181 (https://phabricator.wikimedia.org/T436517) (owner: 10Ayounsi) [12:17:53] (03PS1) 10Dreamy Jazz: EventStreamConfig: Fix User-Agent stream config for some schemas [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1333182 (https://phabricator.wikimedia.org/T432848) [12:18:01] jouncebot: nowandnext [12:18:01] For the next 0 hour(s) and 41 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1200) [12:18:01] In 0 hour(s) and 41 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1300) [12:18:41] (03CR) 10Ladsgroup: "Look good to me but Clem might have ideas. On thing to think about. Maybe we can just enable it on staging and then enable it for prod aft" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333155 (owner: 10Hnowlan) [12:19:38] (03PS1) 10Filippo Giunchedi: install_server: add fallback partition recipe to reuse-parts [puppet] - 10https://gerrit.wikimedia.org/r/1333183 (https://phabricator.wikimedia.org/T436058) [12:19:41] (03PS1) 10Filippo Giunchedi: install_server: refactor reuse-parts.sh [puppet] - 10https://gerrit.wikimedia.org/r/1333184 (https://phabricator.wikimedia.org/T436058) [12:19:43] (03PS1) 10Filippo Giunchedi: install_server: add reuse-parts support for standard EFI recipes [puppet] - 10https://gerrit.wikimedia.org/r/1333185 (https://phabricator.wikimedia.org/T436058) [12:19:53] Going to use scap [12:20:01] !log atsuko@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [12:20:42] !log jmm@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: apply [12:21:07] !log ayounsi@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM netflow1004.eqiad.wmnet - ayounsi@cumin1003" [12:21:09] 10SRE-tools, 06Infrastructure-Foundations, 10Phabricator, 13Patch-For-Review: offboard-user: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T420324#12275267 (10Aklapper) @MoritzMuehlenhoff Would you know who could review/test/merg... [12:21:09] (03CR) 10CI reject: [V:04-1] install_server: add reuse-parts support for standard EFI recipes [puppet] - 10https://gerrit.wikimedia.org/r/1333185 (https://phabricator.wikimedia.org/T436058) (owner: 10Filippo Giunchedi) [12:21:11] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM netflow1004.eqiad.wmnet - ayounsi@cumin1003" [12:21:11] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:21:11] !log ayounsi@cumin1003 START - Cookbook sre.dns.wipe-cache netflow1004.eqiad.wmnet on all recursors [12:21:14] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netflow1004.eqiad.wmnet on all recursors [12:21:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1333182 (https://phabricator.wikimedia.org/T432848) (owner: 10Dreamy Jazz) [12:21:46] (03PS1) 10Marostegui: orchestrator.conf: Do not auto-recover test-s4 [puppet] - 10https://gerrit.wikimedia.org/r/1333186 [12:21:48] !log ayounsi@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM netflow1004.eqiad.wmnet - ayounsi@cumin1003" [12:21:52] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM netflow1004.eqiad.wmnet - ayounsi@cumin1003" [12:22:06] (03PS2) 10Dreamy Jazz: EventStreamConfig: Register the abuse_review_interaction stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329360 (https://phabricator.wikimedia.org/T435517) [12:22:29] !log ayounsi@cumin1003 START - Cookbook sre.hosts.reimage for host netflow1004.eqiad.wmnet with OS trixie [12:24:25] (03Merged) 10jenkins-bot: EventStreamConfig: Fix User-Agent stream config for some schemas [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1333182 (https://phabricator.wikimedia.org/T432848) (owner: 10Dreamy Jazz) [12:24:26] (03PS3) 10Jelto: scaffold: Fix helm release NOTES.txt text [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332556 (https://phabricator.wikimedia.org/T433589) [12:25:31] !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1333182|EventStreamConfig: Fix User-Agent stream config for some schemas (T432848)]] [12:25:34] T432848: EventLogging: Record confidence score results - https://phabricator.wikimedia.org/T432848 [12:25:40] (03PS2) 10Filippo Giunchedi: install_server: add reuse-parts support for standard EFI recipes [puppet] - 10https://gerrit.wikimedia.org/r/1333185 (https://phabricator.wikimedia.org/T436058) [12:25:47] (03CR) 10Ayounsi: [C:03+2] Add netflow1004 to kafka ACL [puppet] - 10https://gerrit.wikimedia.org/r/1333181 (https://phabricator.wikimedia.org/T436517) (owner: 10Ayounsi) [12:26:45] (03CR) 10Jelto: scaffold: Fix helm release NOTES.txt text (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332556 (https://phabricator.wikimedia.org/T433589) (owner: 10Jelto) [12:26:46] (03CR) 10Jgiannelos: [C:03+1] push-notifications: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333170 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [12:27:25] (03CR) 10CWilliams: orchestrator.conf: Do not auto-recover test-s4 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1333186 (owner: 10Marostegui) [12:28:35] (03CR) 10Marostegui: orchestrator.conf: Do not auto-recover test-s4 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1333186 (owner: 10Marostegui) [12:28:46] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply [12:28:50] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply [12:28:57] (03PS2) 10Brouberol: blunderbuss: allow egress to the internal gitlab endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333153 (https://phabricator.wikimedia.org/T435292) [12:29:49] !log dreamyjazz@deploy1003 dreamyjazz: Backport for [[gerrit:1333182|EventStreamConfig: Fix User-Agent stream config for some schemas (T432848)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:30:28] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-analytics-test: apply [12:30:32] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-analytics-test: apply [12:31:05] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-platform-eng: apply [12:31:09] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-platform-eng: apply [12:31:15] !log jmm@deploy1003 helmfile [eqiad] START helmfile.d/services/proton: apply [12:32:16] (03CR) 10CWilliams: orchestrator.conf: Do not auto-recover test-s4 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1333186 (owner: 10Marostegui) [12:32:26] Still testing... [12:32:29] !log jmm@deploy1003 helmfile [eqiad] DONE helmfile.d/services/proton: apply [12:32:33] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply [12:32:34] !log ayounsi@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on netflow1004.eqiad.wmnet with reason: host reimage [12:32:36] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply [12:32:55] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply [12:32:59] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply [12:33:20] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2004.wikimedia.org with reason: host reimage [12:34:12] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [12:34:45] !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment [12:35:09] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [12:35:40] (03CR) 10JMeybohm: [C:03+2] k8s: Disable user namespaces support [puppet] - 10https://gerrit.wikimedia.org/r/1332695 (https://phabricator.wikimedia.org/T427069) (owner: 10JMeybohm) [12:35:42] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [12:36:34] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [12:36:59] (03PS2) 10Marostegui: orchestrator.conf: Do not auto-recover test-s4 [puppet] - 10https://gerrit.wikimedia.org/r/1333186 [12:37:33] (03CR) 10Marostegui: orchestrator.conf: Do not auto-recover test-s4 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1333186 (owner: 10Marostegui) [12:38:03] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow1004.eqiad.wmnet with reason: host reimage [12:38:34] (03PS1) 10Blake: Revert "memcached: Rebuild configuration before reloading certificates." [puppet] - 10https://gerrit.wikimedia.org/r/1333190 [12:41:08] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2004.wikimedia.org with reason: host reimage [12:41:15] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1333186 (owner: 10Marostegui) [12:41:56] !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1333182|EventStreamConfig: Fix User-Agent stream config for some schemas (T432848)]] (duration: 16m 25s) [12:41:59] T432848: EventLogging: Record confidence score results - https://phabricator.wikimedia.org/T432848 [12:42:40] (03CR) 10Marostegui: [C:03+2] orchestrator.conf: Do not auto-recover test-s4 [puppet] - 10https://gerrit.wikimedia.org/r/1333186 (owner: 10Marostegui) [12:43:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:45:46] (03PS1) 10Blake: memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1333192 [12:45:49] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1333192 (owner: 10Blake) [12:46:08] (03CR) 10CI reject: [V:04-1] memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1333192 (owner: 10Blake) [12:46:16] (03CR) 10Atsuko: [C:03+1] blunderbuss: allow egress to the internal gitlab endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333153 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [12:46:51] (03PS2) 10Blake: memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1333192 (https://phabricator.wikimedia.org/T353511) [12:47:19] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1333192 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [12:47:39] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [12:48:03] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [12:48:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:49:33] (03CR) 10AOkoth: "It's not a hard requirement really. I'll split them." [puppet] - 10https://gerrit.wikimedia.org/r/1332730 (https://phabricator.wikimedia.org/T436635) (owner: 10AOkoth) [12:49:51] (03CR) 10Blake: [C:03+2] memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1333192 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [12:51:56] (03PS1) 10Muehlenhoff: Temporarily remove puppetserver2004/1003 for reboots [dns] - 10https://gerrit.wikimedia.org/r/1333194 [12:52:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.72% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:52:22] (03CR) 10Elukey: [C:03+1] Allow cumin1004 in SSH firewall rules as cumin master [puppet] - 10https://gerrit.wikimedia.org/r/1333138 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [12:52:44] (03CR) 10Muehlenhoff: [C:03+2] Allow cumin1004 in SSH firewall rules as cumin master [puppet] - 10https://gerrit.wikimedia.org/r/1333138 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [12:52:44] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host netflow1004.eqiad.wmnet with OS trixie [12:52:44] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host netflow1004.eqiad.wmnet [12:54:16] (03PS4) 10Jelto: etherpad: add helm chart for etherpad service [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327571 (https://phabricator.wikimedia.org/T435509) [12:56:14] (03CR) 10Jelto: etherpad: add helm chart for etherpad service (034 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327571 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [12:56:32] (03CR) 10CI reject: [V:04-1] etherpad: add helm chart for etherpad service [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327571 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [12:57:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.72% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:57:44] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2004.wikimedia.org with OS bookworm [12:57:58] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275445 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host urldownloader... [12:59:17] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [12:59:31] (03PS5) 10Jelto: etherpad: add helm chart for etherpad service [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327571 (https://phabricator.wikimedia.org/T435509) [12:59:46] (03PS2) 10JMeybohm: validating-admission-policies: Update for k8s 1.34 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333038 (https://phabricator.wikimedia.org/T436380) [13:00:04] Lucas_WMDE, urbanecm, and TheresNoTime: That opportune time for a UTC afternoon backport window deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1300). [13:00:05] aude: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:05] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [13:00:14] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/datasets-config-next: apply [13:00:23] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/datasets-config-next: apply [13:00:37] I can deploy my patch [13:00:42] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply [13:01:01] (03CR) 10Aude: Enable ReadingLists for logged-in users on phase 1 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332717 (https://phabricator.wikimedia.org/T434922) (owner: 10Aude) [13:01:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by aude@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332717 (https://phabricator.wikimedia.org/T434922) (owner: 10Aude) [13:01:56] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply [13:02:21] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [13:02:23] (03Merged) 10jenkins-bot: Enable ReadingLists for logged-in users on phase 1 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332717 (https://phabricator.wikimedia.org/T434922) (owner: 10Aude) [13:02:25] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply [13:02:45] !log aude@deploy1003 Started scap sync-world: Backport for [[gerrit:1332717|Enable ReadingLists for logged-in users on phase 1 wikis (T434922)]] [13:02:48] T434922: Enable ReadingLists for all logged-in users [Sept 1] on phase 1 wikis (arwiki, frwiki, and idwiki) - https://phabricator.wikimedia.org/T434922 [13:02:48] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich-next: apply [13:02:52] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich-next: apply [13:05:04] (03CR) 10Clément Goubert: [C:03+1] Revert "memcached: Rebuild configuration before reloading certificates." [puppet] - 10https://gerrit.wikimedia.org/r/1333190 (owner: 10Blake) [13:05:19] (03CR) 10Blake: [C:03+2] Revert "memcached: Rebuild configuration before reloading certificates." [puppet] - 10https://gerrit.wikimedia.org/r/1333190 (owner: 10Blake) [13:05:47] (03PS2) 10Joal: Remove turnilo as it now runs in dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1333147 (https://phabricator.wikimedia.org/T416125) [13:05:58] (03CR) 10JMeybohm: "I did a second pass on the very repetitive CEL expression and was able to simplify it quite a bit by using the optional field expression (" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333038 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [13:06:24] (03CR) 10FNegri: [C:03+2] toolhub: Bump container and enable Toolhub Evolved banner [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332528 (https://phabricator.wikimedia.org/T434556) (owner: 10Raymond Ndibe) [13:06:52] FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [13:07:28] !log aude@deploy1003 aude: Backport for [[gerrit:1332717|Enable ReadingLists for logged-in users on phase 1 wikis (T434922)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:08:21] (03PS1) 10Ladsgroup: thumbor: Supported animated webp thumbnailing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333198 (https://phabricator.wikimedia.org/T290345) [13:08:44] (03Merged) 10jenkins-bot: toolhub: Bump container and enable Toolhub Evolved banner [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332528 (https://phabricator.wikimedia.org/T434556) (owner: 10Raymond Ndibe) [13:09:04] (03PS1) 10Blake: memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1333199 (https://phabricator.wikimedia.org/T353511) [13:09:07] (03CR) 10Blake: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1333199 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:10:04] FIRING: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:10:39] !log fnegri@deploy1003 helmfile [staging] START helmfile.d/services/toolhub: apply [13:10:51] (03CR) 10Brouberol: [C:03+2] Remove turnilo as it now runs in dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1333147 (https://phabricator.wikimedia.org/T416125) (owner: 10Joal) [13:11:02] !log aude@deploy1003 aude: Continuing with deployment [13:11:22] !log fnegri@deploy1003 helmfile [staging] DONE helmfile.d/services/toolhub: apply [13:12:00] (03CR) 10Blake: [C:03+2] memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1333199 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [13:12:46] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-analytics-product: apply [13:12:50] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-analytics-product: apply [13:12:55] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-dev: apply [13:12:59] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-dev: apply [13:13:04] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply [13:13:08] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply [13:13:17] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-fr-tech: apply [13:13:20] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-fr-tech: apply [13:13:28] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply [13:13:32] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply [13:14:09] !log fnegri@deploy1003 helmfile [codfw] START helmfile.d/services/toolhub: apply [13:14:28] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-ml: apply [13:14:32] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-ml: apply [13:14:37] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-research: apply [13:14:41] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-research: apply [13:15:04] !log fnegri@deploy1003 helmfile [codfw] DONE helmfile.d/services/toolhub: apply [13:15:10] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:15:17] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-search: apply [13:15:21] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-search: apply [13:15:38] !log bump urldownloader[12]00[34] to 8G RAM T429175 [13:15:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:15:41] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [13:16:06] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-sre: apply [13:16:10] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-sre: apply [13:16:20] !log jmm@cumin2003 START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1003.wikimedia.org [13:16:53] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275546 (10ops-monitoring-bot) VM urldownloader1003.wikimedia.org rebooted by jmm@cumin2003 with reason: expand RAM [13:17:00] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-wikidata: apply [13:17:04] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-wikidata: apply [13:17:23] (03PS1) 10Kosta Harlan: Special:AbuseReview: Show changes as core's inline diff [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333200 (https://phabricator.wikimedia.org/T436490) [13:17:35] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-wmde: apply [13:17:39] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-wmde: apply [13:17:45] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 01 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333200 (https://phabricator.wikimedia.org/T436490) (owner: 10Kosta Harlan) [13:18:26] aude: are you done deploying? [13:19:13] there are some errors, probably unrelated but need to investigate [13:19:27] !log fnegri@deploy1003 helmfile [eqiad] START helmfile.d/services/toolhub: apply [13:19:54] * atsukoito is very sorry for the noise, we have a lot of stuff to deploy in k8s [13:20:06] think it is related to database dumps or something [13:20:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.11% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:20:18] !log fnegri@deploy1003 helmfile [eqiad] DONE helmfile.d/services/toolhub: apply [13:20:24] (03PS1) 10Muehlenhoff: Add urldownloader[12]00[34] to conf-tool [puppet] - 10https://gerrit.wikimedia.org/r/1333201 (https://phabricator.wikimedia.org/T429175) [13:20:29] just want to be very sure when i ignore the errors and proceed [13:20:32] atsukoito: o/ the noise is more when stuff explode, regular work is not spam :) [13:20:50] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1003.wikimedia.org [13:22:35] everything i am deploying looks good and the errors appear unrelated so i am proceeding [13:22:50] aude: yeah seems unrelated [13:23:09] !log aude@deploy1003 Finished scap sync-world: Backport for [[gerrit:1332717|Enable ReadingLists for logged-in users on phase 1 wikis (T434922)]] (duration: 20m 24s) [13:23:12] T434922: Enable ReadingLists for all logged-in users [Sept 1] on phase 1 wikis (arwiki, frwiki, and idwiki) - https://phabricator.wikimedia.org/T434922 [13:23:59] i am done [13:24:08] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply [13:24:30] 06SRE, 06Infrastructure-Foundations, 10netops: Missing series for BGP session_state from eqiad CRs since upgrade to 23.4R2-S8.7 - https://phabricator.wikimedia.org/T435909#12275570 (10ayounsi) Adding a 3rd netflow host in eqiad shuffled the targets around, it caused the data to be collected for a short while... [13:24:45] !log jmm@cumin2003 START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1004.wikimedia.org [13:24:49] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply [13:25:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333200 (https://phabricator.wikimedia.org/T436490) (owner: 10Kosta Harlan) [13:25:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.11% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:25:19] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275571 (10ops-monitoring-bot) VM urldownloader1004.wikimedia.org rebooted by jmm@cumin2003 with reason: expand RAM [13:25:38] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-d4-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:25:50] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:26:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:26:17] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:26:24] (03Merged) 10jenkins-bot: Special:AbuseReview: Show changes as core's inline diff [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333200 (https://phabricator.wikimedia.org/T436490) (owner: 10Kosta Harlan) [13:26:50] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1333200|Special:AbuseReview: Show changes as core's inline diff (T436490)]] [13:26:53] T436490: AbuseReview: Use core's diff libraries with inline format - https://phabricator.wikimedia.org/T436490 [13:28:18] (03PS1) 10Ayounsi: netflow: fix fastnetmon conf owner for trixie [puppet] - 10https://gerrit.wikimedia.org/r/1333202 (https://phabricator.wikimedia.org/T424478) [13:28:30] (03PS2) 10Giuseppe Lavagetto: shellbox: add gVisor support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333139 (https://phabricator.wikimedia.org/T436649) [13:29:15] (03CR) 10Ayounsi: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1333202 (https://phabricator.wikimedia.org/T424478) (owner: 10Ayounsi) [13:29:18] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1004.wikimedia.org [13:29:33] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [13:30:08] (03CR) 10Muehlenhoff: "Looks good, nit inline" [puppet] - 10https://gerrit.wikimedia.org/r/1333202 (https://phabricator.wikimedia.org/T424478) (owner: 10Ayounsi) [13:30:12] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [13:30:29] 06SRE, 06Infrastructure-Foundations: Re-IP hosts running Cassandra to per-rack subnets in codfw row A and B. - https://phabricator.wikimedia.org/T354871#12275609 (10Eevans) >>! In T354871#12270684, @ayounsi wrote: >> Too many IPs configure on the primary interface, manual migration required. > > Ah right, of... [13:30:34] (03CR) 10Brouberol: [C:03+1] Add is_thumb_generated to webrequest_sampled turnilo conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333087 (https://phabricator.wikimedia.org/T435634) (owner: 10Joal) [13:30:36] (03CR) 10Brouberol: [C:03+2] Add is_thumb_generated to webrequest_sampled turnilo conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333087 (https://phabricator.wikimedia.org/T435634) (owner: 10Joal) [13:31:47] (03CR) 10Tiziano Fogli: [C:03+1] Remove various hardcoded statsd.eqiad references [puppet] - 10https://gerrit.wikimedia.org/r/1332698 (https://phabricator.wikimedia.org/T435340) (owner: 10Hnowlan) [13:32:04] (03CR) 10Ayounsi: netflow: fix fastnetmon conf owner for trixie (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1333202 (https://phabricator.wikimedia.org/T424478) (owner: 10Ayounsi) [13:32:54] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [13:33:24] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [13:33:53] (03CR) 10Arnaudb: [C:03+1] "thanks for the fixes!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327571 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [13:34:10] (03PS2) 10Ayounsi: netflow: fix fastnetmon conf owner for trixie [puppet] - 10https://gerrit.wikimedia.org/r/1333202 (https://phabricator.wikimedia.org/T424478) [13:34:15] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1332523 (https://phabricator.wikimedia.org/T435225) [13:34:18] (03CR) 10Ayounsi: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1333202 (https://phabricator.wikimedia.org/T424478) (owner: 10Ayounsi) [13:34:19] (03CR) 10Tiziano Fogli: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332523 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [13:34:22] !log jmm@cumin2003 START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2003.wikimedia.org [13:34:56] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275628 (10ops-monitoring-bot) VM urldownloader2003.wikimedia.org rebooted by jmm@cumin2003 with reason: expand RAM [13:35:07] (03PS1) 10Gkyziridis: ml-services: Initial deployment of Qwen3.8-27B-FP8 model on experimental ns. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333204 (https://phabricator.wikimedia.org/T436644) [13:36:12] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1333202 (https://phabricator.wikimedia.org/T424478) (owner: 10Ayounsi) [13:36:47] (03CR) 10Tiziano Fogli: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332523 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [13:38:03] (03PS3) 10Tiziano Fogli: Puppet 8: Replace unscoped legacy facts in module prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1332523 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [13:38:08] (03CR) 10Tiziano Fogli: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332523 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [13:38:15] (03CR) 10Muehlenhoff: [C:03+2] Temporarily remove puppetserver2004/1003 for reboots [dns] - 10https://gerrit.wikimedia.org/r/1333194 (owner: 10Muehlenhoff) [13:38:45] !log jmm@dns1004 START - running authdns-update [13:38:50] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2003.wikimedia.org [13:39:04] !log jmm@cumin2003 START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2004.wikimedia.org [13:39:27] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275636 (10ops-monitoring-bot) VM urldownloader2004.wikimedia.org rebooted by jmm@cumin2003 with reason: expand RAM [13:39:31] (03CR) 10Ayounsi: [C:03+2] netflow: fix fastnetmon conf owner for trixie [puppet] - 10https://gerrit.wikimedia.org/r/1333202 (https://phabricator.wikimedia.org/T424478) (owner: 10Ayounsi) [13:40:56] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:41:08] !log jmm@dns1004 END - running authdns-update [13:42:01] !log joal@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply [13:43:03] PROBLEM - Host doh7003 is DOWN: PING CRITICAL - Packet loss = 100% [13:43:09] RECOVERY - Host doh7003 is UP: PING OK - Packet loss = 0%, RTA = 112.87 ms [13:43:23] FIRING: GnmiInterfaceCountersDrop: ... [13:43:23] lsw1-d1-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-d1-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [13:43:23] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2004.wikimedia.org [13:44:10] FIRING: BFDdown: BFD session down between cr1-magru and fe80::b6f9:5d02:e530:e143 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:44:49] RESOLVED: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:46:41] (03PS1) 10Ottomata: Revert "kafka: remove Kafka mirror maker configs" [puppet] - 10https://gerrit.wikimedia.org/r/1333206 [13:46:47] (03PS4) 10Tiziano Fogli: Puppet 8: Replace unscoped legacy facts in module prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1332523 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [13:46:54] (03CR) 10Tiziano Fogli: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332523 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [13:48:16] !log kharlan@deploy1003 kharlan: Backport for [[gerrit:1333200|Special:AbuseReview: Show changes as core's inline diff (T436490)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:48:20] T436490: AbuseReview: Use core's diff libraries with inline format - https://phabricator.wikimedia.org/T436490 [13:49:03] (03CR) 10CI reject: [V:04-1] Revert "kafka: remove Kafka mirror maker configs" [puppet] - 10https://gerrit.wikimedia.org/r/1333206 (owner: 10Ottomata) [13:49:10] RESOLVED: BFDdown: BFD session down between cr1-magru and fe80::b6f9:5d02:e530:e143 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:49:21] (03CR) 10Jelto: [C:03+1] "lgtm now" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333038 (https://phabricator.wikimedia.org/T436380) (owner: 10JMeybohm) [13:49:40] !log kharlan@deploy1003 kharlan: Continuing with deployment [13:50:19] !log joal@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply [13:50:32] (03PS1) 10Brouberol: Define the +_x4-analytics._tcp SRV record [dns] - 10https://gerrit.wikimedia.org/r/1333208 [13:51:45] (03CR) 10Ladsgroup: "We need to add x3 later too" [dns] - 10https://gerrit.wikimedia.org/r/1333208 (owner: 10Brouberol) [13:52:17] (03CR) 10Ladsgroup: [C:03+2] Define the +_x4-analytics._tcp SRV record [dns] - 10https://gerrit.wikimedia.org/r/1333208 (owner: 10Brouberol) [13:53:32] (03CR) 10Jelto: [C:03+1] "lgtm now, thank you!" [puppet] - 10https://gerrit.wikimedia.org/r/1332730 (https://phabricator.wikimedia.org/T436635) (owner: 10AOkoth) [13:53:42] !log ladsgroup@dns1004 START - running authdns-update [13:54:08] (03CR) 10Tiziano Fogli: [C:03+1] Puppet 8: Replace unscoped legacy facts in module prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1332523 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [13:55:31] (03PS3) 10Giuseppe Lavagetto: shellbox: add gVisor support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333139 (https://phabricator.wikimedia.org/T436649) [13:55:31] (03PS1) 10Giuseppe Lavagetto: validating-admission-policies: add gVisor enforcement policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333210 (https://phabricator.wikimedia.org/T436655) [13:55:43] !log joal@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply [13:56:00] !log ladsgroup@dns1004 END - running authdns-update [13:56:19] !log joal@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply [13:56:26] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply [13:56:30] !log ladsgroup@dns1004 START - running authdns-update [13:57:05] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply [13:58:21] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply [13:58:32] (03PS2) 10Ottomata: Partially revert "kafka: remove Kafka mirror maker configs" [puppet] - 10https://gerrit.wikimedia.org/r/1333206 [13:58:55] !log ladsgroup@dns1004 END - running authdns-update [13:58:55] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply [13:59:07] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply [13:59:21] (03PS3) 10Ottomata: Partially revert "kafka: remove Kafka mirror maker configs" [puppet] - 10https://gerrit.wikimedia.org/r/1333206 (https://phabricator.wikimedia.org/T432089) [13:59:41] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply [13:59:57] !log joal@deploy1003 Started deploy [analytics/refinery@19f4b59] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@19f4b595] [14:00:01] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply [14:00:03] FIRING: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:00:04] Deploy window Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1400) [14:00:28] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply [14:00:30] (03PS1) 10JMeybohm: validating-admission-policies: Cosmetic changes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333211 (https://phabricator.wikimedia.org/T436380) [14:00:30] (03CR) 10Elukey: "What if we run a docker image in deployment-prep, rather than keeping old and unused puppet code? It will be closer to production, and sur" [puppet] - 10https://gerrit.wikimedia.org/r/1333206 (https://phabricator.wikimedia.org/T432089) (owner: 10Ottomata) [14:00:43] !log joal@deploy1003 Finished deploy [analytics/refinery@19f4b59] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@19f4b595] (duration: 00m 45s) [14:00:59] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 100% [14:01:18] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply [14:01:39] (03CR) 10CI reject: [V:04-1] Partially revert "kafka: remove Kafka mirror maker configs" [puppet] - 10https://gerrit.wikimedia.org/r/1333206 (https://phabricator.wikimedia.org/T432089) (owner: 10Ottomata) [14:01:57] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply [14:02:03] (03CR) 10Ottomata: "OOOOOOOO good idea! Why not! I think...we can do this with just horizon UI...looking into it." [puppet] - 10https://gerrit.wikimedia.org/r/1333206 (https://phabricator.wikimedia.org/T432089) (owner: 10Ottomata) [14:02:21] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [14:02:35] !log jmm@cumin2003 START - Cookbook sre.puppet.disable-merges [14:02:36] !log jmm@cumin2003 END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) [14:02:48] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [14:03:07] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [14:03:14] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet [14:03:19] !log Removed openjdk-17 packages from contint1002/contint2002 following relocation of CI Jenkins to contint1003/contint2003 # T418521 [14:03:21] (03CR) 10Tiziano Fogli: "recheck" [alerts] - 10https://gerrit.wikimedia.org/r/1329662 (owner: 10CDanis) [14:03:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:03:22] T418521: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521 [14:03:36] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply [14:04:25] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply [14:04:28] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1333200|Special:AbuseReview: Show changes as core's inline diff (T436490)]] (duration: 37m 37s) [14:04:31] T436490: AbuseReview: Use core's diff libraries with inline format - https://phabricator.wikimedia.org/T436490 [14:04:32] (03PS1) 10Dpogorzelski: kserve-llm-inference: Fix readme [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333212 [14:04:40] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [14:05:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:05:08] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [14:05:39] !log joal@deploy1003 Started deploy [analytics/refinery@19f4b59] (thin): Regular analytics weekly train THIN [analytics/refinery@19f4b595] [14:06:14] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [14:06:46] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [14:07:10] !log fceratto@cumin1003 START - Cookbook sre.mysql.depool depool pc1022: Test [14:07:10] !log fceratto@cumin1003 START - Cookbook sre.mysql.parsercache [14:07:13] !log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [14:07:13] !log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1022: Test [14:08:17] PROBLEM - OSPF status on cr2-magru is CRITICAL: OSPFv2: 2/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:08:22] FIRING: [2x] GnmiInterfaceCountersDrop: cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:08:55] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:09:03] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet [14:09:32] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet [14:09:39] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:10:08] !log fceratto@cumin1003 START - Cookbook sre.mysql.pool pool pc1022: Test [14:10:09] !log fceratto@cumin1003 START - Cookbook sre.mysql.parsercache [14:10:10] FIRING: BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:10:16] !log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [14:10:16] !log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1022: Test [14:11:17] RECOVERY - OSPF status on cr2-magru is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:12:15] (03CR) 10Tiziano Fogli: WIP: SLOP: gate ProbeDown on if a PoP is pooled (032 comments) [alerts] - 10https://gerrit.wikimedia.org/r/1329662 (owner: 10CDanis) [14:12:57] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] scaffold: Fix helm release NOTES.txt text (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332556 (https://phabricator.wikimedia.org/T433589) (owner: 10Jelto) [14:13:05] !log joal@deploy1003 Finished deploy [analytics/refinery@19f4b59] (thin): Regular analytics weekly train THIN [analytics/refinery@19f4b595] (duration: 07m 26s) [14:13:55] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:14:17] PROBLEM - OSPF status on cr2-magru is CRITICAL: OSPFv2: 2/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:14:24] !log jmm@cumin2003 START - Cookbook sre.puppet.disable-merges [14:14:25] !log jmm@cumin2003 END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) [14:14:39] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:15:10] FIRING: [3x] BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:15:31] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet [14:15:38] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply [14:15:43] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply [14:15:44] (03CR) 10Tiziano Fogli: alertmanager: add irc output for critical netops alerts (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1328172 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [14:16:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:16:48] (03PS1) 10Muehlenhoff: Revert "Temporarily remove puppetserver2004/1003 for reboots" [dns] - 10https://gerrit.wikimedia.org/r/1333217 [14:17:01] (03Abandoned) 10Ottomata: Partially revert "kafka: remove Kafka mirror maker configs" [puppet] - 10https://gerrit.wikimedia.org/r/1333206 (https://phabricator.wikimedia.org/T432089) (owner: 10Ottomata) [14:17:13] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [14:17:40] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [14:18:33] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12275821 (10MoritzMuehlenhoff) I've extended the RAM of the old nodes (to be repurposed as Bookworm nodes) also to 8... [14:19:10] (03CR) 10Muehlenhoff: [C:03+2] Revert "Temporarily remove puppetserver2004/1003 for reboots" [dns] - 10https://gerrit.wikimedia.org/r/1333217 (owner: 10Muehlenhoff) [14:19:25] (03CR) 10Tiziano Fogli: [C:03+1] "I left a comment on I9476536fe74f572ce44a74fc6bd8ad489c5a66c6, but otherwise LGTM. Thanks!" [alerts] - 10https://gerrit.wikimedia.org/r/1327077 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [14:19:29] !log jmm@dns1004 START - running authdns-update [14:19:39] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:20:09] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply [14:20:10] FIRING: [6x] BFDdown: BFD session down between cr1-magru and 195.200.68.161 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:20:30] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply [14:20:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:21:47] !log jmm@dns1004 END - running authdns-update [14:21:58] !log jmm@dns1004 START - running authdns-update [14:22:17] RECOVERY - OSPF status on cr2-magru is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:22:22] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [14:22:32] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [14:23:44] !log hashar@deploy1003 Started deploy [integration/docroot@780c44c]: opensource.yaml: Add CheckUser [14:23:58] !log hashar@deploy1003 Finished deploy [integration/docroot@780c44c]: opensource.yaml: Add CheckUser (duration: 00m 14s) [14:24:22] !log jmm@dns1004 END - running authdns-update [14:24:39] RESOLVED: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:24:41] !log installing curl security updates [14:24:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:24:43] (03CR) 10Tiziano Fogli: centralserver: add utility to delete oldest file based on disk usage (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1321669 (https://phabricator.wikimedia.org/T434690) (owner: 10Cwhite) [14:25:09] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply [14:25:10] RESOLVED: [6x] BFDdown: BFD session down between cr1-magru and 195.200.68.161 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:25:13] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply [14:25:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:26:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:27:40] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [14:27:43] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [14:28:00] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [14:28:05] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [14:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1430) [14:30:39] FIRING: CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (2a02:ec80:700:fe06::1) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=asw1-b3-magru:9804&var-bgp_group=core&var-bgp_neighbor=cr1-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:31:39] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [14:32:26] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [14:32:29] 10ops-eqiad, 06SRE, 06DC-Ops: Inbound errors on interface cr2-eqiad:et-1/1/5 (Transport: cr2-codfw:et-0/1/4 (Lumen, 449169461)) - https://phabricator.wikimedia.org/T435748#12275902 (10Jclark-ctr) a:03Jclark-ctr [14:33:16] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply [14:34:14] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply [14:34:39] 10ops-eqiad, 06SRE, 06DC-Ops: Inbound errors on interface cr2-eqiad:et-1/1/5 (Transport: cr2-codfw:et-0/1/4 (Lumen, 449169461)) - https://phabricator.wikimedia.org/T435748#12275926 (10Jclark-ctr) 05Open→03Resolved duplicate errors to T435810 [14:35:36] (03CR) 10Tiziano Fogli: "+1 from an o11y perspective. I'll leave the final call to @cmooney@wikimedia.org." [alerts] - 10https://gerrit.wikimedia.org/r/1321906 (owner: 10Ayounsi) [14:35:39] RESOLVED: CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (2a02:ec80:700:fe06::1) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=asw1-b3-magru:9804&var-bgp_group=core&var-bgp_neighbor=cr1-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:35:45] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply [14:35:53] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply [14:36:03] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply [14:36:07] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply [14:36:25] (03CR) 10Tiziano Fogli: "recheck" [alerts] - 10https://gerrit.wikimedia.org/r/1319831 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [14:39:10] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12275963 (10Jclark-ctr) @fgiunchedi I am back from sabbatical starting today. I do not see any progress on moving them since your last comment. I will tak... [14:39:27] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12275964 (10Jclark-ctr) a:05fgiunchedi→03Jclark-ctr [14:45:54] !log Restarted Gerrit on gerrit1003 and gerrit2002 [14:45:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:46:17] PROBLEM - OSPF status on cr2-magru is CRITICAL: OSPFv2: 2/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:46:39] FIRING: CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (2a02:ec80:700:fe06::1) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=asw1-b3-magru:9804&var-bgp_group=core&var-bgp_neighbor=cr1-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:46:45] (03CR) 10Tiziano Fogli: [C:03+1] team-sre: add dcops tag [alerts] - 10https://gerrit.wikimedia.org/r/1319831 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [14:48:00] (03CR) 10Tiziano Fogli: [C:03+1] team-sre: add infrastructure-foundations tag [alerts] - 10https://gerrit.wikimedia.org/r/1319832 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [14:48:10] FIRING: BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:48:44] (03CR) 10Tiziano Fogli: [C:03+1] team-sre: add data-persistence [alerts] - 10https://gerrit.wikimedia.org/r/1319830 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [14:48:55] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:48:57] !log Restarting Gerrit primary on gerrit2003 [14:48:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:49:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:51:39] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:51:50] (03CR) 10Clément Goubert: [C:03+1] "LGTM, worth spending the time to test in staging before rolling out to prod." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333155 (owner: 10Hnowlan) [14:51:58] !log Restarted CI Jenkins on contint1003 [14:51:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:52:41] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12276004 (10MoritzMuehlenhoff) [14:53:10] FIRING: [4x] BFDdown: BFD session down between cr1-magru and 195.200.68.161 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:53:17] RECOVERY - OSPF status on cr2-magru is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:53:55] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:54:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:55:42] !log Restarted Jenkins on releases1003 [14:55:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:56:39] RESOLVED: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:57:02] (03CR) 10Clément Goubert: [C:03+1] api-gateway: Support x-wikimedia-debug routing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311964 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [14:57:17] (03CR) 10Tiziano Fogli: [C:03+1] team-sre: make CertAlmostExpired team-aware [alerts] - 10https://gerrit.wikimedia.org/r/1311834 (https://phabricator.wikimedia.org/T414662) (owner: 10Hnowlan) [14:58:03] (03CR) 10Mpostoronca: EventStreamConfig: Register the abuse_review_interaction stream (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329360 (https://phabricator.wikimedia.org/T435517) (owner: 10Dreamy Jazz) [14:58:10] RESOLVED: [5x] BFDdown: BFD session down between cr1-magru and 195.200.68.161 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:58:59] (03CR) 10Clément Goubert: [C:03+1] api-gateway: Support Host-based diversion in the mw-api plugin [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322140 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [15:00:05] jelto, arnoldokoth, mutante, and arnaudb: SRE Collaboration Services office hours (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1500). Please do the needful. [15:00:48] (03CR) 10Tiziano Fogli: team-sre/hardware: add BBU check (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1308071 (https://phabricator.wikimedia.org/T430149) (owner: 10Hnowlan) [15:02:48] (03CR) 10Hnowlan: team-sre/hardware: add BBU check (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1308071 (https://phabricator.wikimedia.org/T430149) (owner: 10Hnowlan) [15:04:01] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission phab1004 - https://phabricator.wikimedia.org/T436635#12276051 (10Arnoldokoth) [15:06:13] (03CR) 10AOkoth: [C:03+2] phabricator: decom phab1004 [puppet] - 10https://gerrit.wikimedia.org/r/1332730 (https://phabricator.wikimedia.org/T436635) (owner: 10AOkoth) [15:06:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.8% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:08:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:08:52] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/datasets-config: apply [15:09:00] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/datasets-config: apply [15:09:07] !log joal@deploy1003 Started deploy [analytics/refinery@19f4b59]: Regular analytics weekly train [analytics/refinery@19f4b595] [15:10:13] (03CR) 10Hnowlan: [C:03+2] team-sre: make CertAlmostExpired team-aware [alerts] - 10https://gerrit.wikimedia.org/r/1311834 (https://phabricator.wikimedia.org/T414662) (owner: 10Hnowlan) [15:11:35] !log aokoth@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on phab1004.eqiad.wmnet with reason: Decom [15:12:27] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission phab1004 - https://phabricator.wikimedia.org/T436635#12276074 (10Arnoldokoth) Created silence ID `12c5000c-4299-42c1-b016-27ca844783ca` for 5 days. [15:12:28] 10ops-eqiad, 06SRE, 06DC-Ops: Lumen eqiad<->codfw transport outage (Aug 2026). - https://phabricator.wikimedia.org/T435810#12276076 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr Confirmed with ayounsi looks healthy. Resolving for now will reopen if it returns [15:12:48] (03Merged) 10jenkins-bot: team-sre: make CertAlmostExpired team-aware [alerts] - 10https://gerrit.wikimedia.org/r/1311834 (https://phabricator.wikimedia.org/T414662) (owner: 10Hnowlan) [15:13:42] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission phab1004 - https://phabricator.wikimedia.org/T436635#12276082 (10Arnoldokoth) [15:14:29] !log aokoth@cumin1003 START - Cookbook sre.hosts.decommission for hosts phab1004.eqiad.wmnet [15:14:40] !log joal@deploy1003 Finished deploy [analytics/refinery@19f4b59]: Regular analytics weekly train [analytics/refinery@19f4b595] (duration: 05m 32s) [15:18:36] PROBLEM - Host asw1-b3-magru is DOWN: PING CRITICAL - Packet loss = 100% [15:18:56] PROBLEM - Host durum7004 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:12] PROBLEM - Host ncredir7004 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:15] (03CR) 10Brouberol: "@mmuhlenhoff@wikimedia.org could I trouble you with a review?" [puppet] - 10https://gerrit.wikimedia.org/r/1319853 (https://phabricator.wikimedia.org/T421952) (owner: 10Brouberol) [15:19:20] PROBLEM - Host doh7003 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:24] PROBLEM - Host prometheus7002 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:34] RECOVERY - Host prometheus7002 is UP: PING OK - Packet loss = 0%, RTA = 112.97 ms [15:19:38] RECOVERY - Host ncredir7004 is UP: PING OK - Packet loss = 0%, RTA = 113.00 ms [15:19:38] RECOVERY - Host asw1-b3-magru is UP: PING OK - Packet loss = 0%, RTA = 121.20 ms [15:19:52] RECOVERY - Host doh7003 is UP: PING OK - Packet loss = 0%, RTA = 112.96 ms [15:20:07] uh [15:20:16] PROBLEM - OSPF status on cr2-magru is CRITICAL: OSPFv2: 2/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:20:18] !log aokoth@cumin1003 START - Cookbook sre.dns.netbox [15:20:18] RECOVERY - Host durum7004 is UP: PING OK - Packet loss = 0%, RTA = 113.10 ms [15:20:18] XioNoX: is there something happening in magru for network work? [15:20:45] FIRING: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:22:10] FIRING: [2x] BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:22:39] FIRING: [3x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:23:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:23:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.77% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:23:55] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:25:45] aokoth@cumin1003 decommission (PID 1837884) is awaiting input [15:27:10] RESOLVED: [2x] BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:27:39] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:27:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:27:58] !log aokoth@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: phab1004.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - aokoth@cumin1003" [15:28:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.77% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:29:15] !log aokoth@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: phab1004.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - aokoth@cumin1003" [15:29:15] !log aokoth@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:29:16] !log aokoth@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts phab1004.eqiad.wmnet [15:29:17] (03PS1) 10Blake: memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1333235 (https://phabricator.wikimedia.org/T353511) [15:29:22] (03CR) 10Blake: [C:03+2] memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1333235 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [15:29:26] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission phab1004 - https://phabricator.wikimedia.org/T436635#12276218 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by aokoth@cumin1003 for hosts: `phab1004.eqiad.wmnet` - phab1004.eqiad.wmnet (**PASS**)... [15:30:29] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:30:31] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission phab1004 - https://phabricator.wikimedia.org/T436635#12276221 (10Arnoldokoth) [15:30:54] (03PS1) 10Elukey: docker_registry: move away from Swift completely [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) [15:33:18] RECOVERY - OSPF status on cr2-magru is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:33:44] (03CR) 10Hnowlan: [C:03+2] "Thank you! And thanks for the scap fix." [puppet] - 10https://gerrit.wikimedia.org/r/1332699 (https://phabricator.wikimedia.org/T435340) (owner: 10Hnowlan) [15:33:52] PROBLEM - Host asw1-b3-magru is DOWN: PING CRITICAL - Packet loss = 100% [15:33:55] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:33:59] 10ops-ulsfo, 06DC-Ops: Inbound errors on interface asw1-23-ulsfo:mgmt0 (Core: msw2-ulsfo:18) - https://phabricator.wikimedia.org/T436668 (10phaultfinder) 03NEW [15:34:00] 10ops-ulsfo, 06DC-Ops: Inbound errors on interface asw1-22-ulsfo:mgmt0 (Core: msw1-ulsfo:18) - https://phabricator.wikimedia.org/T436669 (10phaultfinder) 03NEW [15:34:12] PROBLEM - Host durum7004 is DOWN: PING CRITICAL - Packet loss = 100% [15:34:24] PROBLEM - Host bast7002 is DOWN: PING CRITICAL - Packet loss = 100% [15:34:28] PROBLEM - Host ncredir7004 is DOWN: PING CRITICAL - Packet loss = 100% [15:34:40] PROBLEM - Host prometheus7002 is DOWN: PING CRITICAL - Packet loss = 100% [15:34:42] PROBLEM - Host doh7003 is DOWN: PING CRITICAL - Packet loss = 100% [15:34:44] PROBLEM - Host ncredir7003 is DOWN: PING CRITICAL - Packet loss = 100% [15:34:48] (03CR) 10Hnowlan: [C:03+2] Remove various hardcoded statsd.eqiad references [puppet] - 10https://gerrit.wikimedia.org/r/1332698 (https://phabricator.wikimedia.org/T435340) (owner: 10Hnowlan) [15:34:49] (03PS1) 10Muehlenhoff: Allow cumin1004 in alertmanager and IRC notifications [puppet] - 10https://gerrit.wikimedia.org/r/1333239 (https://phabricator.wikimedia.org/T427897) [15:35:02] PROBLEM - Host tcp-proxy7002 is DOWN: PING CRITICAL - Packet loss = 100% [15:35:17] XioNoX: ^ hardware issue or maintenance? [15:35:26] RECOVERY - Host durum7004 is UP: PING OK - Packet loss = 0%, RTA = 112.86 ms [15:35:26] RECOVERY - Host ncredir7003 is UP: PING OK - Packet loss = 0%, RTA = 113.01 ms [15:35:26] RECOVERY - Host ncredir7004 is UP: PING OK - Packet loss = 0%, RTA = 113.45 ms [15:35:28] RECOVERY - Host asw1-b3-magru is UP: PING OK - Packet loss = 0%, RTA = 145.85 ms [15:35:30] (for asw1-b3-magru) [15:35:32] RECOVERY - Host prometheus7002 is UP: PING OK - Packet loss = 0%, RTA = 113.13 ms [15:35:38] PROBLEM - Host 2a02:ec80:700:1:195:200:68:4 is DOWN: PING CRITICAL - Packet loss = 100% [15:35:48] RECOVERY - Host tcp-proxy7002 is UP: PING OK - Packet loss = 0%, RTA = 113.31 ms [15:35:56] RECOVERY - Host bast7002 is UP: PING OK - Packet loss = 0%, RTA = 112.89 ms [15:36:10] RECOVERY - Host doh7003 is UP: PING OK - Packet loss = 0%, RTA = 113.03 ms [15:36:18] wow [15:36:18] PROBLEM - OSPF status on cr2-magru is CRITICAL: OSPFv2: 2/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:36:26] moritzm: not a maintenance [15:36:32] RECOVERY - Host 2a02:ec80:700:1:195:200:68:4 is UP: PING OK - Packet loss = 0%, RTA = 112.53 ms [15:36:47] Cloudflare might be having a moment; no idea if that's related. See https://downdetector.com [15:37:06] (03PS1) 10Blake: memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1333240 (https://phabricator.wikimedia.org/T353511) [15:37:36] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:37:46] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:37:50] RESOLVED: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:37:56] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:38:10] FIRING: [2x] BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:38:23] (03PS2) 10Muehlenhoff: Allow cumin1004 in alertmanager and IRC notifications [puppet] - 10https://gerrit.wikimedia.org/r/1333239 (https://phabricator.wikimedia.org/T427897) [15:38:39] FIRING: [2x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (2a02:ec80:700:fe06::1) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:38:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:38:55] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:39:20] (03CR) 10Blake: [C:03+2] memcached: continue rollout of PKI use. [puppet] - 10https://gerrit.wikimedia.org/r/1333240 (https://phabricator.wikimedia.org/T353511) (owner: 10Blake) [15:41:12] (03CR) 10STran: [C:03+2] Update stream config for user_info_card_interaction [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332684 (https://phabricator.wikimedia.org/T435585) (owner: 10Mszwarc) [15:41:39] (03CR) 10Jgiannelos: [C:03+2] push-notifications: Use urldownloader LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333170 (https://phabricator.wikimedia.org/T429175) (owner: 10Clément Goubert) [15:41:40] (03PS2) 10CDanis: interface: Don't touch interfaces file if absent [puppet] - 10https://gerrit.wikimedia.org/r/1332813 (owner: 10BCornwall) [15:41:40] (03PS3) 10Dreamy Jazz: EventStreamConfig: Register the abuse_review_interaction stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329360 (https://phabricator.wikimedia.org/T435517) [15:41:40] (03CR) 10Dreamy Jazz: EventStreamConfig: Register the abuse_review_interaction stream (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329360 (https://phabricator.wikimedia.org/T435517) (owner: 10Dreamy Jazz) [15:41:41] (03CR) 10CDanis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332813 (owner: 10BCornwall) [15:42:23] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:42:29] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:42:44] moritzm: looks like a faulty interface flapping between cr1-magru and asw1-b3-magru I explicitely disabled it so things should stabilize [15:42:54] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:43:10] RESOLVED: [4x] BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:43:51] RESOLVED: SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b3-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:44:39] sukhe: ^ [15:44:40] FIRING: [3x] BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:45:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.59% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:45:44] XioNoX: thanks! [15:45:51] (03CR) 10CDanis: "This is a no-op in production (PCC reports "ControllerNoHostsError", which I believe means zero matches, plus, searching https://puppetboa" [puppet] - 10https://gerrit.wikimedia.org/r/1332813 (owner: 10BCornwall) [15:46:05] (03CR) 10Brouberol: [C:03+1] Grant sudo privileges for the analytics-experiment-users group [puppet] - 10https://gerrit.wikimedia.org/r/1328595 (https://phabricator.wikimedia.org/T416709) (owner: 10Atsuko) [15:48:13] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12276376 (10fgiunchedi) Thank you @Jclark-ctr for picking this up! Yes having two hosts in their final allocation one each per E4/F4 would be great. We're... [15:49:40] RESOLVED: [3x] BFDdown: BFD session down between cr2-magru and 195.200.68.160 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:49:51] ack, thx [15:50:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:51:32] (03PS1) 10Muehlenhoff: Allow cumin1004 for RAPI access [puppet] - 10https://gerrit.wikimedia.org/r/1333242 (https://phabricator.wikimedia.org/T427897) [15:51:38] !log installing mesa security updates [15:51:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:53:34] (03CR) 10Mszwarc: "Removed CR+2, not to merge before deployment window :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332684 (https://phabricator.wikimedia.org/T435585) (owner: 10Mszwarc) [15:55:51] (03CR) 10STran: [C:03+1] "Oops, sorry!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1332684 (https://phabricator.wikimedia.org/T435585) (owner: 10Mszwarc) [15:58:00] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [15:58:00] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [15:58:00] PROBLEM - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [15:59:29] 10ops-magru: magru: faulty link between cr1-magru and asw1-b3-magru - https://phabricator.wikimedia.org/T436675 (10ayounsi) 03NEW p:05Triage→03High [16:00:05] jhathaway and rzl: OwO what's this, a deployment window?? Puppet request window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1600). nyaa~ [16:00:05] No Gerrit patches in the queue for this window AFAICS. [16:01:20] (03PS1) 10CDobbins: prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) [16:02:10] (03CR) 10CI reject: [V:04-1] prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins) [16:02:47] (03PS2) 10CDobbins: prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) [16:03:00] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:03:35] (03CR) 10CI reject: [V:04-1] prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins) [16:04:12] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12276484 (10MoritzMuehlenhoff) [16:05:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:06:59] (03PS3) 10CDobbins: prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) [16:10:31] (03PS3) 10Aleksandar Mastilovic: blunderbuss: allow egress to the internal gitlab endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333153 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [16:10:37] (03PS1) 10Ladsgroup-claude: Package the project and stop using pkg_resources [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333246 [16:10:38] (03PS1) 10Ladsgroup-claude: Route all logging through thumbor's logger [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333247 [16:10:38] (03PS1) 10Ladsgroup-claude: Tidy up resource and rounding leftovers [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333248 [16:10:38] (03PS1) 10Ladsgroup-claude: Replace python-memcached with pymemcache [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333249 [16:10:39] (03PS1) 10Ladsgroup-claude: Call ImageMagick 7's magick instead of the deprecated convert [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333250 [16:10:41] (03PS1) 10Ladsgroup-claude: Clean up requirements and add a dependency bump path [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333251 [16:10:45] (03PS1) 10Ladsgroup-claude: Add a unit test layer that runs without the container [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333252 [16:10:49] (03PS1) 10Ladsgroup-claude: Replace flake8 with ruff and apply its fixes [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 [16:10:53] (03PS1) 10Ladsgroup-claude: Apply ruff format [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333254 [16:11:01] (03CR) 10Aleksandar Mastilovic: [C:03+1] blunderbuss: allow egress to the internal gitlab endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333153 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:01] (03CR) 10Ladsgroup: "I'm not 100% sure on the blubber and make file side of things but otherwise looks good." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333246 (owner: 10Ladsgroup-claude) [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:48] 10ops-magru: magru: faulty link between cr1-magru and asw1-b3-magru - https://phabricator.wikimedia.org/T436675#12276549 (10RobH) CS1342833 filed, listed myself, Arzhel, and Cathal on the watchlist for the ticket. > Support, > > We've recently experienced some connectivity issues on our router to switch link M... [16:18:02] (03CR) 10Atsuko: [C:03+2] blunderbuss: allow egress to the internal gitlab endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333153 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [16:20:41] (03Merged) 10jenkins-bot: blunderbuss: allow egress to the internal gitlab endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333153 (https://phabricator.wikimedia.org/T435292) (owner: 10Brouberol) [16:23:17] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [16:24:14] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [16:25:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.56% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:27:45] (03PS2) 10Ladsgroup: Package the project and stop using pkg_resources [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333246 (https://phabricator.wikimedia.org/T435609) (owner: 10Ladsgroup-claude) [16:28:20] (03PS1) 10JMeybohm: vap: Add support for overriding validationActions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333256 (https://phabricator.wikimedia.org/T436380) [16:29:43] 10ops-drmrs, 06DC-Ops, 06Traffic: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12276666 (10RobH) The ram landed yesterday, filing remote hands shortly to work on this host. [16:31:26] (03PS3) 10Hnowlan: thumbor: use native subprocess timeouts [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1181657 (https://phabricator.wikimedia.org/T379569) [16:35:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.59% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:36:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 17.46% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:40:56] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware: decommission phab1004 - https://phabricator.wikimedia.org/T436635#12276709 (10Jclark-ctr) a:03Jclark-ctr [16:40:58] 10ops-drmrs, 06DC-Ops, 06Traffic: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12276711 (10RobH) There is a 48 hour minimum window to have remote hands work on things without an urgency fee applied. I picked Friday with a note that the work can actual... [16:41:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:42:10] 10ops-drmrs, 06DC-Ops, 06Traffic: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12276714 (10RobH) [16:42:12] jouncebot now [16:42:12] For the next 0 hour(s) and 17 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1600) [16:42:57] !log dancy@deploy1003 Installing scap version "4.287.0" for 156 host(s) [16:43:45] (03PS1) 10Andrew Bogott: check_and_restart_networkd.sh: check for default route [puppet] - 10https://gerrit.wikimedia.org/r/1333259 [16:47:14] 07Puppet, 10Beta-Cluster-Infrastructure: Unduplicate beta cluster hiera keys set both in Horizon and in ops/puppet - https://phabricator.wikimedia.org/T277680#12276764 (10taavi) 05Open→03Resolved a:03taavi [16:47:23] !log dancy@deploy1003 Installation of scap version "4.287.0" completed for 156 hosts [16:48:19] !log dancy@deploy1003 Started scap sync-world: testing [16:48:24] !log dancy@deploy1003 sync-world aborted: testing (duration: 00m 05s) [16:48:32] (03CR) 10Ahmon Dancy: [C:03+2] mediawiki-dumps-legacy/values.yaml: Drop resources.replicas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326362 (https://phabricator.wikimedia.org/T375514) (owner: 10Ahmon Dancy) [16:49:48] (03CR) 10Andrew Bogott: [C:03+2] check_and_restart_networkd.sh: check for default route [puppet] - 10https://gerrit.wikimedia.org/r/1333259 (owner: 10Andrew Bogott) [16:50:54] (03Merged) 10jenkins-bot: mediawiki-dumps-legacy/values.yaml: Drop resources.replicas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326362 (https://phabricator.wikimedia.org/T375514) (owner: 10Ahmon Dancy) [16:52:52] !log dancy@deploy1003 Started scap sync-world: testing [16:58:00] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:58:00] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:58:00] RECOVERY - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:59:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1700) [17:00:59] !log dancy@deploy1003 Finished scap sync-world: testing (duration: 08m 07s) [17:01:10] (03CR) 10Jasmine: mw-web: upsize for single-DC serving (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [17:03:00] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:04:14] !log bking@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: Matomo version upgrade T431608 [17:04:15] (03CR) 10BCornwall: [V:03+1 C:03+1] "I feel that parsing iptables output could be made more comfortable by using e.g. `python3-iptables` (https://packages.debian.org/stable/py" [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins) [17:04:16] T431608: Upgrade Matomo from 4.16.1 to 5.12.0 (latest stable) - https://phabricator.wikimedia.org/T431608 [17:06:52] FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [17:08:50] 10ops-codfw, 06SRE, 06DC-Ops, 10decommission-hardware: decommission phab2002 - https://phabricator.wikimedia.org/T436632#12276865 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [17:09:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:10:42] (03PS1) 10Jforrester: Add thumb.* to allowed hosts for commons images [extensions/WikiLambda] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333263 (https://phabricator.wikimedia.org/T436579) [17:11:01] (03PS1) 10Jforrester: Add thumb.* to allowed hosts for commons images [extensions/WikiLambda] (wmf/1.47.0-wmf.17) - 10https://gerrit.wikimedia.org/r/1333264 (https://phabricator.wikimedia.org/T436579) [17:16:51] 06SRE, 10corto, 10Incident Tooling: Increase trusted volunteer's visibility into production incidents - https://phabricator.wikimedia.org/T426137#12276917 (10Novem_Linguae) @SomeRandomDeveloper. Sounds like you already signed an NDA but are not in the phabricator NDA group. Normally this group is obtained by... [17:36:06] (03CR) 10Ottomata: "Wow strange. Perhaps we should rename the python-webapp chart then eh?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [17:36:24] 10ops-codfw, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691 (10Jhancock.wm) 03NEW [17:36:33] (03CR) 10Ottomata: "Oh, sorry, you have that in values.yaml. Maybe just add a short comment in helmfile.yaml then." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [17:39:44] (03PS2) 10DLynch: editcheck-headless: Add deployment for the technical pilot [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) [17:40:27] (03CR) 10DLynch: "I added a short explanation into helmfile.yaml. 👍🏻" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [17:41:34] 10ops-codfw, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12277095 (10Jhancock.wm) @ssingh could you, or someone in traffic, verify the information in Installation Details of this task? Also, please update preseed.yaml and site.pp to include this s... [17:46:44] (03CR) 10Ottomata: [C:03+1] Declare the webrequest.dumps.v1 stream in EventStreamConfig [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328201 (https://phabricator.wikimedia.org/T425087) (owner: 10Btullis) [17:47:12] (03CR) 10BCornwall: [C:03+1] Geo-maps: Update Meta mapping for september 2026 [dns] - 10https://gerrit.wikimedia.org/r/1333134 (owner: 10Slyngshede) [17:53:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:58:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:58:40] 06SRE, 10corto, 10Incident Tooling: Increase trusted volunteer's visibility into production incidents - https://phabricator.wikimedia.org/T426137#12277171 (10hnowlan) >>! In T426137#12272967, @SomeRandomDeveloper wrote: > Since the view policy for incident tasks was changed to #wmf-nda, users like me who are... [18:00:05] dancy and hashar: Deploy window MediaWiki train - Utc-7+Utc-0 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T1800) [18:00:37] o/ [18:01:07] (03PS1) 10TrainBranchBot: group0 to 1.47.0-wmf.18 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1333272 (https://phabricator.wikimedia.org/T430837) [18:01:10] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dancy@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1333272 (https://phabricator.wikimedia.org/T430837) (owner: 10TrainBranchBot) [18:02:07] (03Merged) 10jenkins-bot: group0 to 1.47.0-wmf.18 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1333272 (https://phabricator.wikimedia.org/T430837) (owner: 10TrainBranchBot) [18:02:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.42% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [18:07:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [18:08:37] FIRING: [2x] GnmiInterfaceCountersDrop: cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [18:12:37] 10ops-codfw, 06SRE, 06DC-Ops, 10decommission-hardware: decommission cumin2002.codfw.wmnet - https://phabricator.wikimedia.org/T436330#12277231 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [18:14:00] !log dancy@deploy1003 rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.18 refs T430837 [18:14:07] T430837: 1.47.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T430837 [18:14:52] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware: decommission phab1004 - https://phabricator.wikimedia.org/T436635#12277241 (10Jclark-ctr) [18:14:59] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware: decommission phab1004 - https://phabricator.wikimedia.org/T436635#12277242 (10Jclark-ctr) 05Open→03Resolved [18:20:16] 10ops-codfw, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12277263 (10Jhancock.wm) a:03ssingh [18:23:42] (03PS1) 10Ssingh: sretest2013: set role to cache::text [puppet] - 10https://gerrit.wikimedia.org/r/1333275 (https://phabricator.wikimedia.org/T436691) [18:25:31] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thanks!!" [puppet] - 10https://gerrit.wikimedia.org/r/1333239 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [18:32:28] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12277300 (10ssingh) >>! In T436691#12277094, @Jhancock.wm wrote: > @ssingh could you, or someone in traffic, verify the information in Installation Details of... [18:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [18:45:43] (03PS1) 10Jdlrobson: Preserve showlogin query parameter on redirects [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333277 (https://phabricator.wikimedia.org/T435248) [18:54:59] (03PS1) 10Ssingh: Revert^2 "wikimedia.org: add TXT record for BIMI" [dns] - 10https://gerrit.wikimedia.org/r/1333278 [19:03:24] (03CR) 10Ssingh: [C:03+2] Revert^2 "wikimedia.org: add TXT record for BIMI" [dns] - 10https://gerrit.wikimedia.org/r/1333278 (owner: 10Ssingh) [19:03:38] !log sukhe@dns1004 START - running authdns-update [19:05:55] (03CR) 10Ssingh: [C:03+1] "I can take care of rolling this out Wed Sep 2 (and pool the hosts before the change)." [puppet] - 10https://gerrit.wikimedia.org/r/1333201 (https://phabricator.wikimedia.org/T429175) (owner: 10Muehlenhoff) [19:06:01] !log sukhe@dns1004 END - running authdns-update [19:06:07] !log jclark@cumin1003 START - Cookbook sre.dns.netbox [19:07:17] 10SRE-Access-Requests, 06Data-Engineering: Requesting Kerberos access for kamila - https://phabricator.wikimedia.org/T436703#12277508 (10Raine) Late-tagging this as an access request, as I realised also need group membership. [19:10:54] (03PS1) 10Kamila Součková: admin: add kamila to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1333280 (https://phabricator.wikimedia.org/T436703) [19:11:14] !log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudcephosd1055 to eqiad - jclark@cumin1003" [19:11:18] !log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudcephosd1055 to eqiad - jclark@cumin1003" [19:11:18] !log jclark@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [19:12:26] !log jclark@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1054 [19:12:29] !log jclark@cumin1003 END (ERROR) - Cookbook sre.network.configure-switch-interfaces (exit_code=97) for host cloudcephosd1054 [19:12:32] !log jclark@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1055 [19:12:42] !log jclark@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1055 [19:12:45] !log jclark@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1056 [19:12:54] !log jclark@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1056 [19:13:24] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:14:37] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:14:48] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:16:15] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:21:46] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host aqs1016.eqiad.wmnet with OS bookworm [19:22:50] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:23:47] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:24:02] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:24:37] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:25:06] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:25:19] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:25:33] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:25:42] FIRING: [2x] ProbeDown: Service aqs1016-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:25:53] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:30:24] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:30:39] !log cdobbins@cumin1003 START - Cookbook sre.hosts.remove-downtime for cp5022.eqsin.wmnet [19:30:40] !log cdobbins@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp5022.eqsin.wmnet [19:30:43] FIRING: [4x] ProbeDown: Service aqs1016-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:30:53] !log cdobbins@cumin1003 conftool action : set/pooled=yes; selector: name=cp5022.* [reason: update IP addrs] [19:32:28] !log jhancock@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1055 [19:32:35] !log jhancock@cumin1003 END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudcephosd1055 [19:32:36] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12277608 (10Jclark-ctr) @elukey I saw some emails regarding changes to provisioning while I was out on sabbatical. I’m having some issues with provisionin... [19:34:14] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations: attempts to reimage cp5022 are failing - https://phabricator.wikimedia.org/T436386#12277616 (10CDobbins) @elukey: yep, feel free to close it! Thank you so much for helping us out. [19:35:29] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:38:55] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:39:44] !log eevans@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host aqs1016.eqiad.wmnet with OS bookworm [19:40:55] !log eevans@cumin1003 START - Cookbook sre.hosts.provision for host aqs1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [19:41:57] !log jhancock@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:42:51] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host aqs1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [19:42:54] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:45:41] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host aqs1016.eqiad.wmnet with OS bookworm [19:47:30] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:50:42] RESOLVED: [4x] ProbeDown: Service aqs1016-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:52:58] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:53:07] (03CR) 10Andrew Bogott: [C:03+1] "This looks good. I think we need to do something about powersave mode on Dell hosts too, though... if I look at /sys/devices/system/cpu/cp" [cookbooks] - 10https://gerrit.wikimedia.org/r/1333146 (https://phabricator.wikimedia.org/T435537) (owner: 10Elukey) [19:54:57] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:56:37] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:56:54] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:56:56] !log eevans@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host aqs1016.eqiad.wmnet with OS bookworm [19:57:11] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host aqs1016.eqiad.wmnet with OS bookworm [19:57:15] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [19:57:35] cdobbins@cumin1003 reimage (PID 1462957) is awaiting input [19:59:01] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T2000). [20:00:05] No Gerrit patches in the queue for this window AFAICS. [20:11:37] !log eevans@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host aqs1016.eqiad.wmnet with OS bookworm [20:11:52] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host aqs1016.eqiad.wmnet with OS bookworm [20:17:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:17:24] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in module prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1332523 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:17:38] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332515 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:21:51] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in profile analytics [puppet] - 10https://gerrit.wikimedia.org/r/1332515 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:21:58] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in module bigtop [puppet] - 10https://gerrit.wikimedia.org/r/1332512 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:23:15] 06SRE, 10corto, 10Incident Tooling: Increase trusted volunteer's visibility into production incidents - https://phabricator.wikimedia.org/T426137#12277753 (10SomeRandomDeveloper) >>! In T426137#12276917, @Novem_Linguae wrote: > @SomeRandomDeveloper. Sounds like you already signed an NDA but are not in the ph... [20:24:43] !log eevans@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on aqs1016.eqiad.wmnet with reason: host reimage [20:26:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 18.04% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:28:18] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile mariadb [puppet] - 10https://gerrit.wikimedia.org/r/1332520 (https://phabricator.wikimedia.org/T435225) [20:28:25] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332520 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:28:27] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1016.eqiad.wmnet with reason: host reimage [20:31:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:36:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:36:59] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in profile mariadb [puppet] - 10https://gerrit.wikimedia.org/r/1332520 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:37:22] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in mariadb::config [puppet] - 10https://gerrit.wikimedia.org/r/1329665 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:38:00] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332502 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:38:01] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332503 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:38:03] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332501 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:38:05] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332516 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:41:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:41:51] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module rsync [puppet] - 10https://gerrit.wikimedia.org/r/1332502 (https://phabricator.wikimedia.org/T435225) [20:41:56] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332502 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:42:27] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in profile puppet_compiler [puppet] - 10https://gerrit.wikimedia.org/r/1332503 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:44:06] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module rancid [puppet] - 10https://gerrit.wikimedia.org/r/1332501 (https://phabricator.wikimedia.org/T435225) [20:44:10] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332501 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:45:36] (03PS1) 10Southparkfan: Redis instance arclamp: allow dual-stack, disable protected mode [puppet] - 10https://gerrit.wikimedia.org/r/1333288 [20:46:22] (03PS2) 10Southparkfan: Redis instance arclamp: allow dual-stack, disable protected mode [puppet] - 10https://gerrit.wikimedia.org/r/1333288 (https://phabricator.wikimedia.org/T436464) [20:46:50] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in module rancid [puppet] - 10https://gerrit.wikimedia.org/r/1332501 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:47:51] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs1016.eqiad.wmnet with OS bookworm [20:48:41] (03CR) 10CI reject: [V:04-1] Redis instance arclamp: allow dual-stack, disable protected mode [puppet] - 10https://gerrit.wikimedia.org/r/1333288 (https://phabricator.wikimedia.org/T436464) (owner: 10Southparkfan) [20:50:48] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module bacula [puppet] - 10https://gerrit.wikimedia.org/r/1332516 (https://phabricator.wikimedia.org/T435225) [20:50:49] (03PS3) 10Southparkfan: Redis instance arclamp: allow dual-stack, disable protected mode [puppet] - 10https://gerrit.wikimedia.org/r/1333288 (https://phabricator.wikimedia.org/T436464) [20:50:54] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1332516 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:51:02] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in module rsync [puppet] - 10https://gerrit.wikimedia.org/r/1332502 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [20:51:25] FIRING: [3x] ProbeDown: Service aqs1016-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:53:43] (03PS1) 10Urbanecm: mediawiki::maintenance::growthexperiments: Document interval expectations [puppet] - 10https://gerrit.wikimedia.org/r/1333291 (https://phabricator.wikimedia.org/T436659) [20:56:25] FIRING: [4x] ProbeDown: Service aqs1016-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:59:00] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [21:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260901T2100) [21:04:22] okay to take the deploy window? [21:04:42] (03PS1) 10Jdlrobson: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333293 [21:04:50] (03PS1) 10Jdlrobson: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf/1.47.0-wmf.17) - 10https://gerrit.wikimedia.org/r/1333294 [21:05:53] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333293 (owner: 10Jdlrobson) [21:05:56] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf/1.47.0-wmf.17) - 10https://gerrit.wikimedia.org/r/1333294 (owner: 10Jdlrobson) [21:06:25] FIRING: [4x] ProbeDown: Service aqs1016-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:06:52] FIRING: NetworkDeviceAlarmActive: Alarm active on cr1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [21:07:40] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/CentralNotice] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333293 (owner: 10Jdlrobson) [21:07:41] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333277 (https://phabricator.wikimedia.org/T435248) (owner: 10Jdlrobson) [21:08:32] (03Merged) 10jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333293 (owner: 10Jdlrobson) [21:11:25] RESOLVED: [4x] ProbeDown: Service aqs1016-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:14:26] (03Merged) 10jenkins-bot: Preserve showlogin query parameter on redirects [core] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333277 (https://phabricator.wikimedia.org/T435248) (owner: 10Jdlrobson) [21:14:52] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1333293|Merge branch 'master' into wmf_deploy]], [[gerrit:1333277|Preserve showlogin query parameter on redirects (T435248)]] [21:14:56] T435248: Allow display of login link on create account form - https://phabricator.wikimedia.org/T435248 [21:16:51] jdlrobson: Please be aware of https://phabricator.wikimedia.org/T436663 which is affecting deployment error checks [21:19:46] (03PS1) 10Ahmon Dancy: kubernetes.yaml: Add logstash_check settings to mw_deployments [puppet] - 10https://gerrit.wikimedia.org/r/1333296 (https://phabricator.wikimedia.org/T435419) [21:20:22] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in module bacula [puppet] - 10https://gerrit.wikimedia.org/r/1332516 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [21:20:48] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1333293|Merge branch 'master' into wmf_deploy]], [[gerrit:1333277|Preserve showlogin query parameter on redirects (T435248)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:20:51] T435248: Allow display of login link on create account form - https://phabricator.wikimedia.org/T435248 [21:28:35] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [21:37:08] dancy: yes im hitting that so should I ignore failure and continue? [21:37:53] The x4/test-s4/foreach() errors can be ignored. [21:38:16] i.e,. it looks like you're good to go [21:38:38] So Ignore failure and continue deployment [21:38:44] yes [21:38:48] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1333293|Merge branch 'master' into wmf_deploy]], [[gerrit:1333277|Preserve showlogin query parameter on redirects (T435248)]] (duration: 23m 55s) [21:38:50] thanks for confirming [21:38:51] T435248: Allow display of login link on create account form - https://phabricator.wikimedia.org/T435248 [21:38:58] I have one more to do after this.. (sorry!) [21:39:05] i suspect i'll hit same issue [21:39:07] No prob. It will be the same. [21:39:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [extensions/CentralNotice] (wmf/1.47.0-wmf.17) - 10https://gerrit.wikimedia.org/r/1333294 (owner: 10Jdlrobson) [21:41:23] (03Merged) 10jenkins-bot: Merge branch 'master' into wmf_deploy [extensions/CentralNotice] (wmf/1.47.0-wmf.17) - 10https://gerrit.wikimedia.org/r/1333294 (owner: 10Jdlrobson) [21:41:51] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1333294|Merge branch 'master' into wmf_deploy]] [21:47:43] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1333294|Merge branch 'master' into wmf_deploy]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:52:36] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [21:59:56] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1333294|Merge branch 'master' into wmf_deploy]] (duration: 18m 05s) [22:05:06] finished [22:08:38] FIRING: [2x] GnmiInterfaceCountersDrop: cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [22:12:53] (03PS1) 10Southparkfan: LabsServices: adjust Arc Lamp and Excimer URIs to use service RRs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1333301 (https://phabricator.wikimedia.org/T436464) [22:28:05] 06SRE, 10corto, 10Incident Tooling: Increase trusted volunteer's visibility into production incidents - https://phabricator.wikimedia.org/T426137#12278249 (10Novem_Linguae) >>! In T426137#12277753, @SomeRandomDeveloper wrote: > However, out of the 9 other members in #acl_security_volunteer, 3 are also not in... [23:22:47] (03PS1) 10Jdlrobson: Campaigns should not override existing campaign query strings [extensions/Campaigns] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1333307 (https://phabricator.wikimedia.org/T436681) [23:30:39] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:38:55] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-magru:et-0/0/1 (Core: asw1-b3-magru:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [23:41:18] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1333309 [23:41:18] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1333309 (owner: 10TrainBranchBot) [23:42:54] FIRING: [4x] CoreBGPDown: Core BGP session down between asw1-b3-magru and cr1-magru (195.200.68.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [23:50:31] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1333309 (owner: 10TrainBranchBot)