[00:01:10] FIRING: [3x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [00:06:10] FIRING: [3x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [00:13:12] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:18:45] FIRING: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [00:36:54] FIRING: CirrusSearchTitleSuggestIndexTooOld: Some search indices that power autocomplete have not been updated recently - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - TODO - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchTitleSuggestIndexTooOld [00:40:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [00:53:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [01:11:08] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1326085 [01:11:08] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1326085 (owner: 10TrainBranchBot) [01:21:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:23:11] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1326085 (owner: 10TrainBranchBot) [01:32:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:00:20] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:08] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 47s) [02:08:12] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:12] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:18:12] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:41:08] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [alerts] - 10https://gerrit.wikimedia.org/r/1311834 (https://phabricator.wikimedia.org/T414662) (owner: 10Hnowlan) [03:05:50] FIRING: DiskSpace: Disk space build2001:9100:/ 2.395% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=build2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:20:35] RESOLVED: DiskSpace: Disk space build2001:9100:/ 3.082% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=build2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:32:31] FIRING: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [03:48:36] FIRING: [2x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:52:31] RESOLVED: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [04:00:22] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [04:01:22] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [04:02:48] PROBLEM - Check unit status of statograph_post on alert1002 is CRITICAL: CRITICAL: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [04:06:10] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [04:12:48] RECOVERY - Check unit status of statograph_post on alert1002 is OK: OK: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [04:36:54] FIRING: CirrusSearchTitleSuggestIndexTooOld: Some search indices that power autocomplete have not been updated recently - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - TODO - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchTitleSuggestIndexTooOld [04:40:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [05:21:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:32:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:43:36] (03PS1) 10KartikMistry: Update Mint to 2026-06-04-131507-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326088 (https://phabricator.wikimedia.org/T321316) [06:06:48] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host gerrit2003.wikimedia.org [06:07:06] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host vrts1003.eqiad.wmnet [06:08:46] FIRING: [2x] GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable [06:09:46] FIRING: [2x] GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in codfw - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable [06:13:22] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host vrts1003.eqiad.wmnet [06:13:38] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2003.wikimedia.org [06:13:46] RESOLVED: [4x] GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable [06:14:46] RESOLVED: [2x] GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in codfw - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable [06:18:12] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:25:56] (03CR) 10Samwilson: [C:03+1] "Looks good!" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1325499 (https://phabricator.wikimedia.org/T419815) (owner: 10Ladsgroup) [06:38:54] (03CR) 10Filippo Giunchedi: [C:03+2] site: get cloudvirts ready for E4 -> C8 move [puppet] - 10https://gerrit.wikimedia.org/r/1325458 (https://phabricator.wikimedia.org/T431682) (owner: 10Filippo Giunchedi) [06:41:12] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1049.eqiad.wmnet [06:42:48] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12219727 (10fgiunchedi) [06:44:47] filippo@cumin1003 decommission (PID 245016) is awaiting input [06:48:31] (03CR) 10Muehlenhoff: Upgrade to trixie (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1325499 (https://phabricator.wikimedia.org/T419815) (owner: 10Ladsgroup) [06:49:59] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [06:52:24] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet [06:52:52] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1325922 (https://phabricator.wikimedia.org/T357791) (owner: 10Eevans) [06:53:53] !log installing apr-util security updates [06:53:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:54:00] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcumin1001.eqiad.wmnet [06:54:19] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [06:54:45] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [06:54:45] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [06:54:47] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1049.eqiad.wmnet [06:55:00] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1051.eqiad.wmnet [06:55:07] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12219735 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1049.eqiad.wmnet` - cloudvirt1049.eqiad.wmnet (... [06:55:48] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcumin2001.codfw.wmnet [06:58:56] jmm@cumin2003 drain-node (PID 2373867) is awaiting input [06:59:36] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin2001.codfw.wmnet [06:59:37] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1043.eqiad.wmnet [06:59:47] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin1001.eqiad.wmnet [06:59:50] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [07:00:05] Amir1, urbanecm, and awight: Time to do the UTC morning backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T0700). [07:00:05] Nvdtn19: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:02:42] (03CR) 10Dragoniez: "Have you joined the IRC channel?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1216721 (https://phabricator.wikimedia.org/T405724) (owner: 10Nvdtn19) [07:04:05] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:05:00] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1043.eqiad.wmnet [07:05:00] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:05:01] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [07:05:02] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1051.eqiad.wmnet [07:05:19] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1054.eqiad.wmnet [07:05:20] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12219761 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1051.eqiad.wmnet` - cloudvirt1051.eqiad.wmnet (... [07:05:32] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet [07:07:20] (03CR) 10Nvdtn19: "Yes" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1216721 (https://phabricator.wikimedia.org/T405724) (owner: 10Nvdtn19) [07:08:25] (03CR) 10Filippo Giunchedi: [C:03+1] P:dumps: Remove unused variable [puppet] - 10https://gerrit.wikimedia.org/r/1325490 (https://phabricator.wikimedia.org/T417824) (owner: 10Majavah) [07:09:24] (03PS1) 10Marostegui: instances.yaml: Add db1275 [puppet] - 10https://gerrit.wikimedia.org/r/1326109 [07:10:05] Is there still a deployer around? [07:10:29] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [07:10:46] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1275 [puppet] - 10https://gerrit.wikimedia.org/r/1326109 (owner: 10Marostegui) [07:12:26] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1275 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96098 and previous config saved to /var/cache/conftool/dbconfig/20260817-071225-marostegui.json [07:12:31] T407942: Productionize db12[65-90] - https://phabricator.wikimedia.org/T407942 [07:12:54] (03PS1) 10Marostegui: db1275: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326110 [07:13:33] (03CR) 10Marostegui: [C:03+2] db1275: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326110 (owner: 10Marostegui) [07:13:49] Nvdtn19: Unfortunately you might have missed them. If there's no deployer around you'll need to reschedule the backport again [07:13:55] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1275: Pool back [07:14:47] Thanks, I'll reschedule [07:15:00] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:15:28] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1044.eqiad.wmnet [07:15:35] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:15:35] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [07:15:37] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1054.eqiad.wmnet [07:15:42] (03PS1) 10Marostegui: mariadb: Productionize db1289 [puppet] - 10https://gerrit.wikimedia.org/r/1326111 (https://phabricator.wikimedia.org/T407942) [07:15:58] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12219772 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1054.eqiad.wmnet` - cloudvirt1054.eqiad.wmnet (... [07:16:03] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1055.eqiad.wmnet [07:16:57] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1289 [puppet] - 10https://gerrit.wikimedia.org/r/1326111 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [07:17:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 17 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1216721 (https://phabricator.wikimedia.org/T405724) (owner: 10Nvdtn19) [07:18:40] (03PS1) 10Marostegui: db1279: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326112 [07:20:29] (03CR) 10Marostegui: [C:03+2] db1279: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326112 (owner: 10Marostegui) [07:20:44] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1044.eqiad.wmnet [07:21:59] (03PS6) 10Dpogorzelski: ml-serve: fallback for GPU usage and power metrics [puppet] - 10https://gerrit.wikimedia.org/r/1321545 [07:22:17] (03CR) 10Dpogorzelski: ml-serve: fallback for GPU usage and power metrics (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1321545 (owner: 10Dpogorzelski) [07:22:49] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [07:24:29] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1175.eqiad.wmnet onto db1289.eqiad.wmnet [07:24:33] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1175: Depool db1175.eqiad.wmnet to then clone it to db1289.eqiad.wmnet - marostegui@cumin1003 [07:24:58] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1175: Depool db1175.eqiad.wmnet to then clone it to db1289.eqiad.wmnet - marostegui@cumin1003 [07:25:06] Dragoniez, Nvdtn19: I'm sorry for arriving late. I can deploy your change if you're able to test and confirm that it works? [07:25:08] (03PS1) 10Marostegui: instances.yaml: Add db1279 [puppet] - 10https://gerrit.wikimedia.org/r/1326114 [07:25:30] !log slyngshede@cumin1003 START - Cookbook sre.hosts.reboot-single for host idm1001.wikimedia.org [07:25:49] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1279 [puppet] - 10https://gerrit.wikimedia.org/r/1326114 (owner: 10Marostegui) [07:26:00] Nvdtn19: Are you still there? [07:26:06] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1044.eqiad.wmnet [07:26:11] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1044.eqiad.wmnet [07:26:11] phuedx: no worries :) [07:26:13] Yes, you can deploy it [07:27:09] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1055.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:27:16] Nvdtn19: Do you have the WikimediaDebug browser extension installed? See https://wikitech.wikimedia.org/wiki/WikimediaDebug [07:27:38] Yes I installed [07:28:10] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1055.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:28:10] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [07:28:11] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1055.eqiad.wmnet [07:28:17] !log extend the disk of ldap-rw1001 by 80G T331699 [07:28:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:28:21] T331699: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699 [07:28:36] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12219799 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1055.eqiad.wmnet` - cloudvirt1055.eqiad.wmnet (... [07:28:52] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1056.eqiad.wmnet [07:29:20] (03CR) 10Dpogorzelski: [C:03+2] ml-serve: fallback for GPU usage and power metrics [puppet] - 10https://gerrit.wikimedia.org/r/1321545 (owner: 10Dpogorzelski) [07:29:27] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm1001.wikimedia.org [07:29:43] !log slyngshede@cumin1003 START - Cookbook sre.hosts.reboot-single for host idm-test1001.wikimedia.org [07:30:39] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1279 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96102 and previous config saved to /var/cache/conftool/dbconfig/20260817-073038-marostegui.json [07:30:40] Nvdtn19: I'm just logging in and setting up. Gimme a mo [07:30:44] T407942: Productionize db12[65-90] - https://phabricator.wikimedia.org/T407942 [07:30:44] (03PS1) 10Slyngshede: IDM: Switch over for reboot [dns] - 10https://gerrit.wikimedia.org/r/1326116 [07:31:03] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1279: Pool back [07:31:11] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet [07:31:55] (03CR) 10Slyngshede: [C:03+2] IDM: Switch over for reboot [dns] - 10https://gerrit.wikimedia.org/r/1326116 (owner: 10Slyngshede) [07:32:05] !log slyngshede@dns1004 START - running authdns-update [07:33:20] (03PS1) 10Marostegui: mariadb: Productionize db1288 [puppet] - 10https://gerrit.wikimedia.org/r/1326117 (https://phabricator.wikimedia.org/T407942) [07:33:39] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm-test1001.wikimedia.org [07:33:54] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet [07:34:07] !log slyngshede@dns1004 END - running authdns-update [07:34:19] Nvdtn19: I've reviewed the change. I double-checked that RelatedArticles was enabled on the wiki. I'll ping you once the change is ready to test [07:34:24] (03CR) 10TrainBranchBot: [C:03+2] "Approved by phuedx@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1216721 (https://phabricator.wikimedia.org/T405724) (owner: 10Nvdtn19) [07:34:32] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1288 [puppet] - 10https://gerrit.wikimedia.org/r/1326117 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [07:34:36] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [07:34:43] phuedx: ok [07:35:22] (03Merged) 10jenkins-bot: viwikivoyage: enable relatedarticle and pop-up [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1216721 (https://phabricator.wikimedia.org/T405724) (owner: 10Nvdtn19) [07:35:43] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1279 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96104 and previous config saved to /var/cache/conftool/dbconfig/20260817-073542-marostegui.json [07:35:48] T407942: Productionize db12[65-90] - https://phabricator.wikimedia.org/T407942 [07:36:12] !log slyngshede@cumin1003 START - Cookbook sre.hosts.reboot-single for host idm2001.wikimedia.org [07:36:13] !log phuedx@deploy1003 Started scap sync-world: Backport for [[gerrit:1216721|viwikivoyage: enable relatedarticle and pop-up (T405724)]] [07:36:19] T405724: Adding RelatedArticles and Popups extension for vi.wikivoyage - https://phabricator.wikimedia.org/T405724 [07:36:28] (03CR) 10Marostegui: [C:03+1] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [07:38:16] (03PS1) 10Marostegui: site.pp: Fixed typo [puppet] - 10https://gerrit.wikimedia.org/r/1326119 [07:38:29] (03CR) 10Klausman: [C:03+1] elasticsearch: remove non-working freeze/unfreeze code [software/spicerack] - 10https://gerrit.wikimedia.org/r/1325955 (https://phabricator.wikimedia.org/T433306) (owner: 10Bking) [07:38:35] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1056.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:38:37] (03CR) 10Marostegui: [V:03+2 C:03+2] site.pp: Fixed typo [puppet] - 10https://gerrit.wikimedia.org/r/1326119 (owner: 10Marostegui) [07:39:00] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1056.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:39:00] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [07:39:02] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1056.eqiad.wmnet [07:39:19] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12219841 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1056.eqiad.wmnet` - cloudvirt1056.eqiad.wmnet (... [07:40:03] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet [07:40:11] !log slyngshede@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm2001.wikimedia.org [07:40:20] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet [07:41:29] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1174.eqiad.wmnet onto db1288.eqiad.wmnet [07:41:34] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1174: Depool db1174.eqiad.wmnet to then clone it to db1288.eqiad.wmnet - marostegui@cumin1003 [07:41:49] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1174: Depool db1174.eqiad.wmnet to then clone it to db1288.eqiad.wmnet - marostegui@cumin1003 [07:42:01] (03PS1) 10Marostegui: db1281: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326122 [07:42:02] (03PS7) 10Dreamy Jazz: Register the mediawiki.wikimedia_antiabuse.content_policy_score stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1321579 (https://phabricator.wikimedia.org/T432848) (owner: 10Mpostoronca) [07:42:21] (03PS1) 10Muehlenhoff: proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326123 [07:42:32] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet [07:42:46] (03CR) 10Marostegui: [C:03+2] db1281: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326122 (owner: 10Marostegui) [07:43:21] FIRING: [3x] ProbeDown: Service ganeti1029:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:44:22] (03PS1) 10Marostegui: instances.yaml: Add db1281 [puppet] - 10https://gerrit.wikimedia.org/r/1326124 (https://phabricator.wikimedia.org/T407942) [07:44:49] marostegui@cumin1003 clone (PID 253418) is awaiting input [07:45:03] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1281 [puppet] - 10https://gerrit.wikimedia.org/r/1326124 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [07:45:30] (03CR) 10Federico Ceratto: [C:03+2] cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [07:46:35] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet [07:47:50] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1281 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96108 and previous config saved to /var/cache/conftool/dbconfig/20260817-074749-marostegui.json [07:47:54] T407942: Productionize db12[65-90] - https://phabricator.wikimedia.org/T407942 [07:48:20] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1281: Pool back [07:49:52] (03Merged) 10jenkins-bot: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [07:50:49] !log filippo@cumin1003 START - Cookbook sre.hosts.decommission for hosts cloudvirt1057.eqiad.wmnet [07:51:43] !log bump space for prometheus k8s-aux in eqiad [07:51:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:52:40] !log phuedx@deploy1003 nvdtn19, phuedx: Backport for [[gerrit:1216721|viwikivoyage: enable relatedarticle and pop-up (T405724)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:52:44] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet [07:52:44] T405724: Adding RelatedArticles and Popups extension for vi.wikivoyage - https://phabricator.wikimedia.org/T405724 [07:52:49] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet [07:53:15] Nvdtn19: The change is ready to test. Please test it thoroughly on the debug servers [07:53:21] FIRING: [4x] ProbeDown: Service ganeti1029:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:54:08] (03CR) 10Filippo Giunchedi: [C:03+2] hieradata: enable dumps-nfs.w.o usage in production [puppet] - 10https://gerrit.wikimedia.org/r/1324597 (https://phabricator.wikimedia.org/T432212) (owner: 10Filippo Giunchedi) [07:54:33] (03PS1) 10Marostegui: mariadb: Productionize db1287 [puppet] - 10https://gerrit.wikimedia.org/r/1326125 (https://phabricator.wikimedia.org/T407942) [07:54:39] phuedx: Do I just need to switch on in the WikimediaDebug extension? [07:54:45] (03PS19) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [07:55:16] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet [07:55:24] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [07:55:28] Nvdtn19: Yes. k8s-mwdebug should be selected. Click the button from Off to On and refresh the page [07:55:34] (03PS2) 10Marostegui: mariadb: Productionize db1287 [puppet] - 10https://gerrit.wikimedia.org/r/1326125 (https://phabricator.wikimedia.org/T407942) [07:56:30] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1287 [puppet] - 10https://gerrit.wikimedia.org/r/1326125 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [07:56:58] I can see that Popups are enabled on viwikivoyage. There are errors in the browser console but they appear to come from another gadget (confirmed by loading viwikivoyage in a private browsing instance) [07:58:06] Is there anything else I need to do? I've turned the ON button and refresh but haven't seen any changes [07:58:54] Oh, popup works [07:59:04] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1275: Pool back [07:59:08] jmm@cumin2003 drain-node (PID 2387745) is awaiting input [07:59:36] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1057.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [07:59:39] RelatedArticles also working [08:00:23] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1057.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003" [08:00:23] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:00:25] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1057.eqiad.wmnet [08:00:28] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1201.eqiad.wmnet onto db1287.eqiad.wmnet [08:00:32] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1201: Depool db1201.eqiad.wmnet to then clone it to db1287.eqiad.wmnet - marostegui@cumin1003 [08:00:44] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12219926 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: `cloudvirt1057.eqiad.wmnet` - cloudvirt1057.eqiad.wmnet (... [08:00:57] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1201: Depool db1201.eqiad.wmnet to then clone it to db1287.eqiad.wmnet - marostegui@cumin1003 [08:01:01] Nvdtn19: If you're happy to proceed, then let me know and I'll continue with the deployment [08:01:30] phuedx, Nvdtn19: (LGTM as well) [08:01:37] Ok. You can continue [08:02:07] !log phuedx@deploy1003 nvdtn19, phuedx: Continuing with deployment [08:02:12] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet [08:02:41] jouncebot next [08:02:41] In 1 hour(s) and 57 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1000) [08:02:55] FIRING: [2x] SystemdUnitFailed: amd-smi-gpu-partition.service on ml-serve1015:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:03:19] FIRING: PuppetFailure: Puppet has failed on ml-serve1015:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:03:26] !log switch production to use dumps-nfs.w.o - T432212 [08:03:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:03:30] T432212: Migrate to dumps-nfs.w.o in production - https://phabricator.wikimedia.org/T432212 [08:03:34] OK. There are a total of 4 event stream config changes that have been lingering for a while. Since the next window is ~2 hours from now, I'm going to deploy those changes in one go [08:06:25] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [08:07:14] Actually, there are only 3 [08:08:36] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet [08:08:55] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet [08:09:03] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet [08:11:00] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet [08:11:46] Nvdtn19: About 50% of the way through the deployment [08:13:05] (03CR) 10Ryan Kemper: [C:03+1] elasticsearch: remove non-working freeze/unfreeze code [software/spicerack] - 10https://gerrit.wikimedia.org/r/1325955 (https://phabricator.wikimedia.org/T433306) (owner: 10Bking) [08:13:21] FIRING: [3x] ProbeDown: Service ganeti1031:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:15:25] !log phuedx@deploy1003 Finished scap sync-world: Backport for [[gerrit:1216721|viwikivoyage: enable relatedarticle and pop-up (T405724)]] (duration: 39m 12s) [08:15:30] T405724: Adding RelatedArticles and Popups extension for vi.wikivoyage - https://phabricator.wikimedia.org/T405724 [08:15:56] Nvdtn19, Dragoniez: The change has been deployed. Please look at vi.wikivoyage to make sure everything looks OK [08:16:13] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1279: Pool back [08:17:02] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet [08:17:07] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet [08:18:05] phuedx: the changes is ok [08:18:21] FIRING: [4x] ProbeDown: Service ganeti1031:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:18:42] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1035.eqiad.wmnet [08:19:24] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host ldap-rw1001.wikimedia.org with OS trixie [08:19:39] 06SRE, 06Infrastructure-Foundations, 07LDAP: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12219971 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host ldap-rw1001.wikimedia.org with OS trixie [08:19:43] (03CR) 10TrainBranchBot: [C:03+2] "Approved by phuedx@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324725 (https://phabricator.wikimedia.org/T429898) (owner: 10Phuedx) [08:19:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by phuedx@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313923 (https://phabricator.wikimedia.org/T415370) (owner: 10Phuedx) [08:19:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by phuedx@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325546 (https://phabricator.wikimedia.org/T434790) (owner: 10Bearloga) [08:19:56] phuedx: sorry i was afk for a moment [08:20:06] Dragoniez: np :) [08:20:06] no issues from my side, thanks for your help [08:20:27] Nvdtn19: congrats [08:20:47] (03Merged) 10jenkins-bot: EventStreamConfig: Mark product_metrics.web_base and .web_base_with_ip as Test Kitchen streams [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324725 (https://phabricator.wikimedia.org/T429898) (owner: 10Phuedx) [08:20:52] (03Merged) 10jenkins-bot: EventStreamConfig: Remove unused web_ui_scroll* streams [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313923 (https://phabricator.wikimedia.org/T415370) (owner: 10Phuedx) [08:20:56] (03Merged) 10jenkins-bot: EventStreamConfig: Remove Watchlist click stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325546 (https://phabricator.wikimedia.org/T434790) (owner: 10Bearloga) [08:21:15] !log phuedx@deploy1003 Started scap sync-world: Backport for [[gerrit:1324725|EventStreamConfig: Mark product_metrics.web_base and .web_base_with_ip as Test Kitchen streams (T429898 T430322)]], [[gerrit:1313923|EventStreamConfig: Remove unused web_ui_scroll* streams (T415370)]], [[gerrit:1325546|EventStreamConfig: Remove Watchlist click stream (T434790)]] [08:21:24] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12219988 (10fgiunchedi) >>! In T431682#12210883, @fgiunchedi wrote: >>>! In T431682#12209331, @VRiley-WMF wrote: >> Hey @fgiunchedi Yes, that would work for me! W... [08:21:27] T429898: [EventGate] Add configurable UA denylist - https://phabricator.wikimedia.org/T429898 [08:21:27] T430322: [EventGate] Record metrics for instruments and experiments - https://phabricator.wikimedia.org/T430322 [08:21:28] T415370: Migrate all instruments using mw.eventLog.submitInteraction to mw.testKitchen.getInstrument() - https://phabricator.wikimedia.org/T415370 [08:21:28] T434790: Decommission Watchlist CTR baseline data - https://phabricator.wikimedia.org/T434790 [08:22:55] jmm@cumin2003 drain-node (PID 2390253) is awaiting input [08:23:09] !log phuedx@deploy1003 bearloga, phuedx: Backport for [[gerrit:1324725|EventStreamConfig: Mark product_metrics.web_base and .web_base_with_ip as Test Kitchen streams (T429898 T430322)]], [[gerrit:1313923|EventStreamConfig: Remove unused web_ui_scroll* streams (T415370)]], [[gerrit:1325546|EventStreamConfig: Remove Watchlist click stream (T434790)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug [08:23:09] ). Changes can now be verified there. [08:23:14] Nvdtn19: BTW ext.gadget.DetectProtectionLevels is throwing some errors on Special:Version. You might wanna take a look [08:23:31] They're unrelated to this patch tho [08:26:20] action=streamconfigs looks OK on Meta. The new stream config settings are coming through for web_base and web_base_with_ip. The stream configs removed in the other commits aren't present [08:26:25] 06SRE, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 06Release-Engineering-Team: Periodic script failure due to "Fatal error: no version entry for `Please`" - https://phabricator.wikimedia.org/T434979#12220054 (10Aklapper) [08:26:27] !log phuedx@deploy1003 bearloga, phuedx: Continuing with deployment [08:26:51] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1035.eqiad.wmnet [08:27:27] Dragoniez: Oh. I will take a look later [08:30:52] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db2209: Maintenance needed [08:31:08] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: Maintenance needed [08:31:15] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2209 firmware upgrade after a crash - https://phabricator.wikimedia.org/T431952#12220063 (10ops-monitoring-bot) Completed depooling of db2209 by marostegui@cumin1003: Maintenance needed [08:31:48] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2209 firmware upgrade after a crash - https://phabricator.wikimedia.org/T431952#12220069 (10Marostegui) >>! In T431952#12217091, @Jhancock.wm wrote: > the full story is that I'm gonna update the firmware on these smaller parts, generate a new tech support report, send... [08:31:49] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2209.codfw.wmnet with reason: Maintenance [08:32:46] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-rw1001.wikimedia.org with reason: host reimage [08:33:29] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1281: Pool back [08:33:57] !log phuedx@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324725|EventStreamConfig: Mark product_metrics.web_base and .web_base_with_ip as Test Kitchen streams (T429898 T430322)]], [[gerrit:1313923|EventStreamConfig: Remove unused web_ui_scroll* streams (T415370)]], [[gerrit:1325546|EventStreamConfig: Remove Watchlist click stream (T434790)]] (duration: 12m 42s) [08:34:17] T429898: [EventGate] Add configurable UA denylist - https://phabricator.wikimedia.org/T429898 [08:34:17] T430322: [EventGate] Record metrics for instruments and experiments - https://phabricator.wikimedia.org/T430322 [08:34:17] T415370: Migrate all instruments using mw.eventLog.submitInteraction to mw.testKitchen.getInstrument() - https://phabricator.wikimedia.org/T415370 [08:34:18] T434790: Decommission Watchlist CTR baseline data - https://phabricator.wikimedia.org/T434790 [08:34:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1035.eqiad.wmnet [08:34:55] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1035.eqiad.wmnet [08:35:28] (03PS20) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [08:35:33] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1053.eqiad.wmnet [08:35:37] Logs look clean. I'll continue to monitor them [08:35:43] !log UTC morning backport window done [08:35:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:36:25] (03PS1) 10Marostegui: mariadb: Decommission db1152 [puppet] - 10https://gerrit.wikimedia.org/r/1326216 (https://phabricator.wikimedia.org/T434480) [08:36:39] FIRING: [6x] CirrusSearchTitleSuggestIndexTooOld: Some search indices that power autocomplete have not been updated recently - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - TODO - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchTitleSuggestIndexTooOld [08:36:46] !log marostegui@cumin1003 START - Cookbook sre.mysql.decommission [08:36:55] !log marostegui@cumin1003 START - Cookbook sre.hosts.decommission for hosts db1152.eqiad.wmnet [08:37:56] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1053.eqiad.wmnet [08:38:21] FIRING: [3x] ProbeDown: Service ganeti1035:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:38:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-rw1001.wikimedia.org with reason: host reimage [08:38:45] (03CR) 10Marostegui: [C:03+2] mariadb: Decommission db1152 [puppet] - 10https://gerrit.wikimedia.org/r/1326216 (https://phabricator.wikimedia.org/T434480) (owner: 10Marostegui) [08:39:11] (03PS21) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [08:40:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [08:42:35] !log marostegui@cumin1003 START - Cookbook sre.dns.netbox [08:43:07] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1053.eqiad.wmnet [08:43:09] (03CR) 10Tiziano Fogli: [C:03+2] grafana/image-renderer: install grafana-image-renderer [puppet] - 10https://gerrit.wikimedia.org/r/1324724 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [08:43:12] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1053.eqiad.wmnet [08:45:20] (03PS11) 10Majavah: P:wmcs::novaproxy: Add HAProxy in front of all requests [puppet] - 10https://gerrit.wikimedia.org/r/1308623 (https://phabricator.wikimedia.org/T429930) [08:45:20] (03PS4) 10Majavah: P:wmcs::novaproxy: Move rate limiting to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325428 (https://phabricator.wikimedia.org/T429930) [08:45:20] (03PS2) 10Majavah: P:wmcs::novaproxy: Move various headers to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325434 (https://phabricator.wikimedia.org/T429930) [08:45:21] (03PS2) 10Majavah: P:wmcs::novaproxy: Move static bans to HAProxy configuration [puppet] - 10https://gerrit.wikimedia.org/r/1325435 (https://phabricator.wikimedia.org/T429930) [08:46:02] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1054.eqiad.wmnet [08:46:18] !log marostegui@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1152.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" [08:46:24] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1201: Pool db1201.eqiad.wmnet in after cloning [08:46:35] !log marostegui@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1152.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" [08:46:35] !log marostegui@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:46:37] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1152.eqiad.wmnet [08:46:42] !log marostegui@cumin1003 Removing db1152 from zarcillo T434480 [08:46:47] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) [08:46:48] T434480: decommission db1152.eqiad.wmnet - https://phabricator.wikimedia.org/T434480 [08:46:50] 10ops-eqiad, 06DBA, 06DC-Ops: decommission db1152.eqiad.wmnet - https://phabricator.wikimedia.org/T434480#12220201 (10ops-monitoring-bot) db1152 has been deleted from zarcillo [08:46:52] 10ops-eqiad, 06DBA, 06DC-Ops: decommission db1152.eqiad.wmnet - https://phabricator.wikimedia.org/T434480#12220202 (10ops-monitoring-bot) db1152 has been decommissioned by Data Persistence [08:46:58] 10ops-eqiad, 06DBA, 06DC-Ops: decommission db1152.eqiad.wmnet - https://phabricator.wikimedia.org/T434480#12220204 (10ops-monitoring-bot) a:05Marostegui→03None This host is ready for DC-Ops to decommission [08:46:59] (03PS1) 10Blake: mediawiki: optionally use initContainers for sidecars. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325883 (https://phabricator.wikimedia.org/T417800) [08:47:30] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1175: Pool db1175.eqiad.wmnet in after cloning [08:47:54] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1152.eqiad.wmnet - https://phabricator.wikimedia.org/T434480#12220209 (10Marostegui) [08:48:16] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1151.eqiad.wmnet - https://phabricator.wikimedia.org/T434538#12220211 (10Marostegui) [08:48:28] 06SRE, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 06Release-Engineering-Team: Periodic script failure due to "Fatal error: no version entry for `Please`" - https://phabricator.wikimedia.org/T434979#12220215 (10Aklapper) [08:48:42] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1054.eqiad.wmnet [08:48:59] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 1 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [08:48:59] (03CR) 10Majavah: [C:03+2] P:wmcs::novaproxy: Add HAProxy in front of all requests [puppet] - 10https://gerrit.wikimedia.org/r/1308623 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [08:49:11] (03CR) 10Majavah: [C:03+2] P:wmcs::novaproxy: Move rate limiting to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325428 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [08:49:23] (03CR) 10Majavah: [C:03+2] P:wmcs::novaproxy: Move various headers to HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1325434 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [08:50:32] (03CR) 10Majavah: [C:03+2] P:wmcs::novaproxy: Move static bans to HAProxy configuration [puppet] - 10https://gerrit.wikimedia.org/r/1325435 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [08:50:59] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [08:51:39] (03PS1) 10Federico Ceratto: sre.mysql.decommission: keep decommission-hardware tag [cookbooks] - 10https://gerrit.wikimedia.org/r/1326218 (https://phabricator.wikimedia.org/T426613) [08:53:27] (03CR) 10Marostegui: [C:03+1] sre.mysql.decommission: keep decommission-hardware tag [cookbooks] - 10https://gerrit.wikimedia.org/r/1326218 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [08:54:06] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1054.eqiad.wmnet [08:54:23] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1054.eqiad.wmnet [08:55:30] (03PS2) 10Tiziano Fogli: grafana/image-renderer: adjust resource deps [puppet] - 10https://gerrit.wikimedia.org/r/1326219 (https://phabricator.wikimedia.org/T432970) [08:55:42] (03CR) 10Tiziano Fogli: [C:03+2] grafana/image-renderer: adjust resource deps [puppet] - 10https://gerrit.wikimedia.org/r/1326219 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [08:57:42] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1174: Pool db1174.eqiad.wmnet in after cloning [08:58:21] FIRING: [3x] ProbeDown: Service ganeti1054:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:01:20] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-rw1001.wikimedia.org with OS trixie [09:01:31] 06SRE, 06Infrastructure-Foundations, 07LDAP: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12220315 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host ldap-rw1001.wikimedia.org with OS trixie completed: - ldap-rw1... [09:06:33] 10SRE-swift-storage, 06Commons, 06DBA, 10media-backups, and 5 others: old file revisions missing of File:A_Warm_Shade_of_Ivory_-_Henry_Mancini_album_cover.jpg - https://phabricator.wikimedia.org/T428406#12220331 (10KylieTastic) I still keep coming across files with this issue, what is the status of this? [09:10:45] (03PS1) 10Tiziano Fogli: grafana/image-renderer: add renderer token [labs/private] - 10https://gerrit.wikimedia.org/r/1326222 (https://phabricator.wikimedia.org/T432970) [09:11:11] (03CR) 10Tiziano Fogli: [C:03+2] grafana/image-renderer: add renderer token [labs/private] - 10https://gerrit.wikimedia.org/r/1326222 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [09:11:14] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] grafana/image-renderer: add renderer token [labs/private] - 10https://gerrit.wikimedia.org/r/1326222 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [09:13:21] (03PS22) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [09:15:19] !log fceratto@cumin1003 START - Cookbook sre.ganeti.makevm for new host db1901.eqiad.wmnet [09:15:22] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [09:20:57] fceratto@cumin1003 makevm (PID 278683) is awaiting input [09:21:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:26:51] !log fceratto@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003" [09:26:55] !log fceratto@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003" [09:26:55] !log fceratto@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:26:56] !log fceratto@cumin1003 START - Cookbook sre.dns.wipe-cache db1901.eqiad.wmnet on all recursors [09:26:59] !log fceratto@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1901.eqiad.wmnet on all recursors [09:27:28] !log fceratto@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1901.eqiad.wmnet - fceratto@cumin1003" [09:27:32] !log fceratto@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1901.eqiad.wmnet - fceratto@cumin1003" [09:28:18] !log fceratto@cumin1003 START - Cookbook sre.hosts.reimage for host db1901.eqiad.wmnet with OS trixie [09:31:38] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1201: Pool db1201.eqiad.wmnet in after cloning [09:31:46] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1201.eqiad.wmnet onto db1287.eqiad.wmnet [09:32:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:32:44] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1175: Pool db1175.eqiad.wmnet in after cloning [09:33:45] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1175.eqiad.wmnet onto db1289.eqiad.wmnet [09:34:10] (03PS1) 10Marostegui: mariadb: Productionize db1286 [puppet] - 10https://gerrit.wikimedia.org/r/1326223 (https://phabricator.wikimedia.org/T407942) [09:35:30] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1286 [puppet] - 10https://gerrit.wikimedia.org/r/1326223 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [09:39:09] (03CR) 10Federico Ceratto: [C:03+2] sre.mysql.decommission: keep decommission-hardware tag [cookbooks] - 10https://gerrit.wikimedia.org/r/1326218 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [09:39:21] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1172.eqiad.wmnet onto db1286.eqiad.wmnet [09:39:25] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1172: Depool db1172.eqiad.wmnet to then clone it to db1286.eqiad.wmnet - marostegui@cumin1003 [09:40:20] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1172: Depool db1172.eqiad.wmnet to then clone it to db1286.eqiad.wmnet - marostegui@cumin1003 [09:40:52] (03PS1) 10Filippo Giunchedi: team-wmcs: alert only on enabled nova-compute [alerts] - 10https://gerrit.wikimedia.org/r/1326225 (https://phabricator.wikimedia.org/T424802) [09:42:45] (03Merged) 10jenkins-bot: sre.mysql.decommission: keep decommission-hardware tag [cookbooks] - 10https://gerrit.wikimedia.org/r/1326218 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [09:42:56] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1174: Pool db1174.eqiad.wmnet in after cloning [09:43:05] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1174.eqiad.wmnet onto db1288.eqiad.wmnet [09:43:21] marostegui@cumin1003 clone (PID 296216) is awaiting input [09:43:43] !log cgoubert@cumin2003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-codfw [09:43:54] (03CR) 10Majavah: [C:03+1] "LGTM, although now I wonder if we should have an alert for hosts that have been forgotten in disabled adminState for a long time" [alerts] - 10https://gerrit.wikimedia.org/r/1326225 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [09:44:14] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2002,2005-2006,2011-2015,2017-2018,2033-2037].codfw.wmnet [09:45:01] jouncebot: nowandnext [09:45:01] No deployments scheduled for the next 0 hour(s) and 14 minute(s) [09:45:01] In 0 hour(s) and 14 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1000) [09:46:19] (03CR) 10Clément Goubert: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) (owner: 10Clément Goubert) [09:46:20] (03PS1) 10Ladsgroup: Enable desktop/native lazy loading everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326227 (https://phabricator.wikimedia.org/T148047) [09:46:58] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326228 [09:47:44] (03PS1) 10Tiziano Fogli: grafana/image-renderer: adjust dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1326229 (https://phabricator.wikimedia.org/T432970) [09:47:46] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] grafana/image-renderer: adjust dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1326229 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [09:48:28] (03PS3) 10Tiziano Fogli: grafana/image-renderer: set up auth token [puppet] - 10https://gerrit.wikimedia.org/r/1326224 (https://phabricator.wikimedia.org/T432970) [09:48:29] (03CR) 10Tiziano Fogli: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1326224 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [09:48:38] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [09:48:42] !log fceratto@cumin1003 START - Cookbook sre.ganeti.makevm for new host db1902.eqiad.wmnet [09:48:44] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [09:48:57] !log fceratto@cumin1003 START - Cookbook sre.ganeti.makevm for new host db1903.eqiad.wmnet [09:50:04] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [09:51:14] (03PS1) 10Jgiannelos: prv: Enable parsoid rendering for 5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326232 [09:51:24] (03PS7) 10Clément Goubert: site.pp: Configure rdb-lock instances [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) [09:51:31] (03CR) 10Clément Goubert: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) (owner: 10Clément Goubert) [09:52:33] (03Abandoned) 10Filippo Giunchedi: openstack: export compute service info to prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1313958 (https://phabricator.wikimedia.org/T284747) (owner: 10Filippo Giunchedi) [09:52:59] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2002,2005-2006,2011-2015,2017-2018,2033-2037].codfw.wmnet [09:53:56] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [09:54:05] (03PS1) 10Tiziano Fogli: grafana/image-renderer: adjust dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1326233 (https://phabricator.wikimedia.org/T432970) [09:54:06] (03PS1) 10AikoChou: admin_ng: raise revertrisk namespace resource quota for rollout headroom [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) [09:54:08] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] grafana/image-renderer: adjust dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1326233 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [09:54:28] fceratto@cumin1003 makevm (PID 302488) is awaiting input [09:55:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326227 (https://phabricator.wikimedia.org/T148047) (owner: 10Ladsgroup) [09:56:39] (03Merged) 10jenkins-bot: Enable desktop/native lazy loading everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326227 (https://phabricator.wikimedia.org/T148047) (owner: 10Ladsgroup) [09:56:54] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1326227|Enable desktop/native lazy loading everywhere (T148047)]] [09:56:58] T148047: Enable native image lazy loading on desktop - https://phabricator.wikimedia.org/T148047 [09:56:58] (03PS2) 10Blake: rest-gateway: Use envoy-future in staging. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326221 (https://phabricator.wikimedia.org/T421418) [09:58:45] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1326227|Enable desktop/native lazy loading everywhere (T148047)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:59:21] fceratto@cumin1003 makevm (PID 302646) is awaiting input [09:59:48] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1055.eqiad.wmnet [09:59:48] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [09:59:50] (03PS1) 10Clément Goubert: Dummy pass for redis::lock::instance [labs/private] - 10https://gerrit.wikimedia.org/r/1326236 (https://phabricator.wikimedia.org/T427999) [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1000) [10:00:34] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [10:00:53] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [10:00:59] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2002,2005-2006,2011-2015,2017-2018,2033-2037].codfw.wmnet [10:01:03] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2002,2005-2006,2011-2015,2017-2018,2033-2037].codfw.wmnet [10:01:27] (03PS1) 10Tiziano Fogli: grafana/image-renderer: use only hex chars for renderer token [labs/private] - 10https://gerrit.wikimedia.org/r/1326237 (https://phabricator.wikimedia.org/T432970) [10:01:32] (03CR) 10Tiziano Fogli: [C:03+2] grafana/image-renderer: use only hex chars for renderer token [labs/private] - 10https://gerrit.wikimedia.org/r/1326237 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [10:01:34] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] grafana/image-renderer: use only hex chars for renderer token [labs/private] - 10https://gerrit.wikimedia.org/r/1326237 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [10:01:38] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2038-2039,2041-2042,2044,2046,2049-2051,2055-2060].codfw.wmnet [10:01:42] !log fceratto@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1903.eqiad.wmnet - fceratto@cumin1003" [10:01:53] (03CR) 10Majavah: [C:03+2] P:dumps: Remove unused variable [puppet] - 10https://gerrit.wikimedia.org/r/1325490 (https://phabricator.wikimedia.org/T417824) (owner: 10Majavah) [10:03:05] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/proton: apply [10:03:15] (03CR) 10Majavah: "friendly ping on this" [puppet] - 10https://gerrit.wikimedia.org/r/1320816 (owner: 10Majavah) [10:03:25] !log fceratto@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:03:26] !log fceratto@cumin1003 START - Cookbook sre.dns.wipe-cache db1902.eqiad.wmnet on all recursors [10:03:29] !log fceratto@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1902.eqiad.wmnet on all recursors [10:03:34] testing the new envoy version for proton in codfw [10:03:46] (03CR) 10Filippo Giunchedi: [C:03+2] "Absolutely yes, I'll followup on https://phabricator.wikimedia.org/T284747" [alerts] - 10https://gerrit.wikimedia.org/r/1326225 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [10:04:07] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1326227|Enable desktop/native lazy loading everywhere (T148047)]] (duration: 07m 13s) [10:04:08] (03CR) 10Majavah: [C:03+1] team-wmcs: add wikitech alerts [alerts] - 10https://gerrit.wikimedia.org/r/1312960 (https://phabricator.wikimedia.org/T362397) (owner: 10Filippo Giunchedi) [10:04:11] T148047: Enable native image lazy loading on desktop - https://phabricator.wikimedia.org/T148047 [10:04:28] jmm@cumin2003 drain-node (PID 2413760) is awaiting input [10:04:46] fceratto@cumin1003 makevm (PID 302646) is awaiting input [10:04:54] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: apply [10:04:57] (03CR) 10Filippo Giunchedi: [C:03+2] team-wmcs: add wikitech alerts [alerts] - 10https://gerrit.wikimedia.org/r/1312960 (https://phabricator.wikimedia.org/T362397) (owner: 10Filippo Giunchedi) [10:07:00] !log fceratto@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1902.eqiad.wmnet - fceratto@cumin1003" [10:07:31] !log fceratto@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1902.eqiad.wmnet - fceratto@cumin1003" [10:07:43] !log fceratto@cumin1003 START - Cookbook sre.hosts.reimage for host db1902.eqiad.wmnet with OS trixie [10:08:07] (03PS3) 10Tiziano Fogli: grafana/image-renderer: set up auth token [puppet] - 10https://gerrit.wikimedia.org/r/1326224 (https://phabricator.wikimedia.org/T432970) [10:08:07] (03CR) 10Tiziano Fogli: [C:03+2] "@adenisse@wikimedia.org I'm self-merging this since it's the same logic as in I6f3ab826fd14d803eeae4abd7f161c75e132bdbe. I've just added a" [puppet] - 10https://gerrit.wikimedia.org/r/1326224 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [10:08:21] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1903.eqiad.wmnet - fceratto@cumin1003" [10:08:21] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [10:08:24] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [10:08:34] !log fceratto@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1901.eqiad.wmnet with OS trixie [10:08:34] (03CR) 10Clément Goubert: [C:03+2] Dummy pass for redis::lock::instance [labs/private] - 10https://gerrit.wikimedia.org/r/1326236 (https://phabricator.wikimedia.org/T427999) (owner: 10Clément Goubert) [10:08:34] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host db1901.eqiad.wmnet [10:08:38] (03CR) 10Clément Goubert: [V:03+2 C:03+2] Dummy pass for redis::lock::instance [labs/private] - 10https://gerrit.wikimedia.org/r/1326236 (https://phabricator.wikimedia.org/T427999) (owner: 10Clément Goubert) [10:09:45] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1055.eqiad.wmnet [10:10:04] (03CR) 10Clément Goubert: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) (owner: 10Clément Goubert) [10:10:27] !log installing unzip security updates [10:10:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:10:37] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2038-2039,2041-2042,2044,2046,2049-2051,2055-2060].codfw.wmnet [10:10:53] !log fceratto@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:10:53] !log fceratto@cumin1003 START - Cookbook sre.dns.wipe-cache db1903.eqiad.wmnet on all recursors [10:10:57] !log fceratto@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1903.eqiad.wmnet on all recursors [10:11:13] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [10:11:43] PROBLEM - Host dse-k8s-etcd1001 is DOWN: PING CRITICAL - Packet loss = 100% [10:11:43] PROBLEM - Host kubestagemaster1005 is DOWN: PING CRITICAL - Packet loss = 100% [10:11:49] PROBLEM - Host aux-k8s-etcd1003 is DOWN: PING CRITICAL - Packet loss = 100% [10:13:03] !log mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 uls-preferences (T406724) [10:13:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:13:08] T406724: Clean up watchlist and user properties of users if they don't log in for certain time - https://phabricator.wikimedia.org/T406724 [10:15:12] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1055.eqiad.wmnet [10:15:17] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1055.eqiad.wmnet [10:15:25] RECOVERY - Host aux-k8s-etcd1003 is UP: PING OK - Packet loss = 0%, RTA = 0.88 ms [10:15:37] RECOVERY - Host dse-k8s-etcd1001 is UP: PING OK - Packet loss = 0%, RTA = 1.02 ms [10:15:52] (03PS4) 10Ladsgroup: Upgrade to trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1325499 (https://phabricator.wikimedia.org/T419815) [10:16:08] (03CR) 10Ladsgroup: Upgrade to trixie (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1325499 (https://phabricator.wikimedia.org/T419815) (owner: 10Ladsgroup) [10:16:19] RECOVERY - Host kubestagemaster1005 is UP: PING OK - Packet loss = 0%, RTA = 0.89 ms [10:16:39] fceratto@cumin1003 makevm (PID 302646) is awaiting input [10:16:57] FIRING: KubernetesCalicoDown: kubestagemaster1005.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-staging&var-instance=kubestagemaster1005.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:18:21] FIRING: [4x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:18:27] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:18:28] (03PS1) 10Majavah: P:wmcs::novaproxy: Fix X-Forwarded-Proto being set to http [puppet] - 10https://gerrit.wikimedia.org/r/1326240 (https://phabricator.wikimedia.org/T429930) [10:18:49] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2038-2039,2041-2042,2044,2046,2049-2051,2055-2060].codfw.wmnet [10:18:52] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2038-2039,2041-2042,2044,2046,2049-2051,2055-2060].codfw.wmnet [10:19:24] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2061-2062,2064-2065,2067-2077].codfw.wmnet [10:19:48] RESOLVED: [3x] KubernetesCalicoDown: kubestagemaster1005.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:20:01] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1056.eqiad.wmnet [10:21:23] (03PS1) 10JavierMonton: relative-trending: Alerts for missing data Adding 3 alerts to check if Kafka topics are receiving data or not. [alerts] - 10https://gerrit.wikimedia.org/r/1326241 (https://phabricator.wikimedia.org/T431536) [10:21:34] (03CR) 10Majavah: [C:03+2] P:wmcs::novaproxy: Fix X-Forwarded-Proto being set to http [puppet] - 10https://gerrit.wikimedia.org/r/1326240 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [10:23:46] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) (owner: 10Clément Goubert) [10:24:06] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [10:24:08] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=93) for new host db1903.eqiad.wmnet [10:24:19] !log fceratto@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1902.eqiad.wmnet with OS trixie [10:24:19] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host db1902.eqiad.wmnet [10:24:27] !log fceratto@cumin1003 START - Cookbook sre.ganeti.makevm for new host db1902.eqiad.wmnet [10:24:30] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [10:24:44] jmm@cumin2003 drain-node (PID 2416871) is awaiting input [10:25:25] !log fceratto@cumin1003 END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [10:25:29] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host db1902.eqiad.wmnet [10:25:34] (03CR) 10Muehlenhoff: [C:03+1] "Looks good, thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1320816 (owner: 10Majavah) [10:25:42] !log fceratto@cumin1003 START - Cookbook sre.ganeti.makevm for new host db2901.codfw.wmnet [10:25:44] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [10:26:19] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 17 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311141 (https://phabricator.wikimedia.org/T432311) (owner: 10NguoiDungKhongDinhDanh) [10:26:40] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1325499 (https://phabricator.wikimedia.org/T419815) (owner: 10Ladsgroup) [10:26:57] (03CR) 10Majavah: [C:03+2] P:idp::standalone: Remove profile [puppet] - 10https://gerrit.wikimedia.org/r/1320816 (owner: 10Majavah) [10:27:06] (03CR) 10Majavah: [C:03+2] hieradata: Migrate cloudidp-dev to Redis ticket storage [puppet] - 10https://gerrit.wikimedia.org/r/1320824 (https://phabricator.wikimedia.org/T433963) (owner: 10Majavah) [10:28:23] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2061-2062,2064-2065,2067-2077].codfw.wmnet [10:28:42] (03PS1) 10Marostegui: installserver: Update lists of DB hosts [puppet] - 10https://gerrit.wikimedia.org/r/1326242 [10:31:15] fceratto@cumin1003 makevm (PID 330935) is awaiting input [10:31:16] (03CR) 10Marostegui: [C:03+2] installserver: Update lists of DB hosts [puppet] - 10https://gerrit.wikimedia.org/r/1326242 (owner: 10Marostegui) [10:31:21] (03PS4) 10Blake: mediawiki: Refactor lamp.deployment into component containers. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313111 (https://phabricator.wikimedia.org/T417800) [10:32:29] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1056.eqiad.wmnet [10:36:43] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2061-2062,2064-2065,2067-2077].codfw.wmnet [10:36:47] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2061-2062,2064-2065,2067-2077].codfw.wmnet [10:36:52] (03PS1) 10JMeybohm: wikikube: Update eqiad to calico 3.30 [puppet] - 10https://gerrit.wikimedia.org/r/1326243 (https://phabricator.wikimedia.org/T427400) [10:37:18] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2078,2087-2095,2102-2106].codfw.wmnet [10:37:47] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1326243 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [10:38:32] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1056.eqiad.wmnet [10:38:37] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1056.eqiad.wmnet [10:38:43] PROBLEM - LDAP -writable server- on ldap-rw1001 is CRITICAL: Could not search/find objectclasses in dc=wikimedia,dc=org https://wikitech.wikimedia.org/wiki/LDAP%23Troubleshooting [10:38:55] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1057.eqiad.wmnet [10:39:01] (03PS5) 10Blake: mediawiki: Refactor lamp.deployment into component containers. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313111 (https://phabricator.wikimedia.org/T417800) [10:39:30] (03PS1) 10JMeybohm: wikikube-eqiad: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326244 (https://phabricator.wikimedia.org/T427400) [10:41:59] jmm@cumin2003 drain-node (PID 2421687) is awaiting input [10:42:03] (03CR) 10Clément Goubert: [C:03+1] mediawiki: Refactor lamp.deployment into component containers. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313111 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:43:21] FIRING: [3x] ProbeDown: Service ganeti1056:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:45:13] (03CR) 10Muehlenhoff: [C:03+2] proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326123 (owner: 10Muehlenhoff) [10:45:19] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/proton: apply [10:45:45] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2078,2087-2095,2102-2106].codfw.wmnet [10:46:07] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: apply [10:46:12] (03CR) 10JMeybohm: [C:03+2] wikikube: Update eqiad to calico 3.30 [puppet] - 10https://gerrit.wikimedia.org/r/1326243 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [10:46:21] !log jmm@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [10:47:10] !log jmm@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [10:48:18] (03CR) 10Kevin Bazira: [C:03+1] admin_ng: raise revertrisk namespace resource quota for rollout headroom [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) (owner: 10AikoChou) [10:48:56] jmm@cumin2003 drain-node (PID 2421687) is awaiting input [10:53:08] 06SRE, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 06Release-Engineering-Team: Periodic script failure due to "Fatal error: no version entry for `Please`" - https://phabricator.wikimedia.org/T434979#12220790 (10Aklapper) [10:54:11] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2078,2087-2095,2102-2106].codfw.wmnet [10:54:15] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2078,2087-2095,2102-2106].codfw.wmnet [10:54:46] (03PS8) 10Clément Goubert: site.pp: Configure rdb-lock instances [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) [10:54:46] (03PS2) 10Clément Goubert: external_services: Just in case, add redis-lock [puppet] - 10https://gerrit.wikimedia.org/r/1325875 (https://phabricator.wikimedia.org/T434188) [10:54:53] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2107-2115,2124-2129].codfw.wmnet [10:57:20] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1172: Pool db1172.eqiad.wmnet in after cloning [10:59:51] 10SRE-swift-storage, 06Commons, 06DBA, 10media-backups, and 5 others: old file revisions missing of File:A_Warm_Shade_of_Ivory_-_Henry_Mancini_album_cover.jpg - https://phabricator.wikimedia.org/T428406#12220806 (10TheDJ) >>! In T428406#12220331, @KylieTastic wrote: > I still keep coming across files with... [11:02:25] (03CR) 10Clément Goubert: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) (owner: 10Clément Goubert) [11:03:15] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2107-2115,2124-2129].codfw.wmnet [11:03:29] (03PS5) 10Majavah: P:idp: Remove support for memcached [puppet] - 10https://gerrit.wikimedia.org/r/1320817 (https://phabricator.wikimedia.org/T273950) [11:05:11] jouncebot: nowandnext [11:05:11] No deployments scheduled for the next 1 hour(s) and 54 minute(s) [11:05:11] In 1 hour(s) and 54 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1300) [11:05:32] (03PS1) 10Ladsgroup: ImagePage: move TOC element below file link [core] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1326249 (https://phabricator.wikimedia.org/T332644) [11:05:33] (03PS2) 10Tiziano Fogli: slothslos/report2drive: Add proxy configuration to service unit [puppet] - 10https://gerrit.wikimedia.org/r/1326247 (https://phabricator.wikimedia.org/T425795) [11:05:39] (03CR) 10Ladsgroup: [C:03+2] ImagePage: move TOC element below file link [core] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1326249 (https://phabricator.wikimedia.org/T332644) (owner: 10Ladsgroup) [11:05:46] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 3 NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1320817 (https://phabricator.wikimedia.org/T273950) (owner: 10Majavah) [11:06:30] !log fceratto@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db2901.codfw.wmnet - fceratto@cumin1003" [11:06:34] (03CR) 10Ladsgroup: [C:03+2] "Thanks!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325451 (https://phabricator.wikimedia.org/T348856) (owner: 10Ladsgroup) [11:06:55] (03CR) 10Majavah: [V:03+1 C:03+2] P:idp: Remove support for memcached [puppet] - 10https://gerrit.wikimedia.org/r/1320817 (https://phabricator.wikimedia.org/T273950) (owner: 10Majavah) [11:08:16] (03PS1) 10Majavah: apereo_cas: Remove unused type [puppet] - 10https://gerrit.wikimedia.org/r/1326250 [11:08:32] (03CR) 10JMeybohm: [C:03+2] wikikube-eqiad: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326244 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [11:09:23] (03Merged) 10jenkins-bot: thumbor: Use common statsd module [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325451 (https://phabricator.wikimedia.org/T348856) (owner: 10Ladsgroup) [11:09:35] fceratto@cumin1003 makevm (PID 330935) is awaiting input [11:10:55] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [11:11:05] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [11:11:16] !log jmm@deploy1003 helmfile [codfw] START helmfile.d/services/proton: apply [11:11:19] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12220833 (10VRiley-WMF) Perfect. I will be starting to take these down and moving them now. [11:11:20] 06SRE, 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#12220834 (10taavi) [11:11:28] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [11:11:34] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [11:11:37] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2107-2115,2124-2129].codfw.wmnet [11:11:40] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2107-2115,2124-2129].codfw.wmnet [11:12:05] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2130-2143].codfw.wmnet [11:12:12] !log fceratto@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db2901.codfw.wmnet - fceratto@cumin1003" [11:12:13] !log fceratto@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:12:13] !log fceratto@cumin1003 START - Cookbook sre.dns.wipe-cache db2901.codfw.wmnet on all recursors [11:12:17] !log fceratto@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db2901.codfw.wmnet on all recursors [11:12:48] !log fceratto@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db2901.codfw.wmnet - fceratto@cumin1003" [11:12:53] !log fceratto@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db2901.codfw.wmnet - fceratto@cumin1003" [11:13:07] !log fceratto@cumin1003 START - Cookbook sre.hosts.reimage for host db2901.codfw.wmnet with OS trixie [11:13:11] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [11:14:08] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [11:15:31] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [11:15:58] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1057.eqiad.wmnet [11:16:49] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [11:17:11] !log jmm@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: apply [11:17:12] (03Merged) 10jenkins-bot: ImagePage: move TOC element below file link [core] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1326249 (https://phabricator.wikimedia.org/T332644) (owner: 10Ladsgroup) [11:17:23] !log jmm@deploy1003 helmfile [eqiad] START helmfile.d/services/proton: apply [11:18:41] !log jmm@deploy1003 helmfile [eqiad] DONE helmfile.d/services/proton: apply [11:19:01] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1326249|ImagePage: move TOC element below file link (T332644)]] [11:19:05] T332644: Re-examine the section headings at Commons - https://phabricator.wikimedia.org/T332644 [11:19:51] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2130-2143].codfw.wmnet [11:20:29] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [11:21:21] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [11:21:40] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1057.eqiad.wmnet [11:21:46] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1057.eqiad.wmnet [11:22:04] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1326249|ImagePage: move TOC element below file link (T332644)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:22:31] !log jayme@deploy1003 helmfile [eqiad] START helmfile.d/admin 'sync'. [11:22:34] !log jayme@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'sync'. [11:22:47] !log jayme@deploy1003 helmfile [eqiad] START helmfile.d/admin 'sync'. [11:23:45] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [11:25:26] (03Abandoned) 10Ladsgroup: thumbor: pin image versions [deployment-charts] - 10https://gerrit.wikimedia.org/r/967466 (https://phabricator.wikimedia.org/T348856) (owner: 10Hnowlan) [11:26:10] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1058.eqiad.wmnet [11:28:12] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2130-2143].codfw.wmnet [11:28:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:28:15] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2130-2143].codfw.wmnet [11:28:43] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2144-2157].codfw.wmnet [11:28:58] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1326249|ImagePage: move TOC element below file link (T332644)]] (duration: 09m 56s) [11:29:02] T332644: Re-examine the section headings at Commons - https://phabricator.wikimedia.org/T332644 [11:29:13] 10ops-codfw, 06SRE, 06DC-Ops: ganeti2046 doesn't come back after reboot - https://phabricator.wikimedia.org/T434681#12220996 (10JMeybohm) [11:30:10] jmm@cumin2003 drain-node (PID 2432566) is awaiting input [11:31:12] 10SRE-swift-storage, 06Commons, 06DBA, 10media-backups, and 5 others: old file revisions missing of File:A_Warm_Shade_of_Ivory_-_Henry_Mancini_album_cover.jpg - https://phabricator.wikimedia.org/T428406#12221017 (10KylieTastic) >>! In T428406#12220806, @TheDJ wrote: >>>! In T428406#12220331, @KylieTastic w... [11:31:47] (03PS1) 10Marostegui: mariadb: Decommission db1153 [puppet] - 10https://gerrit.wikimedia.org/r/1326251 (https://phabricator.wikimedia.org/T434638) [11:32:32] !log marostegui@cumin1003 START - Cookbook sre.mysql.decommission [11:32:40] !log marostegui@cumin1003 START - Cookbook sre.hosts.decommission for hosts db1153.eqiad.wmnet [11:32:52] (03CR) 10Marostegui: [C:03+2] mariadb: Decommission db1153 [puppet] - 10https://gerrit.wikimedia.org/r/1326251 (https://phabricator.wikimedia.org/T434638) (owner: 10Marostegui) [11:33:15] RESOLVED: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:33:38] (03PS9) 10Clément Goubert: site.pp: Configure rdb-lock instances [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) [11:33:38] (03PS3) 10Clément Goubert: external_services: Just in case, add redis-lock [puppet] - 10https://gerrit.wikimedia.org/r/1325875 (https://phabricator.wikimedia.org/T434188) [11:33:45] (03CR) 10Clément Goubert: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) (owner: 10Clément Goubert) [11:34:01] (03PS1) 10Marostegui: wmnet: Update ms3-master CNAME [dns] - 10https://gerrit.wikimedia.org/r/1326252 (https://phabricator.wikimedia.org/T434638) [11:35:28] (03CR) 10Marostegui: [C:03+2] wmnet: Update ms3-master CNAME [dns] - 10https://gerrit.wikimedia.org/r/1326252 (https://phabricator.wikimedia.org/T434638) (owner: 10Marostegui) [11:35:32] !log marostegui@dns1004 START - running authdns-update [11:35:47] FIRING: HelmReleaseBadStatus: Helm release kube-system/calico on k8s@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s&var-namespace=kube-system - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:36:20] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2144-2157].codfw.wmnet [11:37:25] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1058.eqiad.wmnet [11:37:33] !log marostegui@dns1004 END - running authdns-update [11:38:05] !log marostegui@cumin1003 START - Cookbook sre.dns.netbox [11:38:48] !log jayme@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'sync'. [11:38:49] (03CR) 10Clément Goubert: "Haven't looked at CI output yet, but needs a `Chart.yaml` version bump" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325883 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [11:40:47] RESOLVED: HelmReleaseBadStatus: Helm release kube-system/calico on k8s@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s&var-namespace=kube-system - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:41:13] !log fceratto@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db2901.codfw.wmnet with OS trixie [11:41:14] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host db2901.codfw.wmnet [11:41:57] !log fceratto@cumin1003 START - Cookbook sre.ganeti.makevm for new host db1902.eqiad.wmnet [11:42:31] !log marostegui@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1153.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" [11:42:34] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1172: Pool db1172.eqiad.wmnet in after cloning [11:42:42] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1172.eqiad.wmnet onto db1286.eqiad.wmnet [11:42:54] !log marostegui@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1153.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" [11:42:54] !log marostegui@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:42:55] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1153.eqiad.wmnet [11:43:00] !log marostegui@cumin1003 Removing db1153 from zarcillo T434638 [11:43:05] T434638: decommission db1153.eqiad.wmnet - https://phabricator.wikimedia.org/T434638 [11:43:05] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) [11:43:11] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1153.eqiad.wmnet - https://phabricator.wikimedia.org/T434638#12221155 (10ops-monitoring-bot) db1153 has been deleted from zarcillo [11:43:15] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [11:43:16] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1153.eqiad.wmnet - https://phabricator.wikimedia.org/T434638#12221157 (10ops-monitoring-bot) db1153 has been decommissioned by Data Persistence [11:43:19] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1153.eqiad.wmnet - https://phabricator.wikimedia.org/T434638#12221158 (10ops-monitoring-bot) a:05Marostegui→03None This host is ready for DC-Ops to decommission [11:43:28] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1058.eqiad.wmnet [11:43:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1058.eqiad.wmnet [11:44:04] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2144-2157].codfw.wmnet [11:44:08] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2144-2157].codfw.wmnet [11:45:26] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2158-2171].codfw.wmnet [11:45:58] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 1 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [11:46:28] (03PS5) 10Kamila Součková: php8.5: initial release of 8.5 image stack [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1325907 (https://phabricator.wikimedia.org/T432988) [11:46:58] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [11:48:21] FIRING: [3x] ProbeDown: Service ganeti1058:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:48:34] (03PS2) 10Blake: mediawiki: optionally use initContainers for sidecars. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325883 (https://phabricator.wikimedia.org/T417800) [11:49:37] (03PS3) 10Blake: mediawiki: optionally use initContainers for sidecars. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325883 (https://phabricator.wikimedia.org/T417800) [11:50:08] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1027.eqiad.wmnet [11:51:11] !log updated calico to v3.30.7 on wikikube eqiad - T427400 [11:51:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:51:16] T427400: Update calico to 3.30 - https://phabricator.wikimedia.org/T427400 [11:52:06] (03PS23) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [11:52:58] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2158-2171].codfw.wmnet [11:53:28] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12221186 (10VRiley-WMF) Servers have been physically relocated. Will start powering on the units and updating netbox shortly. [11:54:13] fceratto@cumin1003 makevm (PID 386505) is awaiting input [11:56:07] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1027.eqiad.wmnet [11:57:31] (03PS24) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [11:58:00] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1326250 (owner: 10Majavah) [11:58:14] (03CR) 10Majavah: [C:03+2] apereo_cas: Remove unused type [puppet] - 10https://gerrit.wikimedia.org/r/1326250 (owner: 10Majavah) [11:58:36] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [11:58:39] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host db1902.eqiad.wmnet [11:59:47] Hi! I want to run a maintenance script to add wikidata support for a new language wiki. Let me know if this is a bad time, otherwise I will proceed. Thanks [12:01:05] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2158-2171].codfw.wmnet [12:01:09] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2158-2171].codfw.wmnet [12:01:26] seanleong-wmde: should be fine imo [12:01:36] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2172-2179,2184-2189].codfw.wmnet [12:01:53] fceratto@cumin1003 makevm (PID 399608) is awaiting input [12:02:25] !log fceratto@cumin1003 START - Cookbook sre.ganeti.makevm for new host db2901.codfw.wmnet [12:02:27] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [12:02:29] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1027.eqiad.wmnet [12:02:33] !log fceratto@cumin1003 END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [12:02:35] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1027.eqiad.wmnet [12:02:36] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host db2901.codfw.wmnet [12:02:48] claime thanks! [12:02:58] !log fceratto@cumin1003 START - Cookbook sre.ganeti.makevm for new host db2901.codfw.wmnet [12:03:00] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [12:03:11] FIRING: [2x] SystemdUnitFailed: amd-smi-gpu-partition.service on ml-serve1015:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:03:19] FIRING: PuppetFailure: Puppet has failed on ml-serve1015:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [12:03:21] FIRING: [3x] ProbeDown: Service ganeti1027:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:03:43] (03PS25) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [12:06:26] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:08:58] fceratto@cumin1003 makevm (PID 402168) is awaiting input [12:09:45] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1028.eqiad.wmnet [12:10:10] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 24 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325527 (https://phabricator.wikimedia.org/T433783) (owner: 10Michael Große) [12:10:31] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2172-2179,2184-2189].codfw.wmnet [12:13:05] !log seanleong-wmde@deploy1003 mwscript-k8s job started: foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https # T429955 [12:13:11] T429955: Add Wikidata support for bolwiki - https://phabricator.wikimedia.org/T429955 [12:14:15] jmm@cumin2003 drain-node (PID 2440280) is awaiting input [12:14:32] (03PS26) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [12:18:46] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2172-2179,2184-2189].codfw.wmnet [12:18:50] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2172-2179,2184-2189].codfw.wmnet [12:19:18] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2190-2203].codfw.wmnet [12:22:47] fceratto@cumin1003 makevm (PID 402168) is awaiting input [12:23:02] (03PS1) 10Marostegui: mariadb: Productionize db1284 [puppet] - 10https://gerrit.wikimedia.org/r/1326263 (https://phabricator.wikimedia.org/T407942) [12:23:20] (03PS1) 10Btullis: Fix the webrequest sampling kafkatee instance on an-test-coord1002 [puppet] - 10https://gerrit.wikimedia.org/r/1326264 (https://phabricator.wikimedia.org/T435076) [12:23:29] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1326264 (https://phabricator.wikimedia.org/T435076) (owner: 10Btullis) [12:24:17] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1284 [puppet] - 10https://gerrit.wikimedia.org/r/1326263 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [12:26:11] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet [12:27:00] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [12:27:01] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2190-2203].codfw.wmnet [12:27:07] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host db2901.codfw.wmnet [12:27:08] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326266 [12:27:59] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1170.eqiad.wmnet onto db1284.eqiad.wmnet [12:28:02] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1170: Depool db1170.eqiad.wmnet to then clone it to db1284.eqiad.wmnet - marostegui@cumin1003 [12:28:06] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ldap-maint1001.eqiad.wmnet [12:28:40] (03PS1) 10Samwilson: Allow ogg thumbs in png, jpg, and webp [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1326268 (https://phabricator.wikimedia.org/T430528) [12:30:28] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1170: Depool db1170.eqiad.wmnet to then clone it to db1284.eqiad.wmnet - marostegui@cumin1003 [12:31:56] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint1001.eqiad.wmnet [12:32:12] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet [12:32:18] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet [12:32:28] jouncebot: nowandnext [12:32:28] No deployments scheduled for the next 0 hour(s) and 27 minute(s) [12:32:28] In 0 hour(s) and 27 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1300) [12:33:01] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324832 (owner: 10Ladsgroup) [12:33:21] FIRING: [3x] ProbeDown: Service ganeti1028:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:34:36] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2190-2203].codfw.wmnet [12:34:40] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2190-2203].codfw.wmnet [12:35:07] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2204-2215,2242-2243].codfw.wmnet [12:35:30] (03Merged) 10jenkins-bot: Migrate $wgFlaggedRevsTags from flaggedrevs.php to ext-FlaggedRevs.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324832 (owner: 10Ladsgroup) [12:35:46] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1324832|Migrate $wgFlaggedRevsTags from flaggedrevs.php to ext-FlaggedRevs.php]] [12:35:52] (03PS1) 10Marostegui: db1275: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1326273 [12:36:31] (03CR) 10Marostegui: [C:03+2] db1275: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1326273 (owner: 10Marostegui) [12:36:54] FIRING: [6x] CirrusSearchTitleSuggestIndexTooOld: Some search indices that power autocomplete have not been updated recently - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - TODO - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchTitleSuggestIndexTooOld [12:37:51] !log Finished populateSitesTable for [bolwiki] ([T429955]) [12:37:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:37:55] T429955: Add Wikidata support for bolwiki - https://phabricator.wikimedia.org/T429955 [12:37:56] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1324832|Migrate $wgFlaggedRevsTags from flaggedrevs.php to ext-FlaggedRevs.php]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:38:39] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12221367 (10VRiley-WMF) 1049 CableID 5295 Row 14 Port 9 1051 CableID 5297 Row 15 Port 7 1054 CableID 5352 Row 16 Port 36 1055 CableID 5355 Row 18 Port 39 1056... [12:38:48] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: pki1002 became unresponsive causing several hosts to alert on failed puppet runs. - https://phabricator.wikimedia.org/T434268#12221369 (10elukey) > I think that along with the upgrade host maybe we can implement a failover mechanism where if the e... [12:38:59] (03CR) 10Majavah: [C:03+1] installserver: switch cloudvirt to efi standard recipes [puppet] - 10https://gerrit.wikimedia.org/r/1325459 (https://phabricator.wikimedia.org/T431682) (owner: 10Filippo Giunchedi) [12:40:02] !log ladsgroup@deploy1003 Rolling back deployment [12:40:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:41:21] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1037.eqiad.wmnet [12:41:36] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ldap-maint2001.codfw.wmnet [12:41:43] (03PS1) 10Ladsgroup: Revert "Migrate $wgFlaggedRevsTags from flaggedrevs.php to ext-FlaggedRevs.php" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326275 [12:41:48] (03CR) 10Ladsgroup: [C:03+2] Revert "Migrate $wgFlaggedRevsTags from flaggedrevs.php to ext-FlaggedRevs.php" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326275 (owner: 10Ladsgroup) [12:41:57] (03CR) 10Ladsgroup: "https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-deploy-1-7.0.0-1-2026.08.17?id=Jzq8D6ABMPCwDrNArf5I" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324832 (owner: 10Ladsgroup) [12:43:12] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2204-2215,2242-2243].codfw.wmnet [12:43:45] (03Merged) 10jenkins-bot: Revert "Migrate $wgFlaggedRevsTags from flaggedrevs.php to ext-FlaggedRevs.php" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326275 (owner: 10Ladsgroup) [12:43:46] (03CR) 10Btullis: [C:03+2] Fix the webrequest sampling kafkatee instance on an-test-coord1002 [puppet] - 10https://gerrit.wikimedia.org/r/1326264 (https://phabricator.wikimedia.org/T435076) (owner: 10Btullis) [12:44:26] jmm@cumin2003 drain-node (PID 2446708) is awaiting input [12:45:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint2001.codfw.wmnet [12:45:37] Hi, update: finished running the maintenance script to add Wikidata support. Thank you. [12:49:30] (03CR) 10Ladsgroup: [C:03+2] "Deploying" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1325499 (https://phabricator.wikimedia.org/T419815) (owner: 10Ladsgroup) [12:49:49] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324832|Migrate $wgFlaggedRevsTags from flaggedrevs.php to ext-FlaggedRevs.php]] (duration: 14m 02s) [12:49:57] (03CR) 10Filippo Giunchedi: [C:03+2] installserver: switch cloudvirt to efi standard recipes [puppet] - 10https://gerrit.wikimedia.org/r/1325459 (https://phabricator.wikimedia.org/T431682) (owner: 10Filippo Giunchedi) [12:50:24] 06SRE, 06Infrastructure-Foundations, 10netops: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12221432 (10Jgreen) @cmooney this happens to be a FR maintenance week, is it feasible to plan it sometime tomorrow-Friday? [12:50:56] 06SRE, 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12221433 (10Jgreen) [12:51:19] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2204-2215,2242-2243].codfw.wmnet [12:51:22] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2204-2215,2242-2243].codfw.wmnet [12:51:53] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2248-2261].codfw.wmnet [12:56:39] (03Merged) 10jenkins-bot: Upgrade to trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1325499 (https://phabricator.wikimedia.org/T419815) (owner: 10Ladsgroup) [12:57:25] FIRING: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:57:44] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12221469 (10VRiley-WMF) Updated Netbox device page with new rack location Deleted the cables connected to the device's interfaces Deleted *all* Netbox interfaces f... [12:57:51] (03CR) 10Kamila Součková: "I created https://phabricator.wikimedia.org/T435072 to track the APCu tuning." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1325907 (https://phabricator.wikimedia.org/T432988) (owner: 10Kamila Součková) [12:58:50] (03PS27) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: It is that lovely time of the day again! You are hereby commanded to deploy UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1300). [13:00:05] Superpes, _Gerges, and Hide_on_rosie: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:23] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1037.eqiad.wmnet [13:01:21] o/ [13:01:26] Hi. [13:02:25] RESOLVED: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:02:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:13] (03PS1) 10Filippo Giunchedi: icinga: remove wikitech-static monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1326279 (https://phabricator.wikimedia.org/T362397) [13:04:40] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2248-2261].codfw.wmnet [13:04:54] I can deploy, and start with Hide_on_rosie I guess (since the others aren’t here yet) [13:05:07] * Lucas_WMDE tries to remember where I’ve seen Hide_on_rosie’s name before 😅 [13:05:53] I'm Veritas Sapientiae now, Hide on Rosé is my old wikiname, and it's still my Phabricator and IRC name [13:06:15] I have WikimediaDebug installed so I'm ready [13:06:20] (03PS2) 10Lucas Werkmeister (WMDE): viwiki: Set `noindex,nofollow` for User and User talk [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311141 (https://phabricator.wikimedia.org/T432311) (owner: 10NguoiDungKhongDinhDanh) [13:06:25] ok, I just added the wiki to the commit message [13:06:26] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [13:06:55] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311141 (https://phabricator.wikimedia.org/T432311) (owner: 10NguoiDungKhongDinhDanh) [13:08:04] (03Merged) 10jenkins-bot: viwiki: Set `noindex,nofollow` for User and User talk [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311141 (https://phabricator.wikimedia.org/T432311) (owner: 10NguoiDungKhongDinhDanh) [13:08:19] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1311141|viwiki: Set `noindex,nofollow` for User and User talk (T432311)]] [13:08:20] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1037.eqiad.wmnet [13:08:21] FIRING: [3x] ProbeDown: Service ganeti1037:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:08:24] T432311: Set noindex for viwiki's User and User talk namespaces - https://phabricator.wikimedia.org/T432311 [13:08:25] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1037.eqiad.wmnet [13:08:48] hmm, the patch got +2ed and merged, why didn’t wikibugs report it? [13:08:59] ok I guess my IRC client was lagging behind o_O [13:10:05] •stashbot> https://stashbot.toolforge.org/ [13:10:05] T432311: Set noindex for viwiki's User and User talk namespaces - https://phabricator.wikimedia.org/T432311 [13:10:09] !log lucaswerkmeister-wmde@deploy1003 ndkdd, lucaswerkmeister-wmde: Backport for [[gerrit:1311141|viwiki: Set `noindex,nofollow` for User and User talk (T432311)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:10:15] Hide_on_rosie: please test :) [13:10:44] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1049] - vriley@cumin1003" [13:10:50] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1049] - vriley@cumin1003" [13:10:50] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:11:47] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1049 [13:12:07] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1049 [13:12:49] Go to random user page and talk page information: https://vi.wikipedia.org/w/index.php?title=Th%C3%A0nh_vi%C3%AAn:ChopinChemist&action=info&uselang=en and https://vi.wikipedia.org/w/index.php?title=Th%E1%BA%A3o_lu%E1%BA%ADn_Th%C3%A0nh_vi%C3%AAn:ChopinChemist&action=info&uselang=en shows: "Indexing by robots Disallowed". It's OK now. [13:12:56] (03CR) 10Ladsgroup: "Hi, I'm taking over while Cormac is out. I wonder if we can piggy back on webrequest sampled live that is being done via benthos. Speciall" [puppet] - 10https://gerrit.wikimedia.org/r/1324778 (https://phabricator.wikimedia.org/T431597) (owner: 10Cparle) [13:13:08] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1049.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:14:33] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2248-2261].codfw.wmnet [13:14:37] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2248-2261].codfw.wmnet [13:14:40] yay [13:14:46] !log lucaswerkmeister-wmde@deploy1003 ndkdd, lucaswerkmeister-wmde: Continuing with deployment [13:15:02] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2262-2275].codfw.wmnet [13:15:32] (03PS1) 10Filippo Giunchedi: team-wmcs: page on nova-reported cloudvirt status [alerts] - 10https://gerrit.wikimedia.org/r/1326288 (https://phabricator.wikimedia.org/T424802) [13:17:28] (03CR) 10Bking: [C:03+2] elasticsearch: remove non-working freeze/unfreeze code [software/spicerack] - 10https://gerrit.wikimedia.org/r/1325955 (https://phabricator.wikimedia.org/T433306) (owner: 10Bking) [13:18:58] (03PS1) 10Ladsgroup: thumbor: Upgrade to trixie [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326289 (https://phabricator.wikimedia.org/T419815) [13:19:29] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1311141|viwiki: Set `noindex,nofollow` for User and User talk (T432311)]] (duration: 11m 11s) [13:19:34] T432311: Set noindex for viwiki's User and User talk namespaces - https://phabricator.wikimedia.org/T432311 [13:19:53] Hide_on_rosie: should be done, thank you! [13:20:16] still no sign of the other two people who wanted to deploy patches, let’s wait a bit more [13:21:19] Thanks [13:21:24] (03PS1) 10Aqu: hadoop/spark35: Disable SQL options redaction regex [puppet] - 10https://gerrit.wikimedia.org/r/1326290 (https://phabricator.wikimedia.org/T405360) [13:21:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:21:41] (03CR) 10Aqu: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1326290 (https://phabricator.wikimedia.org/T405360) (owner: 10Aqu) [13:21:42] !log manually done sudo -i docker-registryctl --debug delete-tags 'docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-3.3.0-py3.11-2026-08-17-*' to remove incorrect tags [13:21:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:21:53] (03CR) 10Slyngshede: Varnish: Reject non-thumb requests to thumbs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [13:23:26] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1045.eqiad.wmnet [13:23:56] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1049.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:26:08] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1170: Pool db1170.eqiad.wmnet in after cloning [13:26:10] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1049.eqiad.wmnet with OS trixie [13:26:28] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12221666 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1049.eqiad.wmnet with OS trixie [13:26:30] jmm@cumin2003 drain-node (PID 2456838) is awaiting input [13:26:31] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1049.eqiad.wmnet with OS trixie [13:26:54] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12221673 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1049.eqiad.wmnet with OS trixie executed with er... [13:27:13] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2262-2275].codfw.wmnet [13:30:00] (03PS1) 10AOkoth: phabricator: fix php version in restart script [puppet] - 10https://gerrit.wikimedia.org/r/1326292 (https://phabricator.wikimedia.org/T433988) [13:30:13] (03PS2) 10Aqu: hadoop/spark35: Disable SQL options redaction regex [puppet] - 10https://gerrit.wikimedia.org/r/1326290 (https://phabricator.wikimedia.org/T405360) [13:30:47] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet [13:31:53] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [13:32:34] !log UTC afternoon backport+config window done [13:32:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:33:03] (03PS1) 10Filippo Giunchedi: team-wmcs: gate NeutronAgentDown based on host maintenance status [alerts] - 10https://gerrit.wikimedia.org/r/1326294 (https://phabricator.wikimedia.org/T424802) [13:33:41] (03PS1) 10Tiziano Fogli: grafana/image-renderer: remove GIR plugin [debs/grafana-plugins] - 10https://gerrit.wikimedia.org/r/1326295 (https://phabricator.wikimedia.org/T432970) [13:34:28] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2262-2275].codfw.wmnet [13:34:32] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2262-2275].codfw.wmnet [13:34:46] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1049.eqiad.wmnet with OS trixie [13:34:59] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2276-2289].codfw.wmnet [13:35:00] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12221763 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1049.eqiad.wmnet with OS trixie [13:36:07] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet [13:36:12] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet [13:36:22] (03CR) 10Muehlenhoff: [C:03+1] "Fantastic!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326289 (https://phabricator.wikimedia.org/T419815) (owner: 10Ladsgroup) [13:36:34] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet [13:36:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [13:39:39] jmm@cumin2003 drain-node (PID 2458491) is awaiting input [13:41:32] (03CR) 10Phuedx: [C:03+2] eventgate-analytics-external: Enable testKitchen transform [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324728 (https://phabricator.wikimedia.org/T429898) (owner: 10Phuedx) [13:42:30] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Degraded RAID on an-worker1191 - https://phabricator.wikimedia.org/T431828#12221791 (10BTullis) [13:42:31] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12221792 (10BTullis) [13:42:47] (03PS1) 10Filippo Giunchedi: team-wmcs: escalate NeutronAgentDown warning -> critical [alerts] - 10https://gerrit.wikimedia.org/r/1326296 (https://phabricator.wikimedia.org/T424802) [13:43:16] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2276-2289].codfw.wmnet [13:43:58] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet [13:44:00] (03Merged) 10jenkins-bot: eventgate-analytics-external: Enable testKitchen transform [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324728 (https://phabricator.wikimedia.org/T429898) (owner: 10Phuedx) [13:44:25] !log phuedx@deploy1003 helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply [13:46:17] !log phuedx@deploy1003 helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply [13:47:01] (03CR) 10Ladsgroup: [C:03+2] thumbor: Upgrade to trixie [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326289 (https://phabricator.wikimedia.org/T419815) (owner: 10Ladsgroup) [13:47:04] (03PS1) 10Filippo Giunchedi: team-wmcs: add alert on cloudvirt in maintenance for multiple days [alerts] - 10https://gerrit.wikimedia.org/r/1326297 (https://phabricator.wikimedia.org/T284747) [13:47:07] RECOVERY - Host ps1-b7-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.11 ms [13:47:11] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326298 [13:48:01] !log sukhe@cumin1003 START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy [13:48:10] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Degraded RAID on an-worker1191 - https://phabricator.wikimedia.org/T431828#12221825 (10BTullis) Hi @VRiley-WMF - that's no problem. These Hadoop workwers and their multiple RAID0 volumes are a real pain and I will be glad to see the back of them... [13:48:42] !log sukhe@cumin1003 START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir and A:ncredir [13:49:21] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet [13:49:30] (03Merged) 10jenkins-bot: thumbor: Upgrade to trixie [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326289 (https://phabricator.wikimedia.org/T419815) (owner: 10Ladsgroup) [13:49:40] !log vriley@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1049.eqiad.wmnet with reason: host reimage [13:49:47] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet [13:49:51] (03CR) 10Dbrant: [C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326298 (owner: 10PipelineBot) [13:49:56] !log phuedx@deploy1003 helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [13:50:11] (03PS2) 10JavierMonton: namespaces: webrequest-page-view [puppet] - 10https://gerrit.wikimedia.org/r/1321974 (https://phabricator.wikimedia.org/T433962) [13:50:16] (03CR) 10Klausman: [C:03+2] namespaces: webrequest-page-view [puppet] - 10https://gerrit.wikimedia.org/r/1321974 (https://phabricator.wikimedia.org/T433962) (owner: 10JavierMonton) [13:50:22] (03CR) 10Klausman: [V:03+2 C:03+2] namespaces: webrequest-page-view [puppet] - 10https://gerrit.wikimedia.org/r/1321974 (https://phabricator.wikimedia.org/T433962) (owner: 10JavierMonton) [13:50:29] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2276-2289].codfw.wmnet [13:50:32] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2276-2289].codfw.wmnet [13:50:53] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1047.eqiad.wmnet [13:51:01] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2290-2303].codfw.wmnet [13:51:08] !log phuedx@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [13:52:15] (03Merged) 10jenkins-bot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326298 (owner: 10PipelineBot) [13:52:50] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1049.eqiad.wmnet with reason: host reimage [13:53:37] !log phuedx@deploy1003 helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply [13:54:21] !log phuedx@deploy1003 helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply [13:54:32] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [13:54:54] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [13:55:00] !log dbrant@deploy1003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [13:55:58] !log dbrant@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [13:56:01] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1179.mgmt:22 - https://phabricator.wikimedia.org/T435017#12221852 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [13:56:16] (03PS10) 10Clément Goubert: site.pp: Configure rdb-lock instances [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) [13:56:17] (03PS4) 10Clément Goubert: external_services: Just in case, add redis-lock [puppet] - 10https://gerrit.wikimedia.org/r/1325875 (https://phabricator.wikimedia.org/T434188) [13:56:26] !log dbrant@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [13:56:34] !log sudo -i cookbook sre.cdn.roll-upgrade-ats --query 'A:cp-ulsfo or A:cp-drmrs' --task-id T434620 --reason '9.2.15 upgrade' [13:56:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:56:38] T434620: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Mentimeter - https://phabricator.wikimedia.org/T434620 [13:56:39] !log cdobbins@cumin1003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-ulsfo or A:cp-drmrs and A:cp - 9.2.15 upgrade (T434620) [13:56:55] !log dbrant@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [13:57:06] !log dbrant@deploy1003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [13:57:21] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-presto1018.mgmt:22 - https://phabricator.wikimedia.org/T435016#12221877 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [13:57:21] jmm@cumin2003 drain-node (PID 2465118) is awaiting input [13:57:37] !log dbrant@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [13:57:58] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for clouddb1016.mgmt:22 - https://phabricator.wikimedia.org/T435015#12221883 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [13:58:28] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for backup1013.mgmt:22 - https://phabricator.wikimedia.org/T435014#12221887 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [13:58:50] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for ganeti1043.mgmt:22 - https://phabricator.wikimedia.org/T435013#12221891 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [13:59:06] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [13:59:12] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for ganeti1042.mgmt:22 - https://phabricator.wikimedia.org/T435012#12221895 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [13:59:29] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for clouddumps1001.mgmt:22 - https://phabricator.wikimedia.org/T435011#12221899 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:00:01] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS bookworm [14:00:01] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for ganeti1044.mgmt:22 - https://phabricator.wikimedia.org/T435010#12221906 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:00:04] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1182.eqiad.wmnet with OS bookworm [14:00:21] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1218.mgmt:22 - https://phabricator.wikimedia.org/T435009#12221913 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:00:52] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for cp1106.mgmt:22 - https://phabricator.wikimedia.org/T435008#12221919 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:01:17] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-presto1017.mgmt:22 - https://phabricator.wikimedia.org/T435007#12221926 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:01:49] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [14:02:43] btullis@cumin1003 reimage (PID 470211) is awaiting input [14:02:54] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1188.eqiad.wmnet with OS bookworm [14:02:58] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1189.eqiad.wmnet with OS bookworm [14:03:19] (03Abandoned) 10Dbrant: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322920 (owner: 10PipelineBot) [14:03:25] (03Abandoned) 10Dbrant: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320934 (owner: 10PipelineBot) [14:03:39] (03PS1) 10Atsuko: Updating Airflow 3 to 3.3.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326300 [14:04:16] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2290-2303].codfw.wmnet [14:04:21] (03PS2) 10Atsuko: Updating Airflow 3 to 3.3.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326300 [14:04:25] !log klausman@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [14:04:44] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [14:04:48] (03CR) 10Ssingh: "Thanks for the review, Scott! Please merge this whenever you can, or let me know how to and I am happy to do it." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325949 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [14:05:20] (03CR) 10Btullis: [C:03+1] stat hosts: update partitioning to retain /srv data [puppet] - 10https://gerrit.wikimedia.org/r/1325921 (https://phabricator.wikimedia.org/T434562) (owner: 10Bking) [14:06:15] (03CR) 10Klausman: [C:03+1] Updating Airflow 3 to 3.3.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326300 (owner: 10Atsuko) [14:06:28] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326301 [14:06:41] !log klausman@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [14:07:04] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326302 [14:07:13] (03CR) 10Btullis: [C:03+1] Updating Airflow 3 to 3.3.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326300 (owner: 10Atsuko) [14:07:51] (03CR) 10Atsuko: [C:03+2] Updating Airflow 3 to 3.3.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326300 (owner: 10Atsuko) [14:07:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [14:09:00] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1051] - vriley@cumin1003" [14:09:05] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1051] - vriley@cumin1003" [14:09:05] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:09:55] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [14:10:02] (03Merged) 10jenkins-bot: Updating Airflow 3 to 3.3.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326300 (owner: 10Atsuko) [14:11:22] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1170: Pool db1170.eqiad.wmnet in after cloning [14:11:31] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1170.eqiad.wmnet onto db1284.eqiad.wmnet [14:11:31] (03PS1) 10Ssingh: dns1006/dns2006: switch to new pdns-rec [puppet] - 10https://gerrit.wikimedia.org/r/1326303 (https://phabricator.wikimedia.org/T401832) [14:11:37] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1051 [14:11:53] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1051 [14:12:04] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [14:12:04] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2290-2303].codfw.wmnet [14:12:05] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1049.eqiad.wmnet with OS trixie [14:12:08] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2290-2303].codfw.wmnet [14:12:19] (03PS1) 10Bking: cirrussearch: Remove references to OpenSearch 1/relocate hiera values [puppet] - 10https://gerrit.wikimedia.org/r/1326304 (https://phabricator.wikimedia.org/T430163) [14:12:19] (03Abandoned) 10Ssingh: dns1006/dns2006: switch to new pdns-rec [puppet] - 10https://gerrit.wikimedia.org/r/1326303 (https://phabricator.wikimedia.org/T401832) (owner: 10Ssingh) [14:12:22] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12221981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1049.eqiad.wmnet with OS trixie completed: - clo... [14:12:32] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2304-2317].codfw.wmnet [14:12:41] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1326304 (https://phabricator.wikimedia.org/T430163) (owner: 10Bking) [14:12:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [14:13:06] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:13:55] (03CR) 10Atsuko: [C:03+1] cirrussearch: Remove references to OpenSearch 1/relocate hiera values [puppet] - 10https://gerrit.wikimedia.org/r/1326304 (https://phabricator.wikimedia.org/T430163) (owner: 10Bking) [14:14:55] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1143.eqiad.wmnet with reason: host reimage [14:15:11] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for cp1107.mgmt:22 - https://phabricator.wikimedia.org/T435006#12221993 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:15:15] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1182.eqiad.wmnet with reason: host reimage [14:15:33] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1219.mgmt:22 - https://phabricator.wikimedia.org/T435005#12221997 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:15:37] (03PS1) 10Ssingh: dns[12]006|dns4004: use new pdns-rec config [puppet] - 10https://gerrit.wikimedia.org/r/1326305 (https://phabricator.wikimedia.org/T401832) [14:15:53] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for cirrussearch1086.mgmt:22 - https://phabricator.wikimedia.org/T435004#12222002 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:16:29] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for wdqs1024.mgmt:22 - https://phabricator.wikimedia.org/T435003#12222007 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:16:50] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for phab1005.mgmt:22 - https://phabricator.wikimedia.org/T435002#12222013 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:17:11] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1150.mgmt:22 - https://phabricator.wikimedia.org/T435001#12222017 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:17:29] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1189.eqiad.wmnet with reason: host reimage [14:17:31] (03PS1) 10Clément Goubert: Add placeholder $wmgRedisLockPassword [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326306 (https://phabricator.wikimedia.org/T366938) [14:17:41] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for mc1060.mgmt:22 - https://phabricator.wikimedia.org/T435000#12222032 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:17:48] (03CR) 10Ssingh: [C:03+2] dns[12]006|dns4004: use new pdns-rec config [puppet] - 10https://gerrit.wikimedia.org/r/1326305 (https://phabricator.wikimedia.org/T401832) (owner: 10Ssingh) [14:17:59] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for lvs1018.mgmt:22 - https://phabricator.wikimedia.org/T434999#12222041 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:18:09] (03CR) 10Ladsgroup: [C:03+1] Add placeholder $wmgRedisLockPassword [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326306 (https://phabricator.wikimedia.org/T366938) (owner: 10Clément Goubert) [14:18:20] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for logging-hd1002.mgmt:22 - https://phabricator.wikimedia.org/T434998#12222045 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:18:28] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:18:42] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for cirrussearch1085.mgmt:22 - https://phabricator.wikimedia.org/T434997#12222051 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:19:03] !log sukhe@cumin1003 START - Cookbook sre.hosts.reimage for host dns1006.wikimedia.org with OS trixie [14:19:04] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for vrts1003.mgmt:22 - https://phabricator.wikimedia.org/T434996#12222055 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:19:06] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1143.eqiad.wmnet with reason: host reimage [14:19:14] !log sukhe@cumin1003 START - Cookbook sre.hosts.reimage for host dns2006.wikimedia.org with OS trixie [14:19:19] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for ms-fe1018.mgmt:22 - https://phabricator.wikimedia.org/T434993#12222060 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:19:29] !log sukhe@cumin1003 START - Cookbook sre.hosts.reimage for host dns4004.wikimedia.org with OS trixie [14:19:37] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for dse-k8s-wdqs1002.mgmt:22 - https://phabricator.wikimedia.org/T434995#12222065 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:19:58] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for wikikube-ctrl1002.mgmt:22 - https://phabricator.wikimedia.org/T434994#12222070 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF There was a managment switch failure in B7 that generated this issue. The switch has been swapped. [14:20:15] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2304-2317].codfw.wmnet [14:20:32] jouncebot: nowandnext [14:20:32] No deployments scheduled for the next 0 hour(s) and 9 minute(s) [14:20:33] In 0 hour(s) and 9 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1430) [14:21:18] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [14:21:52] !log sukhe@cumin1003 START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet [14:22:10] !log sukhe@cumin1003 START - Cookbook sre.hosts.reboot-single for host acmechief-test2001.codfw.wmnet [14:22:14] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [14:22:20] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:22:31] PROBLEM - Host ntp-c.anycast.wmnet is DOWN: PING CRITICAL - Packet loss = 100% [14:22:40] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1182.eqiad.wmnet with reason: host reimage [14:22:41] oh interesting yes, good point [14:22:54] that's fine, there is sufficient redundancy in the NTP pool but yeah, noted for future [14:23:02] (03PS1) 10Lucas Werkmeister (WMDE): wikidata-query-gui: update query-next staging custom-config.json [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326307 [14:23:17] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] "⇒ I9ee7dbd26e" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 (owner: 10Lucas Werkmeister (WMDE)) [14:23:33] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [14:24:20] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1051.eqiad.wmnet with OS trixie [14:24:38] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12222083 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1051.eqiad.wmnet with OS trixie [14:25:24] !log Deploying wmgRedisLockPassword - T366938 T427999 [14:25:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:25:35] T366938: Reduce relying on database locks - https://phabricator.wikimedia.org/T366938 [14:25:35] T427999: Redis solution for LockManager - https://phabricator.wikimedia.org/T427999 [14:25:36] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test1001.eqiad.wmnet [14:26:08] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test2001.codfw.wmnet [14:26:10] FIRING: [10x] BFDdown: BFD session down between cr1-codfw and 208.80.153.107 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:26:46] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1189.eqiad.wmnet with reason: host reimage [14:27:00] PROBLEM - Host an-worker1191 is DOWN: PING CRITICAL - Packet loss = 100% [14:27:19] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1054] - vriley@cumin1003" [14:27:24] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1054] - vriley@cumin1003" [14:27:24] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:27:42] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1054 [14:28:00] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1054 [14:28:05] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2304-2317].codfw.wmnet [14:28:08] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2304-2317].codfw.wmnet [14:28:37] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2318-2331].codfw.wmnet [14:29:19] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:29:30] RECOVERY - Host an-worker1191 is UP: PING OK - Packet loss = 0%, RTA = 0.14 ms [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1430) [14:31:14] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12222128 (10LSobanski) p:05Triage→03Unbreak! [14:31:48] (03CR) 10Arnaudb: "some comments inline!" [puppet] - 10https://gerrit.wikimedia.org/r/1326292 (https://phabricator.wikimedia.org/T433988) (owner: 10AOkoth) [14:32:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cgoubert@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326306 (https://phabricator.wikimedia.org/T366938) (owner: 10Clément Goubert) [14:32:28] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12222140 (10LSobanski) p:05Unbreak!→03Medium [14:34:24] (03Merged) 10jenkins-bot: Add placeholder $wmgRedisLockPassword [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326306 (https://phabricator.wikimedia.org/T366938) (owner: 10Clément Goubert) [14:34:39] !log cgoubert@deploy1003 Started scap sync-world: Backport for [[gerrit:1326306|Add placeholder $wmgRedisLockPassword (T366938 T427999)]] [14:34:45] T366938: Reduce relying on database locks - https://phabricator.wikimedia.org/T366938 [14:34:45] T427999: Redis solution for LockManager - https://phabricator.wikimedia.org/T427999 [14:35:04] (03CR) 10Xcollazo: [C:03+1] hadoop/spark35: Disable SQL options redaction regex [puppet] - 10https://gerrit.wikimedia.org/r/1326290 (https://phabricator.wikimedia.org/T405360) (owner: 10Aqu) [14:35:38] (03CR) 10Scott French: [C:03+1] "Sounds good. I'll aim to pick this up during the infra window at 17:00 UTC today." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325949 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [14:36:24] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2318-2331].codfw.wmnet [14:36:26] (03CR) 10Eevans: [C:03+2] aqs: Cassandra 5.0.8 upgrade [puppet] - 10https://gerrit.wikimedia.org/r/1324813 (https://phabricator.wikimedia.org/T433026) (owner: 10Eevans) [14:36:26] !log sukhe@cumin1003 START - Cookbook sre.hosts.reboot-single for host acmechief1002.eqiad.wmnet [14:36:28] !log cgoubert@deploy1003 cgoubert: Backport for [[gerrit:1326306|Add placeholder $wmgRedisLockPassword (T366938 T427999)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:37:19] !log cgoubert@deploy1003 cgoubert: Continuing with deployment [14:37:29] !log sukhe@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on dns1006.wikimedia.org with reason: host reimage [14:37:51] (03CR) 10Ladsgroup: [C:03+1] mw-on-k8s: Add egress to new rdb-lock hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325872 (https://phabricator.wikimedia.org/T427999) (owner: 10Clément Goubert) [14:38:35] !log sukhe@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on dns2006.wikimedia.org with reason: host reimage [14:39:04] (03CR) 10Clément Goubert: [C:03+2] site.pp: Configure rdb-lock instances [puppet] - 10https://gerrit.wikimedia.org/r/1324694 (https://phabricator.wikimedia.org/T434188) (owner: 10Clément Goubert) [14:39:21] (03CR) 10Clément Goubert: [C:03+2] external_services: Just in case, add redis-lock [puppet] - 10https://gerrit.wikimedia.org/r/1325875 (https://phabricator.wikimedia.org/T434188) (owner: 10Clément Goubert) [14:39:22] !log vriley@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1051.eqiad.wmnet with reason: host reimage [14:39:47] (03CR) 10Bking: [C:03+2] stat hosts: update partitioning to retain /srv data [puppet] - 10https://gerrit.wikimedia.org/r/1325921 (https://phabricator.wikimedia.org/T434562) (owner: 10Bking) [14:40:11] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1002.eqiad.wmnet [14:41:28] !log sukhe@cumin1003 START - Cookbook sre.hosts.reboot-single for host acmechief2002.codfw.wmnet [14:41:35] !log cgoubert@deploy1003 Finished scap sync-world: Backport for [[gerrit:1326306|Add placeholder $wmgRedisLockPassword (T366938 T427999)]] (duration: 06m 56s) [14:41:40] T366938: Reduce relying on database locks - https://phabricator.wikimedia.org/T366938 [14:41:41] T427999: Redis solution for LockManager - https://phabricator.wikimedia.org/T427999 [14:41:52] (03CR) 10Btullis: [C:03+2] admin_ng: raise revertrisk namespace resource quota for rollout headroom [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) (owner: 10AikoChou) [14:42:44] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching aqs[2002-2012].codfw.wmnet,aqs[1017-1027].eqiad.wmnet: Upgrade Cassandra to 5.0.8 — T433026 - eevans@cumin1003 [14:42:49] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [14:43:39] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2318-2331].codfw.wmnet [14:43:42] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2318-2331].codfw.wmnet [14:43:45] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet [14:43:55] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1143.eqiad.wmnet with OS bookworm [14:44:21] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet [14:44:42] (03CR) 10Klausman: [C:03+1] admin_ng: raise revertrisk namespace resource quota for rollout headroom [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) (owner: 10AikoChou) [14:44:52] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns1006.wikimedia.org with reason: host reimage [14:45:17] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief2002.codfw.wmnet [14:46:55] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1182.eqiad.wmnet with OS bookworm [14:47:21] PROBLEM - Recursive DNS on 208.80.153.107 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [14:48:18] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1051.eqiad.wmnet with reason: host reimage [14:48:31] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1188.eqiad.wmnet with reason: host reimage [14:48:53] jouncebot: nowandnext [14:48:54] For the next 0 hour(s) and 11 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1430) [14:48:54] In 0 hour(s) and 41 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1530) [14:49:32] claime: are you still deploying? [14:50:18] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1189.eqiad.wmnet with OS bookworm [14:50:35] (03PS3) 10Majavah: Undeploy WP25EasterEggs (II) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318203 (https://phabricator.wikimedia.org/T418134) [14:50:41] taavi: no but if you're about to do a mediawiki deployment I have an egress config change to sneak in if that's ok? [14:50:56] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet [14:50:57] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet [14:51:00] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker-exp2001.codfw.wmnet [14:51:02] yeah I was planning on sneaking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1318203 out in this gap [14:51:04] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [14:51:13] so sure, let me know when I'm good to scap [14:51:23] (03CR) 10Clément Goubert: [C:03+2] mw-on-k8s: Add egress to new rdb-lock hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325872 (https://phabricator.wikimedia.org/T427999) (owner: 10Clément Goubert) [14:51:32] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker-exp2001.codfw.wmnet [14:51:39] will do, it's in CI rn [14:52:21] PROBLEM - Recursive DNS on 208.80.154.77 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [14:52:24] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1188.eqiad.wmnet with reason: host reimage [14:52:41] (03CR) 10CI reject: [V:04-1] admin_ng: raise revertrisk namespace resource quota for rollout headroom [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) (owner: 10AikoChou) [14:54:02] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1047.eqiad.wmnet [14:54:04] (03CR) 10Bking: [C:03+2] cirrussearch: Remove references to OpenSearch 1/relocate hiera values [puppet] - 10https://gerrit.wikimedia.org/r/1326304 (https://phabricator.wikimedia.org/T430163) (owner: 10Bking) [14:55:41] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1055] - vriley@cumin1003" [14:55:46] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1055] - vriley@cumin1003" [14:55:46] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:55:49] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns2006.wikimedia.org with reason: host reimage [14:56:10] (03Merged) 10jenkins-bot: mw-on-k8s: Add egress to new rdb-lock hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325872 (https://phabricator.wikimedia.org/T427999) (owner: 10Clément Goubert) [14:56:17] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1055 [14:56:32] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1055 [14:56:45] (03CR) 10Ottomata: [C:03+1] hadoop/spark35: Disable SQL options redaction regex [puppet] - 10https://gerrit.wikimedia.org/r/1326290 (https://phabricator.wikimedia.org/T405360) (owner: 10Aqu) [14:57:21] PROBLEM - Recursive DNS on 2620:0:860:4:208:80:153:107 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [14:57:22] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:57:23] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.admin rebooting A:liberica-magru and not P{lvs7003*} and A:liberica [14:57:28] taavi: you're good to go, I have another patch to add but it can wait [14:57:35] (03CR) 10TrainBranchBot: [C:03+2] "Approved by taavi@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318203 (https://phabricator.wikimedia.org/T418134) (owner: 10Majavah) [14:57:37] thanks! [14:57:40] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker-exp2001.codfw.wmnet [14:57:41] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker-exp2001.codfw.wmnet [14:57:41] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-codfw [14:58:03] I forgot that mw-debug doesn't use the global egress definitions T_T [14:58:45] !log sukhe@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy [14:59:34] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet [14:59:49] (03CR) 10Btullis: [C:03+2] "recheck" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) (owner: 10AikoChou) [14:59:59] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet [15:00:02] (03Merged) 10jenkins-bot: Undeploy WP25EasterEggs (II) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318203 (https://phabricator.wikimedia.org/T418134) (owner: 10Majavah) [15:00:18] !log taavi@deploy1003 Started scap sync-world: Backport for [[gerrit:1318203|Undeploy WP25EasterEggs (II) (T418134)]] [15:00:23] T418134: [EPIC] Undeploy WP25EasterEggs extension & related WP25 birthday features - https://phabricator.wikimedia.org/T418134 [15:00:44] (03PS1) 10Clément Goubert: mw-debug: Add missing rdb-lock egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326310 (https://phabricator.wikimedia.org/T427999) [15:01:42] cdobbins@cumin1003 roll-upgrade-ats (PID 469730) is awaiting input [15:01:54] (03CR) 10Ladsgroup: [C:03+1] mw-debug: Add missing rdb-lock egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326310 (https://phabricator.wikimedia.org/T427999) (owner: 10Clément Goubert) [15:02:24] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [15:03:10] vriley@cumin1003 provision (PID 477406) is awaiting input [15:03:27] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1055.eqiad.wmnet with OS trixie [15:03:48] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12222395 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1055.eqiad.wmnet with OS trixie [15:04:14] !log taavi@deploy1003 taavi: Backport for [[gerrit:1318203|Undeploy WP25EasterEggs (II) (T418134)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:04:24] !log sukhe@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir and A:ncredir [15:05:01] !log taavi@deploy1003 taavi: Continuing with deployment [15:05:21] PROBLEM - Recursive DNS on 2620:0:861:3:208:80:154:77 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [15:05:23] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [15:05:36] some DNS alerts expected, reimages [15:05:41] keeping an eye out, nothing to worry [15:06:14] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [15:06:15] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1051.eqiad.wmnet with OS trixie [15:06:29] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12222406 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1051.eqiad.wmnet with OS trixie completed: - clo... [15:06:52] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12222417 (10VRiley-WMF) [15:06:55] !log sukhe@cumin1003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-magru and not P{lvs7003*} and A:liberica [15:08:19] RECOVERY - Recursive DNS on 208.80.154.77 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [15:08:19] RECOVERY - Recursive DNS on 2620:0:861:3:208:80:154:77 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [15:09:14] !log taavi@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318203|Undeploy WP25EasterEggs (II) (T418134)]] (duration: 08m 56s) [15:09:19] T418134: [EPIC] Undeploy WP25EasterEggs extension & related WP25 birthday features - https://phabricator.wikimedia.org/T418134 [15:09:24] claime: all done [15:09:59] taavi: tyvm [15:10:17] (03CR) 10CI reject: [V:04-1] admin_ng: raise revertrisk namespace resource quota for rollout headroom [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) (owner: 10AikoChou) [15:12:10] (03CR) 10Ssingh: "0 tests failed, 0 tests skipped, 45 tests passed" [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [15:12:52] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1188.eqiad.wmnet with OS bookworm [15:13:01] sukhe@cumin1003 reimage (PID 474215) is awaiting input [15:13:23] !log sukhe@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns4004.wikimedia.org with OS trixie [15:13:48] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [15:16:10] FIRING: [10x] BFDdown: BFD session down between cr1-codfw and 208.80.153.107 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:16:19] RECOVERY - Recursive DNS on 208.80.153.107 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [15:16:19] RECOVERY - Recursive DNS on 2620:0:860:4:208:80:153:107 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [15:16:21] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.admin rebooting A:liberica-ulsfo and A:liberica [15:17:37] btullis@cumin1003 reimage (PID 496772) is awaiting input [15:17:48] btullis@cumin1003 reimage (PID 496783) is awaiting input [15:17:56] btullis@cumin1003 reimage (PID 496792) is awaiting input [15:18:04] btullis@cumin1003 reimage (PID 496824) is awaiting input [15:18:08] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1154.eqiad.wmnet with OS bookworm [15:18:11] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1176.eqiad.wmnet with OS bookworm [15:18:21] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1191.eqiad.wmnet with OS bookworm [15:18:36] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1190.eqiad.wmnet with OS bookworm [15:19:01] !log sukhe@cumin1003 START - Cookbook sre.hosts.reimage for host dns4004.wikimedia.org with OS trixie [15:19:19] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns1006.wikimedia.org with OS trixie [15:19:47] (03PS2) 10Btullis: admin_ng: raise revertrisk namespace resource quota for rollout headroom [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) (owner: 10AikoChou) [15:19:55] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1056~] - vriley@cumin1003" [15:20:01] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1056~] - vriley@cumin1003" [15:20:01] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:20:26] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1056 [15:21:10] FIRING: [10x] BFDdown: BFD session down between cr1-codfw and 208.80.153.107 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:21:16] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: pki1002 became unresponsive causing several hosts to alert on failed puppet runs. - https://phabricator.wikimedia.org/T434268#12222532 (10MoritzMuehlenhoff) >>! In T434268#12221369, @elukey wrote: >> I think that along with the upgrade host maybe... [15:23:04] !log sukhe@cumin1003 START - Cookbook sre.hosts.remove-downtime for dns1006.wikimedia.org [15:23:05] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns1006.wikimedia.org [15:23:26] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns1006.wikimedia.org,service=authdns-update [15:23:29] (03CR) 10Btullis: "recheck" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) (owner: 10AikoChou) [15:23:30] vriley@cumin1003 configure-switch-interfaces (PID 497918) is awaiting input [15:23:34] !log sukhe@dns1004 START - running authdns-update [15:24:07] (03PS1) 10Phuedx: [eventgate] Bump to v1.33.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326314 (https://phabricator.wikimedia.org/T430322) [15:25:27] (03CR) 10TChin: [C:03+1] [eventgate] Bump to v1.33.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326314 (https://phabricator.wikimedia.org/T430322) (owner: 10Phuedx) [15:25:31] !log sukhe@dns1004 END - running authdns-update [15:25:39] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1056 [15:25:53] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns2006.wikimedia.org with OS trixie [15:26:22] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [15:26:39] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns1006.wikimedia.org [15:26:54] !log sukhe@cumin1003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-ulsfo and A:liberica [15:27:12] RECOVERY - Host ntp-c.anycast.wmnet is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [15:29:07] !log sukhe@cumin1003 START - Cookbook sre.hosts.remove-downtime for dns2006.wikimedia.org [15:29:08] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns2006.wikimedia.org [15:29:12] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns2006.wikimedia.org,service=authdns-update [15:29:13] !log sukhe@dns1004 START - running authdns-update [15:30:05] jan_drewniak: How many deployers does it take to do Wikimedia Portals Update deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1530). [15:31:10] !log sukhe@dns1004 END - running authdns-update [15:31:31] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns2006.wikimedia.org [15:33:06] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1190.eqiad.wmnet with reason: host reimage [15:33:41] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1191.eqiad.wmnet with reason: host reimage [15:33:57] (03CR) 10CI reject: [V:04-1] admin_ng: raise revertrisk namespace resource quota for rollout headroom [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) (owner: 10AikoChou) [15:34:09] (03Merged) 10jenkins-bot: admin_ng: raise revertrisk namespace resource quota for rollout headroom [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326234 (https://phabricator.wikimedia.org/T426947) (owner: 10AikoChou) [15:34:10] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage [15:34:32] !log failover Ganeti master in eqiad to ganeti1046 [15:34:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:35:16] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [15:35:44] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1056.eqiad.wmnet with OS trixie [15:35:51] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [15:36:02] (03CR) 10Clément Goubert: [C:03+2] mw-debug: Add missing rdb-lock egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326310 (https://phabricator.wikimedia.org/T427999) (owner: 10Clément Goubert) [15:36:05] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12222609 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1056.eqiad.wmnet with OS trixie [15:36:29] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1054.eqiad.wmnet with OS trixie [15:36:43] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12222612 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1054.eqiad.wmnet with OS trixie [15:36:53] (03CR) 10JHathaway: nftables: add ipencap support (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1322963 (https://phabricator.wikimedia.org/T433601) (owner: 10JHathaway) [15:37:12] PROBLEM - ganeti-wconfd running on ganeti1048 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 111 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [15:37:55] RESOLVED: [2x] SystemdUnitFailed: amd-smi-gpu-partition.service on ml-serve1015:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:38:52] (03Merged) 10jenkins-bot: mw-debug: Add missing rdb-lock egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326310 (https://phabricator.wikimedia.org/T427999) (owner: 10Clément Goubert) [15:39:14] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1190.eqiad.wmnet with reason: host reimage [15:42:53] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1191.eqiad.wmnet with reason: host reimage [15:42:59] !log btullis@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [15:43:21] FIRING: [3x] ProbeDown: Service aqs1021-b:9042 has failed probes (tcp_cassandra_b_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:43:44] !log btullis@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [15:45:21] (03PS1) 10Federico Ceratto: Add MariaDB test-s8 section VMs [puppet] - 10https://gerrit.wikimedia.org/r/1326316 (https://phabricator.wikimedia.org/T435059) [15:45:33] !log btullis@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [15:46:15] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage [15:46:58] !log btullis@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [15:48:19] RESOLVED: PuppetFailure: Puppet has failed on ml-serve1015:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:49:23] (03PS2) 10Jgiannelos: prv: Enable parsoid rendering for 5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326232 (https://phabricator.wikimedia.org/T435115) [15:50:21] !log sukhe@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on dns4004.wikimedia.org with reason: host reimage [15:52:10] (03CR) 10CI reject: [V:04-1] prv: Enable parsoid rendering for 5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326232 (https://phabricator.wikimedia.org/T435115) (owner: 10Jgiannelos) [15:53:13] (03CR) 10Marostegui: "The three of them in eqiad are masters?" [puppet] - 10https://gerrit.wikimedia.org/r/1326316 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [15:53:31] (03CR) 10Jgiannelos: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326232 (https://phabricator.wikimedia.org/T435115) (owner: 10Jgiannelos) [15:53:32] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns4004.wikimedia.org with reason: host reimage [15:55:28] !log cgoubert@deploy1003 helmfile [codfw] START helmfile.d/services/mw-debug: apply [15:55:38] !log cgoubert@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [15:55:48] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-debug: apply [15:55:55] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [15:56:45] (03PS2) 10Scott French: P:etcd::replication: Infer an https-compatible default dst_url [puppet] - 10https://gerrit.wikimedia.org/r/1326309 (https://phabricator.wikimedia.org/T435103) [15:57:14] (03PS2) 10Federico Ceratto: Add MariaDB test-s8 section VMs [puppet] - 10https://gerrit.wikimedia.org/r/1326316 (https://phabricator.wikimedia.org/T435059) [15:58:13] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@ulsfo - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:58:43] (03PS3) 10Scott French: P:etcd::replication: Infer an https-compatible default dst_url [puppet] - 10https://gerrit.wikimedia.org/r/1326309 (https://phabricator.wikimedia.org/T435103) [15:59:08] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1154.eqiad.wmnet with reason: host reimage [15:59:12] (03CR) 10Federico Ceratto: "This is not the final configuration: it's just to run the preseed. Anyhow I removed master from all the nodes for simplicity to deploy the" [puppet] - 10https://gerrit.wikimedia.org/r/1326316 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [15:59:46] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1190.eqiad.wmnet with OS bookworm [16:00:06] (03CR) 10BCornwall: [C:03+1] "Great. Thanks so much for this!" [alerts] - 10https://gerrit.wikimedia.org/r/1319829 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [16:00:10] PROBLEM - Recursive DNS on 198.35.26.34 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [16:01:46] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations: Move Docker images under the /v2/releng prefix to S3 - https://phabricator.wikimedia.org/T432829#12222802 (10elukey) Checked up the list of the past two weeks on both registry2* hosts (the ones pooled): ` /v2/releng/ajv/manifests/0.4.1-s1 /v2/releng... [16:02:05] (03CR) 10Scott French: [C:03+1] "Thanks, Raine!" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1325907 (https://phabricator.wikimedia.org/T432988) (owner: 10Kamila Součková) [16:03:13] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs[2002-2012].codfw.wmnet,aqs[1017-1027].eqiad.wmnet: Upgrade Cassandra to 5.0.8 — T433026 - eevans@cumin1003 [16:03:18] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [16:03:46] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1154.eqiad.wmnet with reason: host reimage [16:04:30] (03CR) 10Phuedx: [C:03+2] [eventgate] Bump to v1.33.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326314 (https://phabricator.wikimedia.org/T430322) (owner: 10Phuedx) [16:05:10] PROBLEM - Recursive DNS on 2620:0:863:2:198:35:26:34 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [16:05:23] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1191.eqiad.wmnet with OS bookworm [16:05:43] (03CR) 10JHathaway: [C:03+1] puppetserver: pull puppet via discovery record [puppet] - 10https://gerrit.wikimedia.org/r/1296495 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [16:06:23] !log btullis@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [16:07:08] (03CR) 10Eevans: [C:03+2] aqs: upgrade to Java 17 [puppet] - 10https://gerrit.wikimedia.org/r/1324814 (https://phabricator.wikimedia.org/T433026) (owner: 10Eevans) [16:07:24] (03Merged) 10jenkins-bot: [eventgate] Bump to v1.33.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326314 (https://phabricator.wikimedia.org/T430322) (owner: 10Phuedx) [16:07:50] !log btullis@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [16:08:13] FIRING: [5x] JobUnavailable: Reduced availability for job haproxy in ops@ulsfo - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:09:08] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1176.eqiad.wmnet with OS bookworm [16:09:56] !log phuedx@deploy1003 helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply [16:11:01] !log phuedx@deploy1003 helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply [16:11:50] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12222861 (10RobH) >>! In T394357#12194858, @MatthewVernon wrote: > @RobH from my point of view, the JBOD disk-swap behaviour seems good, so if @elukey is hap... [16:12:38] !log phuedx@deploy1003 helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [16:13:12] FIRING: [4x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:44] !log phuedx@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [16:15:02] !log phuedx@deploy1003 helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply [16:15:08] RECOVERY - Recursive DNS on 2620:0:863:2:198:35:26:34 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [16:15:08] RECOVERY - Recursive DNS on 198.35.26.34 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [16:16:04] !log phuedx@deploy1003 helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply [16:16:06] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12222903 (10RobH) [16:17:38] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [16:18:13] FIRING: [4x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:19:06] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.admin rebooting A:liberica-drmrs and A:liberica [16:21:45] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1057] - vriley@cumin1003" [16:21:50] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1057] - vriley@cumin1003" [16:21:50] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:22:07] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1057 [16:22:22] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1057 [16:23:12] (03CR) 10Subramanya Sastry: [C:03+1] prv: Enable parsoid rendering for 5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326232 (https://phabricator.wikimedia.org/T435115) (owner: 10Jgiannelos) [16:23:42] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1055.eqiad.wmnet with OS trixie [16:23:56] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [16:23:59] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12222959 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1055.eqiad.wmnet with OS trixie executed with er... [16:26:03] (03PS1) 10Elukey: docker_registry: route /v2/releng to its new s3 backend [puppet] - 10https://gerrit.wikimedia.org/r/1326323 (https://phabricator.wikimedia.org/T432829) [16:27:45] (03CR) 10Krinkle: "It seems change https://gerrit.wikimedia.org/r/c/operations/software/thumbor-plugins/+/1311830 had no effect. (Unless it wasn't deployed y" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322507 (https://phabricator.wikimedia.org/T430528) (owner: 10Samwilson) [16:28:59] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1154.eqiad.wmnet with OS bookworm [16:29:37] !log sukhe@cumin1003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-drmrs and A:liberica [16:30:15] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12223002 (10VRiley-WMF) Hey @fgiunchedi it seems as though that cloudvirt1054, cloudvirt1055, cloudvirt1056, cloudvirt1057 are having issues. I did noticed that th... [16:30:43] vriley@cumin1003 provision (PID 516082) is awaiting input [16:30:57] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [16:31:26] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1057.eqiad.wmnet with OS trixie [16:31:40] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12223011 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1057.eqiad.wmnet with OS trixie [16:33:30] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1157.eqiad.wmnet with OS bookworm [16:33:33] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1158.eqiad.wmnet with OS bookworm [16:33:36] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1159.eqiad.wmnet with OS bookworm [16:33:41] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1183.eqiad.wmnet with OS bookworm [16:33:44] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1192.eqiad.wmnet with OS bookworm [16:34:45] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12223040 (10VRiley-WMF) @jcrespo Thanks, I do apologize for the delay. I assure you this is still on my radar and I'm trying to find an answer for this as soon as possible. [16:35:19] (03CR) 10Marostegui: [C:03+1] Add MariaDB test-s8 section VMs [puppet] - 10https://gerrit.wikimedia.org/r/1326316 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [16:35:54] btullis@cumin1003 reimage (PID 516851) is awaiting input [16:36:32] btullis@cumin1003 reimage (PID 516821) is awaiting input [16:36:54] FIRING: [6x] CirrusSearchTitleSuggestIndexTooOld: Some search indices that power autocomplete have not been updated recently - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - TODO - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchTitleSuggestIndexTooOld [16:38:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [16:38:24] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12223055 (10MoritzMuehlenhoff) [16:39:05] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1180.eqiad.wmnet with OS bookworm [16:39:21] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1193.eqiad.wmnet with OS bookworm [16:39:31] !log btullis@cumin1003 START - Cookbook sre.hosts.move-vlan for host an-worker1180 [16:39:44] !log btullis@cumin1003 START - Cookbook sre.dns.netbox [16:40:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [16:43:22] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [16:43:36] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns4004.wikimedia.org with OS trixie [16:44:08] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns4004.wikimedia.org,service=authdns-update [16:44:14] !log sukhe@dns1004 START - running authdns-update [16:45:32] btullis@cumin1003 reimage (PID 517403) is awaiting input [16:45:39] FIRING: [4x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [16:46:10] !log sukhe@dns1004 END - running authdns-update [16:46:28] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns4004.wikimedia.org [16:47:59] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1192.eqiad.wmnet with reason: host reimage [16:48:47] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1159.eqiad.wmnet with reason: host reimage [16:49:18] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1183.eqiad.wmnet with reason: host reimage [16:49:34] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.admin rebooting A:liberica-eqsin and A:liberica [16:49:39] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1157.eqiad.wmnet with reason: host reimage [16:49:49] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1158.eqiad.wmnet with reason: host reimage [16:50:25] !log btullis@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1180 - btullis@cumin1003" [16:50:30] !log btullis@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1180 - btullis@cumin1003" [16:50:31] !log btullis@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:50:32] !log btullis@cumin1003 START - Cookbook sre.dns.wipe-cache an-worker1180.eqiad.wmnet 17.36.64.10.in-addr.arpa 7.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [16:50:35] !log btullis@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1180.eqiad.wmnet 17.36.64.10.in-addr.arpa 7.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [16:50:36] !log btullis@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host an-worker1180 [16:51:28] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Upgrade to Java 17 — T433026 - eevans@cumin1003 [16:51:33] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [16:54:00] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1192.eqiad.wmnet with reason: host reimage [16:54:11] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12223128 (10elukey) @RobH I think we are good to proceed! [16:54:47] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1193.eqiad.wmnet with reason: host reimage [16:55:59] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1056.eqiad.wmnet with OS trixie [16:56:19] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12223134 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1056.eqiad.wmnet with OS trixie executed with er... [16:56:27] (03PS1) 10DLynch: editcheck-headless: Add chart and deployment for the technical pilot [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326332 (https://phabricator.wikimedia.org/T434109) [16:56:43] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1054.eqiad.wmnet with OS trixie [16:57:00] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12223142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1054.eqiad.wmnet with OS trixie executed with er... [16:57:11] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1183.eqiad.wmnet with reason: host reimage [16:58:04] (03CR) 10Jasmine: [C:03+1] mediawiki: Refactor lamp.deployment into component containers. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313111 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [17:00:04] swfrench-wmf: OwO what's this, a deployment window?? MediaWiki infrastructure (UTC late). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1700). nyaa~ [17:00:05] ryankemper: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Wikidata Query Service weekly deploy . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1700). [17:00:14] o/ [17:00:23] (03CR) 10Scott French: [C:03+2] admin_ng: allow-urldownloaders: update urldownloader VIPs for egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325949 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:00:31] wooo thanks swfrench-wmf! [17:00:36] :) [17:00:42] no problem [17:01:01] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1193.eqiad.wmnet with reason: host reimage [17:01:14] !log sukhe@cumin1003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-eqsin and A:liberica [17:01:58] FYI, I'll be deploying a couple of changes during this infra window, starting with the above. next up thereafter is a puppet change requiring a mediawiki deployment. [17:02:42] !log btullis@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1180 [17:02:42] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1180 [17:04:48] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1157.eqiad.wmnet with reason: host reimage [17:06:29] (03CR) 10Scott French: [C:03+2] Point Test Wiki to new docroot [puppet] - 10https://gerrit.wikimedia.org/r/1315141 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [17:06:51] (03CR) 10Scott French: [C:03+2] "Done" [puppet] - 10https://gerrit.wikimedia.org/r/1315141 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [17:07:07] (03PS1) 10Btullis: Deploy updated opensearch images to all clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326335 [17:09:38] * swfrench-wmf side-eyes at CI on 1325949 [17:09:45] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1158.eqiad.wmnet with reason: host reimage [17:10:13] (03Merged) 10jenkins-bot: admin_ng: allow-urldownloaders: update urldownloader VIPs for egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325949 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:12:41] !log swfrench@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [17:13:56] !log swfrench@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [17:14:26] !log swfrench@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [17:14:40] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1159.eqiad.wmnet with reason: host reimage [17:15:09] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1192.eqiad.wmnet with OS bookworm [17:15:18] !log swfrench@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [17:15:44] !log swfrench@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [17:16:09] !log swfrench@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [17:16:28] !log swfrench@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [17:17:06] !log swfrench@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [17:17:31] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [17:17:48] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1180.eqiad.wmnet with reason: host reimage [17:18:17] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [17:18:59] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [17:20:05] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1183.eqiad.wmnet with OS bookworm [17:20:23] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [17:20:39] PROBLEM - Host an-worker1180 is DOWN: PING CRITICAL - Packet loss = 100% [17:20:47] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [17:21:09] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [17:21:37] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [17:21:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:21:40] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1193.eqiad.wmnet with OS bookworm [17:22:16] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [17:23:16] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1180.eqiad.wmnet with reason: host reimage [17:25:42] RECOVERY - Host an-worker1180 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [17:25:43] sukhe: alright, looks like you should be all set - both for clients that depend on the global policy and for those using external-services policies [17:27:48] next up is that mediawiki change, which I'll deploy in a couple of minutes after checking on a couple of things [17:28:34] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1157.eqiad.wmnet with OS bookworm [17:28:38] note that we may see some httpbb check failures fire in the meantime. this deployment should fix those. [17:30:22] !log cdobbins@cumin1003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-ulsfo or A:cp-drmrs and A:cp - 9.2.15 upgrade (T434620) [17:30:26] T434620: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Mentimeter - https://phabricator.wikimedia.org/T434620 [17:30:56] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Upgrade to Java 17 — T433026 - eevans@cumin1003 [17:31:00] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [17:34:13] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1158.eqiad.wmnet with OS bookworm [17:36:36] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1159.eqiad.wmnet with OS bookworm [17:37:22] swfrench-wmf: <3 [17:38:02] (03CR) 10Brennen Bearnes: phabricator: fix php version in restart script (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1326292 (https://phabricator.wikimedia.org/T433988) (owner: 10AOkoth) [17:38:46] !log swfrench@deploy1003 Started scap sync-world: Deploy "Point Test Wiki to new docroot" - T432412 [17:38:51] T432412: [Eng] iOS - Update app site association file - https://phabricator.wikimedia.org/T432412 [17:39:08] !log cdobbins@cumin1003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-eqsin and A:cp - 9.2.15 upgrade (T434478) [17:39:42] !log swfrench@deploy1003 swfrench: Deploy "Point Test Wiki to new docroot" - T432412 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:43:25] (03PS1) 10Ssingh: dns1005|dns[25]004: use new pdns-rec config [puppet] - 10https://gerrit.wikimedia.org/r/1326341 (https://phabricator.wikimedia.org/T401832) [17:44:55] (03PS1) 10Ssingh: Revert^2 "wmf-config/ProductionServices: set URL for urldownloader to service record" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326342 [17:47:03] (03CR) 10BCornwall: "Marking unresolved" [puppet] - 10https://gerrit.wikimedia.org/r/1320177 (https://phabricator.wikimedia.org/T433830) (owner: 10Majavah) [17:47:32] debugging some lingering httpbb test issues - will update soon [17:47:49] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1180.eqiad.wmnet with OS bookworm [17:51:22] 06SRE, 10SRE-Access-Requests, 06tools-infrastructure-team: Requesting access to wmcs-roots for bliviero - https://phabricator.wikimedia.org/T435123 (10Andrew) 03NEW [17:51:42] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1057.eqiad.wmnet with OS trixie [17:51:46] (03CR) 10Bking: Deploy updated opensearch images to all clusters (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326335 (owner: 10Btullis) [17:52:00] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12223455 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1057.eqiad.wmnet with OS trixie executed with er... [17:52:34] 06SRE, 10SRE-Access-Requests, 06tools-infrastructure-team: Requesting access to wmcs-roots for bliviero - https://phabricator.wikimedia.org/T435123#12223459 (10Andrew) See https://phabricator.wikimedia.org/T428815 for a lot of this having already been done for a different access group. [17:54:28] PROBLEM - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:54:45] ^ known [17:54:54] (03PS1) 10Andrew Bogott: Add bliviero to wmcs-roots and include an ssh key [puppet] - 10https://gerrit.wikimedia.org/r/1326343 (https://phabricator.wikimedia.org/T435123) [17:55:30] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:55:30] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:56:25] ^ known [17:56:33] jouncebot: nowandnext [17:56:33] For the next 0 hour(s) and 3 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T1700) [17:56:33] In 2 hour(s) and 3 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T2000) [17:59:28] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:59:28] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:00:11] alright, httpbb test is getting sorted. the change itself looks good, so we'll proceed with that in the meantime :) [18:00:43] !log swfrench@deploy1003 swfrench: Continuing with deployment [18:02:26] FIRING: OutboundMXQueueHigh: MX host mx-out1001:9154 has many queued messages: 15849 #page - https://wikitech.wikimedia.org/wiki/Postfix - https://grafana.wikimedia.org/d/h36Havfik/mail-postfix-servers - https://alerts.wikimedia.org/?q=alertname%3DOutboundMXQueueHigh [18:04:50] !log swfrench@deploy1003 Finished scap sync-world: Deploy "Point Test Wiki to new docroot" - T432412 (duration: 26m 03s) [18:04:55] T432412: [Eng] iOS - Update app site association file - https://phabricator.wikimedia.org/T432412 [18:06:50] no further mediawiki deployments planned as part of the infra window. be aware, though, that there's an httpbb test being fixed - until that's live (soon) you will have to acknowledge the failure during the testservers phase. [18:07:52] (03PS1) 10Dwisehaupt: Adjust fundraisingdb-read for host reboot [dns] - 10https://gerrit.wikimedia.org/r/1326344 (https://phabricator.wikimedia.org/T419557) [18:07:54] (03PS1) 10Tsevener: Fix httpbb tests for Test Wiki [puppet] - 10https://gerrit.wikimedia.org/r/1326345 (https://phabricator.wikimedia.org/T432412) [18:08:14] 06SRE, 10SRE-Access-Requests, 06tools-infrastructure-team, 13Patch-For-Review: Requesting access to wmcs-roots for bliviero - https://phabricator.wikimedia.org/T435123#12223530 (10BLiviero-WMF) [18:10:57] (03CR) 10Scott French: [C:03+2] "Thanks for the fix!" [puppet] - 10https://gerrit.wikimedia.org/r/1326345 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [18:11:20] (03CR) 10Jgreen: [C:03+2] Adjust fundraisingdb-read for host reboot [dns] - 10https://gerrit.wikimedia.org/r/1326344 (https://phabricator.wikimedia.org/T419557) (owner: 10Dwisehaupt) [18:11:35] (03CR) 10Jgreen: [C:03+1] Adjust fundraisingdb-read for host reboot [dns] - 10https://gerrit.wikimedia.org/r/1326344 (https://phabricator.wikimedia.org/T419557) (owner: 10Dwisehaupt) [18:11:37] !incidents [18:11:37] 8250 (RESOLVED) OutboundMXQueueHigh sre (mx-out1001:9154 eqiad) [18:13:16] (03CR) 10Dwisehaupt: [C:03+2] Adjust fundraisingdb-read for host reboot [dns] - 10https://gerrit.wikimedia.org/r/1326344 (https://phabricator.wikimedia.org/T419557) (owner: 10Dwisehaupt) [18:14:04] !log dwisehaupt@dns1005 START - running authdns-update [18:14:32] 06SRE, 10SRE-Access-Requests, 06tools-infrastructure-team, 13Patch-For-Review: Requesting access to wmcs-roots for bliviero - https://phabricator.wikimedia.org/T435123#12223552 (10Andrew) @mark in theory Belinda is the approver of record for this group but we would like your vote before adding her. Thank you! [18:16:09] !log dwisehaupt@dns1005 END - running authdns-update [18:17:26] RESOLVED: OutboundMXQueueHigh: MX host mx-out1001:9154 has many queued messages: 1728 #page - https://wikitech.wikimedia.org/wiki/Postfix - https://grafana.wikimedia.org/d/h36Havfik/mail-postfix-servers - https://alerts.wikimedia.org/?q=alertname%3DOutboundMXQueueHigh [18:19:28] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:19:28] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:19:44] * swfrench-wmf thumbs up [18:23:48] !log swfrench@cumin2002 START - Cookbook sre.hosts.reboot-single for host conf2007.codfw.wmnet [18:24:14] ^ looks scary, but it is not (host is not yet part of the cluster) [18:24:28] RECOVERY - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:25:30] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:25:30] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:30:04] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf2007.codfw.wmnet [18:34:43] !log swfrench@cumin2002 START - Cookbook sre.hosts.reboot-single for host conf2008.codfw.wmnet [18:41:11] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf2008.codfw.wmnet [18:41:28] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1326309 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [18:42:55] (03PS1) 10Dwisehaupt: Set fundraisingdb-read back to frdb1008 [dns] - 10https://gerrit.wikimedia.org/r/1326349 (https://phabricator.wikimedia.org/T419557) [18:46:34] (03CR) 10Jgreen: [C:03+1] Set fundraisingdb-read back to frdb1008 [dns] - 10https://gerrit.wikimedia.org/r/1326349 (https://phabricator.wikimedia.org/T419557) (owner: 10Dwisehaupt) [18:47:04] (03CR) 10Scott French: "Note that:" [puppet] - 10https://gerrit.wikimedia.org/r/1326309 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [18:47:24] !log swfrench@cumin2002 START - Cookbook sre.hosts.reboot-single for host conf2009.codfw.wmnet [18:52:16] (03CR) 10Dwisehaupt: [C:03+2] Set fundraisingdb-read back to frdb1008 [dns] - 10https://gerrit.wikimedia.org/r/1326349 (https://phabricator.wikimedia.org/T419557) (owner: 10Dwisehaupt) [18:52:45] !log dancy@deploy1003 Installing scap version "4.281.1" for 3 host(s) [18:53:06] !log dwisehaupt@dns1006 START - running authdns-update [18:54:05] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf2009.codfw.wmnet [18:54:16] (03CR) 10RLazarus: [C:03+1] P:etcd::replication: Infer an https-compatible default dst_url [puppet] - 10https://gerrit.wikimedia.org/r/1326309 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [18:54:40] !log dancy@deploy1003 Installation of scap version "4.281.1" completed for 3 hosts [18:55:16] !log dwisehaupt@dns1006 END - running authdns-update [18:55:25] !log dancy@deploy1003 Started scap sync-world: testing T375514 [18:55:29] T375514: Scap k8s deployments sometimes report final progress < 100% - https://phabricator.wikimedia.org/T375514 [18:58:38] !log dancy@deploy1003 Finished scap sync-world: testing T375514 (duration: 03m 13s) [19:00:20] (03CR) 10Scott French: "Thanks for the review!" [puppet] - 10https://gerrit.wikimedia.org/r/1326309 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [19:00:22] (03CR) 10Scott French: [C:03+2] P:etcd::replication: Infer an https-compatible default dst_url [puppet] - 10https://gerrit.wikimedia.org/r/1326309 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [19:03:19] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Upgrade to Java 17 — T433026 - eevans@cumin1003 [19:03:23] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [19:06:04] (03CR) 10Btullis: Deploy updated opensearch images to all clusters (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326335 (owner: 10Btullis) [19:09:33] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [19:10:07] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [19:10:38] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1160.eqiad.wmnet with OS bookworm [19:10:42] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1161.eqiad.wmnet with OS bookworm [19:10:45] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1162.eqiad.wmnet with OS bookworm [19:10:49] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1184.eqiad.wmnet with OS bookworm [19:10:51] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1194.eqiad.wmnet with OS bookworm [19:11:02] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1195.eqiad.wmnet with OS bookworm [19:15:19] (03CR) 10Bking: Deploy updated opensearch images to all clusters (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326335 (owner: 10Btullis) [19:19:26] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply [19:21:26] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [19:24:02] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [software/klaxon] - 10https://gerrit.wikimedia.org/r/1322809 (owner: 10Ssingh) [19:24:53] (03CR) 10Andrea Denisse: [C:03+1] "Toppissimo!! :D" [debs/grafana-plugins] - 10https://gerrit.wikimedia.org/r/1326295 (https://phabricator.wikimedia.org/T432970) (owner: 10Tiziano Fogli) [19:25:11] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1195.eqiad.wmnet with reason: host reimage [19:25:19] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [alerts] - 10https://gerrit.wikimedia.org/r/1319828 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [19:26:01] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1162.eqiad.wmnet with reason: host reimage [19:26:17] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1184.eqiad.wmnet with reason: host reimage [19:26:25] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1160.eqiad.wmnet with reason: host reimage [19:26:41] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1161.eqiad.wmnet with reason: host reimage [19:29:14] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1195.eqiad.wmnet with reason: host reimage [19:32:54] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1161.eqiad.wmnet with reason: host reimage [19:33:41] (03PS1) 10Tsevener: Remove escaped paths in app site association file [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326361 (https://phabricator.wikimedia.org/T432412) [19:34:51] (03PS1) 10Ahmon Dancy: mediawiki-dumps-legacy/values.yaml: Drop resources.replicas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326362 (https://phabricator.wikimedia.org/T375514) [19:36:18] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1184.eqiad.wmnet with reason: host reimage [19:36:28] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply [19:36:30] !log cdobbins@cumin1003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-eqsin and A:cp - 9.2.15 upgrade (T434478) [19:37:22] (03PS1) 10Scott French: hieradata: etcd read-only in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1326355 (https://phabricator.wikimedia.org/T435103) [19:37:23] (03PS1) 10Scott French: hieradata: switch etcd replication from eqiad to codfw [puppet] - 10https://gerrit.wikimedia.org/r/1326356 (https://phabricator.wikimedia.org/T435103) [19:37:24] (03PS1) 10Scott French: hieradata: etcd read-write in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1326357 (https://phabricator.wikimedia.org/T435103) [19:37:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:37:29] (03PS1) 10Scott French: wmnet: Reduce _etcd._tcp.conftool (R/W) SRV TTL to 10s [dns] - 10https://gerrit.wikimedia.org/r/1326363 (https://phabricator.wikimedia.org/T435103) [19:37:31] (03PS1) 10Scott French: wmnet: Switch _etcd._tcp.conftool (R/W) SRV hosts to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1326364 (https://phabricator.wikimedia.org/T435103) [19:37:34] (03PS1) 10Scott French: wmnet: Restore _etcd._tcp.conftool (R/W) SRV TTL to 5M [dns] - 10https://gerrit.wikimedia.org/r/1326365 (https://phabricator.wikimedia.org/T435103) [19:37:49] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 17 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-ite" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326361 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [19:39:40] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1162.eqiad.wmnet with reason: host reimage [19:41:34] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-codfw: Upgrade to Java 17 — T433026 - eevans@cumin1003 [19:41:39] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [19:43:36] FIRING: [2x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:44:29] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1160.eqiad.wmnet with reason: host reimage [19:49:39] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-test: apply [19:49:43] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply [19:49:56] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1195.eqiad.wmnet with OS bookworm [19:55:19] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1161.eqiad.wmnet with OS bookworm [19:58:42] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1184.eqiad.wmnet with OS bookworm [20:00:04] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T2000). [20:00:04] cjming, toni_, and Superpes: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:25] o/ [20:00:30] o/ [20:00:30] \o [20:01:32] i'll go first - toni_ or Superpes, do you need a deployer? [20:01:55] yes please! thanks [20:02:05] I need a deployer :) My patch doesn't require test [20:02:21] ok - sounds good - i'll do both of yours in order [20:02:48] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1162.eqiad.wmnet with OS bookworm [20:02:53] btullis@cumin1003 reimage (PID 548927) is awaiting input [20:03:32] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1326077 (https://phabricator.wikimedia.org/T431493) (owner: 10Clare Ming) [20:06:46] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1160.eqiad.wmnet with OS bookworm [20:08:39] (03Merged) 10jenkins-bot: InstrumentConstructiveEdits: anchor all runs to the nearest `interval` [extensions/WikimediaEvents] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1326077 (https://phabricator.wikimedia.org/T431493) (owner: 10Clare Ming) [20:08:53] !log cjming@deploy1003 Started scap sync-world: Backport for [[gerrit:1326077|InstrumentConstructiveEdits: anchor all runs to the nearest `interval` (T431493)]] [20:08:58] T431493: [DE 1.1.9] An instrument that produces edit_survived events - https://phabricator.wikimedia.org/T431493 [20:10:44] !log cjming@deploy1003 cjming: Backport for [[gerrit:1326077|InstrumentConstructiveEdits: anchor all runs to the nearest `interval` (T431493)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:11:04] !log cjming@deploy1003 cjming: Continuing with deployment [20:11:17] (03PS2) 10Eevans: aqs: set storage compatability mode to `UPGRADING` [puppet] - 10https://gerrit.wikimedia.org/r/1324815 (https://phabricator.wikimedia.org/T433026) [20:11:17] (03PS2) 10Eevans: aqs: set storage compatability mode to `NONE` [puppet] - 10https://gerrit.wikimedia.org/r/1324816 (https://phabricator.wikimedia.org/T433026) [20:11:45] (03CR) 10Bking: [C:03+2] Deploy updated opensearch images to all clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326335 (owner: 10Btullis) [20:12:41] (03CR) 10JHathaway: [C:03+1] team-sre: add infrastructure-foundations tag [alerts] - 10https://gerrit.wikimedia.org/r/1319832 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [20:13:18] (03CR) 10Eevans: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1324815 (https://phabricator.wikimedia.org/T433026) (owner: 10Eevans) [20:15:18] !log cjming@deploy1003 Finished scap sync-world: Backport for [[gerrit:1326077|InstrumentConstructiveEdits: anchor all runs to the nearest `interval` (T431493)]] (duration: 06m 25s) [20:15:22] T431493: [DE 1.1.9] An instrument that produces edit_survived events - https://phabricator.wikimedia.org/T431493 [20:15:59] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326361 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [20:17:09] (03Merged) 10jenkins-bot: Remove escaped paths in app site association file [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326361 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [20:17:21] !log cjming@deploy1003 Started scap sync-world: Backport for [[gerrit:1326361|Remove escaped paths in app site association file (T432412)]] [20:17:26] T432412: [Eng] iOS - Update app site association file - https://phabricator.wikimedia.org/T432412 [20:17:34] (03CR) 10Eevans: [C:03+2] aqs: set storage compatability mode to `UPGRADING` [puppet] - 10https://gerrit.wikimedia.org/r/1324815 (https://phabricator.wikimedia.org/T433026) (owner: 10Eevans) [20:18:13] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:19:11] !log cjming@deploy1003 cjming, tsev: Backport for [[gerrit:1326361|Remove escaped paths in app site association file (T432412)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:19:41] toni_: ^^ on test servers - lmk when to sync [20:21:43] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Set storage compatability to UPGRADING — T433026 - eevans@cumin1003 [20:21:48] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [20:24:36] !log bking@deploy1003 `charlie --services_dir dse-k8s-services -s opensearch-* -e dse-k8s-*` T435125 [20:24:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:24:42] T435125: OpenSearch on K8s: Update/deploy new Docker images - https://phabricator.wikimedia.org/T435125 [20:25:57] inflatador: add "apply" if you don't just want the diffs :) [20:26:09] toni_: gtg? [20:26:26] yep looks good! sorry I was kicked out of the room briefly [20:26:27] rzl Ah, thanks! I was thinking `dry_run` would just be diffs [20:26:34] !log cjming@deploy1003 cjming, tsev: Continuing with deployment [20:26:49] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [20:26:52] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [20:26:58] slightly different semantics but that would make sense too [20:26:59] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [20:27:05] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage: apply [20:27:16] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply [20:27:19] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply [20:27:30] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply [20:27:35] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-apifeatureusage-test: apply [20:27:39] !log bking@deploy1003 `charlie --services_dir dse-k8s-services -s opensearch-* -e dse-k8s-* apply` T435125 [20:27:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:28:05] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [20:28:09] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [20:28:14] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [20:28:20] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [20:28:24] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [20:28:28] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [20:28:40] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [20:28:47] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [20:28:59] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [20:29:03] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [20:29:09] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [20:29:16] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [20:29:25] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply [20:29:29] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply [20:29:35] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply [20:29:39] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply [20:29:43] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub: apply [20:29:49] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub: apply [20:29:59] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply [20:30:03] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply [20:30:08] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply [20:30:14] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-toolhub-test: apply [20:30:19] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply [20:30:21] (03PS2) 10Superpes15: [itwiki/slwiki/tgwiki] Remove temporary Wikipedia 25 logos permanently (already reverted) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324320 (https://phabricator.wikimedia.org/T414265) [20:30:22] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply [20:30:27] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver: apply [20:30:34] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver: apply [20:30:40] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply [20:30:44] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply [20:30:48] !log cjming@deploy1003 Finished scap sync-world: Backport for [[gerrit:1326361|Remove escaped paths in app site association file (T432412)]] (duration: 13m 28s) [20:30:49] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply [20:30:53] T432412: [Eng] iOS - Update app site association file - https://phabricator.wikimedia.org/T432412 [20:30:56] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ttmserver-test: apply [20:31:08] toni_: should be live! [20:32:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324320 (https://phabricator.wikimedia.org/T414265) (owner: 10Superpes15) [20:32:50] thanks! [20:33:11] (03Merged) 10jenkins-bot: [itwiki/slwiki/tgwiki] Remove temporary Wikipedia 25 logos permanently (already reverted) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324320 (https://phabricator.wikimedia.org/T414265) (owner: 10Superpes15) [20:33:23] !log cjming@deploy1003 Started scap sync-world: Backport for [[gerrit:1324320|[itwiki/slwiki/tgwiki] Remove temporary Wikipedia 25 logos permanently (already reverted) (T414265 T414320 T415307)]] [20:33:31] T414265: Requesting temporary logo change for sl.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T414265 [20:33:31] T414320: Requesting temporary logo change for it.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T414320 [20:33:31] T415307: Requesting temporary logo change for tg.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T415307 [20:35:18] !log cjming@deploy1003 cjming, superpes: Backport for [[gerrit:1324320|[itwiki/slwiki/tgwiki] Remove temporary Wikipedia 25 logos permanently (already reverted) (T414265 T414320 T415307)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:35:39] Superpes: lmk when to sync - on mwdebug [20:35:49] No test required [20:35:58] (03CR) 10Scott French: "Thanks, Luca!" [puppet] - 10https://gerrit.wikimedia.org/r/1326323 (https://phabricator.wikimedia.org/T432829) (owner: 10Elukey) [20:35:59] great then syncing [20:36:07] !log cjming@deploy1003 cjming, superpes: Continuing with deployment [20:36:54] FIRING: [6x] CirrusSearchTitleSuggestIndexTooOld: Some search indices that power autocomplete have not been updated recently - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - TODO - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchTitleSuggestIndexTooOld [20:39:15] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [20:39:24] (03CR) 10Bliviero: [V:03+1] Add bliviero to wmcs-roots and include an ssh key [puppet] - 10https://gerrit.wikimedia.org/r/1326343 (https://phabricator.wikimedia.org/T435123) (owner: 10Andrew Bogott) [20:40:08] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [20:40:18] !log cjming@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324320|[itwiki/slwiki/tgwiki] Remove temporary Wikipedia 25 logos permanently (already reverted) (T414265 T414320 T415307)]] (duration: 06m 55s) [20:40:25] T414265: Requesting temporary logo change for sl.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T414265 [20:40:26] T414320: Requesting temporary logo change for it.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T414320 [20:40:26] T415307: Requesting temporary logo change for tg.wikipedia.org (WP25) - https://phabricator.wikimedia.org/T415307 [20:40:29] Superpes: should be live :) [20:40:54] Thanks for your assistance @cjming :) [20:41:00] np! [20:43:07] (03PS1) 10MusikAnimal: PersonalDashboard: add newly renamed *ReviewChangesMlModel setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326373 (https://phabricator.wikimedia.org/T422148) [20:43:22] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [20:45:39] FIRING: [4x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [20:48:58] (03CR) 10Eevans: [C:03+2] aptrepo: Create cassandra50 bookworm component [puppet] - 10https://gerrit.wikimedia.org/r/1325922 (https://phabricator.wikimedia.org/T357791) (owner: 10Eevans) [20:58:16] (03PS1) 10Andrew Bogott: Revert "backy2: increase max read rate to 120MB/second" [puppet] - 10https://gerrit.wikimedia.org/r/1326377 [20:58:16] (03PS1) 10Andrew Bogott: backy2: avoid parallel read/write options [puppet] - 10https://gerrit.wikimedia.org/r/1326378 (https://phabricator.wikimedia.org/T434014) [20:58:32] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Set storage compatability to UPGRADING — T433026 - eevans@cumin1003 [20:58:37] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [20:59:27] (03CR) 10Andrew Bogott: [C:03+2] Revert "backy2: increase max read rate to 120MB/second" [puppet] - 10https://gerrit.wikimedia.org/r/1326377 (owner: 10Andrew Bogott) [21:00:04] alexsanford, Reedy, sbassett, Maryum, and manfredi: That opportune time for a Weekly Security deployment window deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T2100). [21:00:23] I have two patches to deploy [21:03:29] (03PS1) 10Ladsgroup: Introduce main lock manager service [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326380 (https://phabricator.wikimedia.org/T366938) [21:04:21] (03CR) 10CI reject: [V:04-1] Introduce main lock manager service [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326380 (https://phabricator.wikimedia.org/T366938) (owner: 10Ladsgroup) [21:06:55] (03PS2) 10Ladsgroup: Introduce main lock manager service [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326380 (https://phabricator.wikimedia.org/T366938) [21:08:05] running scap for the first patch [21:08:16] James_F I know you're doing one of the deploys [21:08:19] maryum: I have one as well; please ping when you’re fully done. :-) [21:08:21] Ha, snap. [21:08:22] will do! [21:14:34] first patch done about to run scap for the second patch [21:14:37] !log Deployed security fix for T434967 [21:14:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:14:43] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:15:43] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:15:43] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1013.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:15:51] (03PS3) 10Ladsgroup: Introduce main lock manager service [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326380 (https://phabricator.wikimedia.org/T366938) [21:16:18] (03PS1) 10Ahmon Dancy: Merge remote-tracking branch 'origin/master' into train-dev [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1326384 [21:16:41] (03CR) 10Ahmon Dancy: [C:03+2] Merge remote-tracking branch 'origin/master' into train-dev [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1326384 (owner: 10Ahmon Dancy) [21:16:43] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:21:06] (03Merged) 10jenkins-bot: Merge remote-tracking branch 'origin/master' into train-dev [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1326384 (owner: 10Ahmon Dancy) [21:21:38] !log bking@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [21:21:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:21:57] !log bking@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [21:23:35] !log Deployed security fix for T433020 [21:23:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:24:09] James_F I'm done now! [21:25:23] I am noticing a spike in errors in logstash and am investigating whether that was related to my deploy [21:28:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [21:28:40] I'm going to go ahead rollback my patch [21:28:47] I think it's causing the error spike [21:30:57] James_F running a scap to undo my patch [21:32:35] Ack. [21:37:12] !log Undeploy security fix for T433020 [21:37:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:37:23] okay James_F you can go ahead [21:37:37] error rates have stopped spiking [21:38:07] maryum: There’s still untracked stuff in /srv/patches; should I wait and give you time to clear that up? [21:38:14] sorry yes doing that now!! [21:38:15] RESOLVED: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [21:38:21] give me one sec [21:39:43] James_F done! [21:39:54] !log swfrench@deploy1003 mwscript-k8s job started: purgeList.php # T432412 [21:39:59] T432412: [Eng] iOS - Update app site association file - https://phabricator.wikimedia.org/T432412 [21:45:43] maryum: OK if I push the WL patch publicly now? (No need to hold for later release.) [21:45:55] yes that's fine [21:46:24] !log jforrester Deployed security patch for T435085 [21:46:48] OK, I’m done at my end. [21:47:24] awesome [21:47:32] (03CR) 10Jsn.sherman: [C:03+1] PersonalDashboard: add newly renamed *ReviewChangesMlModel setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326373 (https://phabricator.wikimedia.org/T422148) (owner: 10MusikAnimal) [22:10:47] (03PS1) 10Ryan Kemper: cirrus: remove orphaned frozen writes command [puppet] - 10https://gerrit.wikimedia.org/r/1326393 (https://phabricator.wikimedia.org/T433306) [22:11:04] (03CR) 10Ryan Kemper: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1326393 (https://phabricator.wikimedia.org/T433306) (owner: 10Ryan Kemper) [22:24:39] FIRING: TransitBGPDown: Transit BGP session down between cr2-codfw and Hurricane Electric (2001:504:61::1b1b:0:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Transit6&var-bgp_neighbor=Hurricane+Electric - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [22:28:29] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Set storage compatability to UPGRADING — T433026 - eevans@cumin1003 [22:28:33] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [22:29:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr2-codfw and Hurricane Electric (2001:504:61::1b1b:0:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [22:39:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr2-codfw and Hurricane Electric (2001:504:61::1b1b:0:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [22:44:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr2-codfw and Hurricane Electric (2001:504:61::1b1b:0:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [22:59:18] (03CR) 10Ryan Kemper: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1326393 (https://phabricator.wikimedia.org/T433306) (owner: 10Ryan Kemper) [23:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260817T2300) [23:05:11] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-codfw: Set storage compatability to UPGRADING — T433026 - eevans@cumin1003 [23:05:16] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [23:21:20] (03PS1) 10Dduvall: drivers: Driver registration and factory [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1326398 (https://phabricator.wikimedia.org/T432829) [23:21:26] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [23:24:27] (03PS2) 10Dduvall: drivers: Driver registration and factory [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1326398 (https://phabricator.wikimedia.org/T432829) [23:26:25] (03PS1) 10Daimona Eaytoy: Stop setting wgCampaignEventsEnableWorklists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326400 (https://phabricator.wikimedia.org/T429510) [23:26:40] (03CR) 10Eevans: [C:03+2] aqs: set storage compatability mode to `NONE` [puppet] - 10https://gerrit.wikimedia.org/r/1324816 (https://phabricator.wikimedia.org/T433026) (owner: 10Eevans) [23:29:10] (03CR) 10CI reject: [V:04-1] drivers: Driver registration and factory [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1326398 (https://phabricator.wikimedia.org/T432829) (owner: 10Dduvall) [23:30:30] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Set storage compatability to NONE — T433026 - eevans@cumin1003 [23:30:35] T433026: Upgrade aqs cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T433026 [23:37:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:39:43] (03PS3) 10Dduvall: drivers: Driver registration and factory [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1326398 (https://phabricator.wikimedia.org/T432829) [23:41:18] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1326402 [23:41:19] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1326402 (owner: 10TrainBranchBot) [23:43:36] FIRING: [2x] ProbeDown: Service ganeti2046:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:44:52] (03CR) 10CI reject: [V:04-1] drivers: Driver registration and factory [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1326398 (https://phabricator.wikimedia.org/T432829) (owner: 10Dduvall) [23:50:08] (03PS4) 10Dduvall: drivers: Driver registration and factory [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1326398 (https://phabricator.wikimedia.org/T432829) [23:51:49] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1326402 (owner: 10TrainBranchBot)