[00:09:37] (03CR) 10Samwilson: [C:03+1] Tidy up resource and rounding leftovers [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333248 (owner: 10Ladsgroup-claude) [00:23:12] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [00:23:14] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1073.eqiad.wmnet with OS trixie [00:23:30] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12288104 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1073.eqiad.wmnet with OS trixie completed: -... [00:23:56] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1074.eqiad.wmnet with OS trixie [00:24:17] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12288105 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1074.eqiad.wmnet with OS trixie [00:30:38] FIRING: CertAlmostExpired: gNMI TLS certificate for ssw1-d1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [00:33:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:39:00] !log vriley@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1074.eqiad.wmnet with reason: host reimage [00:39:42] FIRING: [3x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [00:44:00] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1074.eqiad.wmnet with reason: host reimage [00:45:34] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12288114 (10VRiley-WMF) [00:46:59] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1065.eqiad.wmnet with OS trixie [00:47:16] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12288116 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1065.eqiad.wmnet with OS trixie [00:59:59] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [01:00:32] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [01:00:33] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1074.eqiad.wmnet with OS trixie [01:00:48] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12288152 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1074.eqiad.wmnet with OS trixie completed: -... [01:02:51] !log vriley@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage [01:08:36] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage [01:12:36] (03CR) 10Scott French: "Thanks for the review!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1328738 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [01:13:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:18:42] (03CR) 10Scott French: mesh: Support Host splits in configuration 1.16.0, networkpolicy 1.3.0 (033 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331858 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [01:19:04] (03PS4) 10Scott French: mesh: Support Host splits in configuration 1.16.0, networkpolicy 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331858 (https://phabricator.wikimedia.org/T427666) [01:19:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:24:20] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1065.eqiad.wmnet with OS trixie [01:24:38] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12288169 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1065.eqiad.wmnet with OS trixie completed: -... [01:24:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:31:08] (03PS5) 10Scott French: mesh: Support Host splits in configuration 1.16.0, networkpolicy 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331858 (https://phabricator.wikimedia.org/T427666) [01:41:41] (03CR) 10Scott French: "Thanks, Reuven!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331858 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [02:00:22] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:01:01] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 00m 39s) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:23:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:27:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [02:28:15] FIRING: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 4.397% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:32:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [02:37:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 0% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:40:15] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-ext releases routed via main (k8s) 2.5s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [02:47:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:52:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:53:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [02:58:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [02:59:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:01:13] FIRING: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [03:02:16] !incidents [03:02:17] 8318 (ACKED) PHPFPMTooBusy sre (mw-api-ext main eqiad) [03:02:17] 8319 (UNACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [03:03:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=drmrs - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:04:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:05:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:06:13] RESOLVED: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [03:06:15] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-internal-scholarly_443: Servers wdqs1027.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [03:07:15] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [03:08:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=drmrs - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:09:30] wrt the PHPFPMTooBusy page, I followed the steps that I could in the runbook (https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded) [03:10:07] but I wasn't able to ssh to any hosts in the k8s cluster (either I don't have perms or I wasn't doing it right) [03:10:12] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:10:15] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-ext releases routed via main (k8s) 2.175s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [03:10:54] lets discuss this in -security [03:12:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [03:12:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 2.982% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:13:15] RESOLVED: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 20.86% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:14:03] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [03:17:15] RESOLVED: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [03:18:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [03:20:44] (03PS1) 10Tryvix1509: Adjust extendedconfirmed calculation to first edit on viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) [03:21:36] (03CR) 10CI reject: [V:04-1] Adjust extendedconfirmed calculation to first edit on viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) (owner: 10Tryvix1509) [03:23:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [03:24:47] (03PS2) 10Tryvix1509: Adjust extendedconfirmed calculation to first edit on viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) [03:26:22] 06SRE, 10LDAP-Access-Requests: Grant Access to wmf, for WMF staff/contractors nda group for gsduser - https://phabricator.wikimedia.org/T435852#12288245 (10GSduser) Hi there - Requested access via https://idm.wikimedia.org/ but @Cpetrillo advised he does not think this will give me what we need and I'm about... [03:28:15] RESOLVED: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [03:31:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.15% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:33:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:47:36] (03PS2) 10Jasmine: docroot: replace wikimedia.org docroot symlink with standard-docroot [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335403 (https://phabricator.wikimedia.org/T427929) [03:51:16] (03PS6) 10Andrew Bogott: Add cloudvps-project-usage.py [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) [03:53:39] (03PS7) 10Andrew Bogott: Add cloudvps-project-usage.py [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) [03:53:40] (03CR) 10CI reject: [V:04-1] Add cloudvps-project-usage.py [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [03:53:45] (03CR) 10Andrew Bogott: Add cloudvps-project-usage.py (036 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [03:55:21] (03PS8) 10Andrew Bogott: Add cloudvps-project-usage.py [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) [03:57:56] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [04:04:03] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [04:12:56] (03PS5) 10Jasmine: apache: Configure wikimedia.org redirect in apache, remove from redirects.dat [puppet] - 10https://gerrit.wikimedia.org/r/1333298 (https://phabricator.wikimedia.org/T427929) [04:30:38] FIRING: CertAlmostExpired: gNMI TLS certificate for ssw1-d1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:39:42] FIRING: [3x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [05:11:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.87% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:11:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.49% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:18:35] ^ This looks bad, it almost paged. [05:19:20] The URL linked in the alert doesn't load for me. https://bit.ly/wmf-fpmsat [05:25:12] It reached 94% and pages at 95%, I want to help but so it won't page but the listed runbook doesn't load. 🙃 [05:26:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.55% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:29:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:40:41] (03PS1) 10Andrea Denisse: mw-on-k8s: Add runbook link using official URL shortener [alerts] - 10https://gerrit.wikimedia.org/r/1335636 (https://phabricator.wikimedia.org/T437010) [05:40:41] (03CR) 10Andrea Denisse: "Not sure who to add for review but git blame shows you modified the file." [alerts] - 10https://gerrit.wikimedia.org/r/1335636 (https://phabricator.wikimedia.org/T437010) (owner: 10Andrea Denisse) [05:40:46] (03CR) 10CI reject: [V:04-1] mw-on-k8s: Add runbook link using official URL shortener [alerts] - 10https://gerrit.wikimedia.org/r/1335636 (https://phabricator.wikimedia.org/T437010) (owner: 10Andrea Denisse) [05:46:08] (03PS2) 10Andrea Denisse: mw-on-k8s: Add runbook link using official URL shortener [alerts] - 10https://gerrit.wikimedia.org/r/1335636 (https://phabricator.wikimedia.org/T437010) [05:46:56] (03CR) 10Dragoniez: "I would personally keep this as a one-line change at #L-4743. The additional stylistic changes aren't necessary for the functional fix, an" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) (owner: 10Tryvix1509) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260904T0600) [06:14:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.8% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:15:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:20:45] !log filippo@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1065 [06:21:23] !log filippo@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1065 [06:21:49] !log filippo@cumin1003 START - Cookbook sre.dns.netbox [06:28:45] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1065 cloud-private - filippo@cumin1003" [06:28:49] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1065 cloud-private - filippo@cumin1003" [06:28:50] !log filippo@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [06:30:08] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12288392 (10fgiunchedi) [06:31:11] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12288393 (10fgiunchedi) a:05VRiley-WMF→03fgiunchedi Hosts are reimaged and I have updated netbox with vlan assignments. Taking the task back to finish implemen... [06:34:52] (03CR) 10Muehlenhoff: [C:03+2] Reimage ldap-rw[12]001 [puppet] - 10https://gerrit.wikimedia.org/r/1334906 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [06:40:40] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host ldap-rw1001.wikimedia.org with OS trixie [06:40:53] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12288401 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host ldap-rw1001.wikimedia.org with OS tr... [06:45:00] 06SRE, 10LDAP-Access-Requests: Grant Access to wmf, for WMF staff/contractors nda group for gsduser - https://phabricator.wikimedia.org/T435852#12288407 (10Aklapper) @GSduser: Can you please explicitly confirm that you have already tried and that you lack access to data you need? Also, please [link your LDAP... [06:52:50] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-rw1001.wikimedia.org with reason: host reimage [06:57:39] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-rw1001.wikimedia.org with reason: host reimage [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260904T0700) [07:00:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:02:52] PROBLEM - Host db2196 #page is DOWN: PING CRITICAL - Packet loss = 100% [07:03:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:03:20] marostegui: ^ ? [07:03:28] !incidents [07:03:28] 8320 (UNACKED) Host db2196 (paged) [07:03:28] 8318 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main eqiad) [07:03:28] 8319 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [07:03:30] checking [07:03:31] !ack [07:03:32] 8320 (ACKED) Host db2196 (paged) [07:03:46] x1 master codfw, nice [07:03:47] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12288435 (10Aklapper) @Geertivp: What exactly happens when you create the account (with the email address which is an admin of that list) on the lists.wikimedia.org website? [07:04:01] I was going to switch it on monday: https://phabricator.wikimedia.org/T436922 [07:04:04] I guess I will do it now [07:04:16] (03Abandoned) 10Giuseppe Lavagetto: haproxy: stop current abuse [puppet] - 10https://gerrit.wikimedia.org/r/1243884 (owner: 10Giuseppe Lavagetto) [07:04:52] for what it's worth, switch port is up, but not receiving any traffic from the host [07:05:00] okay, let me know if I can support with anything [07:05:00] The task you linked is about db2191 and not 2196? [07:05:02] XioNoX: uptime is 42 days btw [07:05:16] jelto: ? [07:05:34] <_joe_> db2196 is the host that is down [07:05:39] <_joe_> not 2191 [07:05:42] XioNoX: ah sorry [07:05:44] right [07:05:44] T436922 is about db2191 [07:05:45] T436922: Switchover x1 master (db2191 -> db2249) - https://phabricator.wikimedia.org/T436922 [07:05:57] so it is a slave [07:06:17] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db2196: down [07:06:18] https://zarcillo.wikimedia.org/api/v1/can_be_depooled/db2196 says it can be depooled, so probably not *that* important :) [07:06:28] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2196: down [07:06:32] Just depooled [07:06:35] <3 [07:07:14] <_joe_> XioNoX: zarcillo says it cannot be depooled actually [07:07:33] I just depooled it [07:07:44] <_joe_> reason "Non-replica instances present" [07:07:46] <_joe_> ^_^ [07:07:52] Before the depool the url returned `{"hostname":"db2196","can_depool":true,"reason":""}`, now it says `{"hostname":"db2196","can_depool":false,"reason":"Non-replica instances present"}` [07:08:25] that makes more sense, thank you for handling the depool [07:08:41] https://phabricator.wikimedia.org/T437015 [07:10:13] (03CR) 10Filippo Giunchedi: "LGTM! See inline for comments" [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [07:13:59] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-rw1001.wikimedia.org with OS trixie [07:14:11] 06SRE, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12288464 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host ldap-rw1001.wikimedia.org with OS trixie... [07:15:21] RECOVERY - Host db2196 #page is UP: PING OK - Packet loss = 0%, RTA = 33.03 ms [07:16:23] PROBLEM - MariaDB Replica IO: x1 #page on db2196 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [07:16:24] PROBLEM - MariaDB Events x1 on db2196 is CRITICAL: CRITICAL - Failed to query events: ERROR 2002 (HY000): Cant connect to local server through socket /run/mysqld/mysqld.sock (2) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [07:16:25] PROBLEM - MariaDB Replica SQL: x1 #page on db2196 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [07:16:25] PROBLEM - mysqld processes on db2196 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [07:16:31] !incidents [07:16:31] 8321 (UNACKED) db2196 (paged)/MariaDB Replica IO: x1 (paged) [07:16:31] 8322 (UNACKED) db2196 (paged)/MariaDB Replica SQL: x1 (paged) [07:16:31] 8320 (RESOLVED) Host db2196 (paged) [07:16:32] 8318 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main eqiad) [07:16:32] 8319 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [07:16:37] !ack [07:16:38] 8321 (ACKED) db2196 (paged)/MariaDB Replica IO: x1 (paged) [07:16:38] 8322 (ACKED) db2196 (paged)/MariaDB Replica SQL: x1 (paged) [07:18:00] PROBLEM - MariaDB read only x1 on db2196 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [07:18:00] PROBLEM - MariaDB Event Scheduler x1 on db2196 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [07:18:23] db2196 just came back, uptime 3 minutes [07:23:15] 10ops-codfw, 06DBA, 06DC-Ops: db2196 crashed - https://phabricator.wikimedia.org/T437015#12288471 (10Marostegui) This host is running the Intel Xeon 5317 and rebooted itself. It was bought: 2024-01-22 so I guess still under warranty? Nothing on `getsel` and this is `getraclog`: ` --------------------------... [07:23:24] RECOVERY - mysqld processes on db2196 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [07:23:25] PROBLEM - MariaDB Replica Lag: x1 #page on db2196 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [07:23:37] !incidents [07:23:37] 8321 (ACKED) db2196 (paged)/MariaDB Replica IO: x1 (paged) [07:23:38] 8322 (ACKED) db2196 (paged)/MariaDB Replica SQL: x1 (paged) [07:23:38] 8323 (UNACKED) db2196 (paged)/MariaDB Replica Lag: x1 (paged) [07:23:38] 8320 (RESOLVED) Host db2196 (paged) [07:23:38] 8318 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main eqiad) [07:23:38] 8319 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [07:23:43] !ack [07:23:43] 8323 (ACKED) db2196 (paged)/MariaDB Replica Lag: x1 (paged) [07:24:02] RECOVERY - MariaDB Event Scheduler x1 on db2196 is OK: Version 10.11.16-MariaDB-log, Uptime 43s, read_only: True, event_scheduler: True, 4682.43 QPS, connection latency: 0.025142s, query latency: 0.000766s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [07:24:02] RECOVERY - MariaDB read only x1 on db2196 is OK: Version 10.11.16-MariaDB-log, Uptime 43s, read_only: True, event_scheduler: True, 4712.95 QPS, connection latency: 0.026996s, query latency: 0.000738s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [07:24:23] RECOVERY - MariaDB Replica IO: x1 #page on db2196 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [07:24:25] RECOVERY - MariaDB Replica Lag: x1 #page on db2196 is OK: OK slave_sql_lag Replication lag: 0.01 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [07:24:26] RECOVERY - MariaDB Replica SQL: x1 #page on db2196 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [07:24:26] RECOVERY - MariaDB Events x1 on db2196 is OK: OK - All 4 events in ops database are ENABLED https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [07:24:48] (03CR) 10Brouberol: [C:03+1] matomo: Restore GeoIP functionality [puppet] - 10https://gerrit.wikimedia.org/r/1335153 (https://phabricator.wikimedia.org/T431608) (owner: 10Bking) [07:29:08] (03PS1) 10Hamish: thwikibooks: update tagline and wordmark [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335706 (https://phabricator.wikimedia.org/T436426) [07:33:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:36:10] (03PS1) 10Trueg: WDQS: Simplification of qlever proxy values [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335712 (https://phabricator.wikimedia.org/T429175) [07:36:37] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 07 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335706 (https://phabricator.wikimedia.org/T436426) (owner: 10Hamish) [07:46:01] (03PS2) 10Trueg: WDQS: Simplification of qlever proxy values [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335712 (https://phabricator.wikimedia.org/T429175) [07:46:22] (03PS1) 10Marostegui: installserver: Do not reimage db1276 [puppet] - 10https://gerrit.wikimedia.org/r/1335717 [07:50:26] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:50:59] (03CR) 10Marostegui: [C:03+2] installserver: Do not reimage db1276 [puppet] - 10https://gerrit.wikimedia.org/r/1335717 (owner: 10Marostegui) [07:51:56] !log elukey@cumin1003 END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:52:15] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:53:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:55:14] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host ldap-rw2001.wikimedia.org with OS trixie [07:55:25] 06SRE, 06Infrastructure-Foundations, 07LDAP: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12288496 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host ldap-rw2001.wikimedia.org with OS trixie [07:55:59] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:56:02] !log ayounsi@cumin1003 START - Cookbook sre.hosts.reimage for host netflow5003.eqsin.wmnet with OS trixie [07:57:29] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:59:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:02:00] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [08:02:30] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [08:03:09] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply [08:03:19] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:03:42] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply [08:04:30] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cassandra-dev2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:04:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:06:18] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:07:53] !log btullis@deploy1003 Started scap sync-world: Trying again for T436913 [08:07:56] T436913: 7za is OOMkilled when dumps are running on a Bookworm-based image - https://phabricator.wikimedia.org/T436913 [08:08:20] !log elukey@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cassandra-dev2003.codfw.wmnet [08:11:25] elukey@cumin1003 upgrade-firmware (PID 2616154) is awaiting input [08:12:48] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-rw2001.wikimedia.org with reason: host reimage [08:13:29] (03PS1) 10Krinkle: Fix empty bases in m(under|over) [extensions/Math] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335752 (https://phabricator.wikimedia.org/T436876) [08:13:53] FIRING: FNMNotReported: FastNetMon metrics not reported - https://wikitech.wikimedia.org/wiki/Fastnetmon - https://w.wiki/8oU - https://alerts.wikimedia.org/?q=alertname%3DFNMNotReported [08:14:29] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.4 point update - https://phabricator.wikimedia.org/T420240#12288522 (10MoritzMuehlenhoff) [08:17:51] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-rw2001.wikimedia.org with reason: host reimage [08:20:14] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cassandra-dev2003.codfw.wmnet [08:23:14] (03PS1) 10Marostegui: db2196: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1335753 (https://phabricator.wikimedia.org/T437015) [08:24:36] !log elukey@cumin1003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cassandra-dev2003.codfw.wmnet [08:25:10] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host cassandra-dev2003.codfw.wmnet [08:25:19] (03CR) 10Marostegui: [C:03+2] db2196: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1335753 (https://phabricator.wikimedia.org/T437015) (owner: 10Marostegui) [08:26:10] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db2196.codfw.wmnet with reason: Host crashed [08:26:21] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops, 13Patch-For-Review: db2196 crashed - https://phabricator.wikimedia.org/T437015#12288573 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=08c74896-bbc0-4ca7-a130-5478cd705d71) set by marostegui@cumin1003 for 4 days, 0:00:00 on 1 host(s) and their ser... [08:26:54] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops, 13Patch-For-Review: db2196 crashed - https://phabricator.wikimedia.org/T437015#12288580 (10Marostegui) @Jhancock.wm this host has mariadb stopped and it is downtimed, so if you need to reboot it to update firmware, please go ahead. Also, can we add this host to the De... [08:28:14] 06SRE, 10SRE-Access-Requests: Requesting access to production for derenrich - https://phabricator.wikimedia.org/T436965#12288601 (10Volans) p:05Triage→03Medium [08:29:02] (03PS1) 10Krinkle: tests: Add user_id and actor in testLogDatabaseRowsForHiddenUser [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335754 (https://phabricator.wikimedia.org/T436971) [08:30:38] FIRING: CertAlmostExpired: gNMI TLS certificate for ssw1-d1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:33:31] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-rw2001.wikimedia.org with OS trixie [08:33:37] 06SRE, 06Infrastructure-Foundations, 07LDAP: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12288619 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host ldap-rw2001.wikimedia.org with OS trixie completed: - ldap-rw2... [08:34:40] 06SRE, 10SRE-Access-Requests: Requesting access to production for derenrich - https://phabricator.wikimedia.org/T436965#12288622 (10Volans) @derenrich that's the key's fingerprint, we need the actual public key, you can find the public key in the `.pub` file created when generating the new key. Adding @thcipr... [08:35:12] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cassandra-dev2003.codfw.wmnet [08:35:14] !log elukey@cumin1003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cassandra-dev2003.codfw.wmnet [08:35:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.33% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:35:56] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm [08:36:50] (03CR) 10Elukey: [C:03+1] Add pki[12]003 to site.pp and preseed [puppet] - 10https://gerrit.wikimedia.org/r/1334986 (https://phabricator.wikimedia.org/T436903) (owner: 10Muehlenhoff) [08:39:02] (03CR) 10Muehlenhoff: [C:03+2] Add pki[12]003 to site.pp and preseed [puppet] - 10https://gerrit.wikimedia.org/r/1334986 (https://phabricator.wikimedia.org/T436903) (owner: 10Muehlenhoff) [08:39:42] FIRING: [3x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [08:40:41] !log btullis@deploy1003 Finished scap sync-world: Trying again for T436913 (duration: 34m 26s) [08:40:45] T436913: 7za is OOMkilled when dumps are running on a Bookworm-based image - https://phabricator.wikimedia.org/T436913 [08:45:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.74% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:45:36] !log ayounsi@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on netflow5003.eqsin.wmnet with reason: host reimage [08:48:15] (03CR) 10Brouberol: [C:03+1] "Yes please!" [dumps] - 10https://gerrit.wikimedia.org/r/1334846 (https://phabricator.wikimedia.org/T436913) (owner: 10Btullis) [08:48:37] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow5003.eqsin.wmnet with reason: host reimage [08:54:16] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage [08:54:23] RESOLVED: [3x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [08:55:24] !log jmm@cumin1004 START - Cookbook sre.ganeti.makevm for new host pki1003.eqiad.wmnet [08:55:26] !log jmm@cumin1004 START - Cookbook sre.dns.netbox [08:58:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:58:55] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage [08:59:22] (03PS1) 10Marostegui: build-mariadb-deb: Adding to the repo [software] - 10https://gerrit.wikimedia.org/r/1335765 [08:59:43] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM pki1003.eqiad.wmnet - jmm@cumin1004" [08:59:46] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM pki1003.eqiad.wmnet - jmm@cumin1004" [08:59:46] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:59:47] !log jmm@cumin1004 START - Cookbook sre.dns.wipe-cache pki1003.eqiad.wmnet on all recursors [08:59:50] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki1003.eqiad.wmnet on all recursors [08:59:59] (03CR) 10Marostegui: [C:03+2] build-mariadb-deb: Adding to the repo [software] - 10https://gerrit.wikimedia.org/r/1335765 (owner: 10Marostegui) [09:00:24] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM pki1003.eqiad.wmnet - jmm@cumin1004" [09:00:27] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM pki1003.eqiad.wmnet - jmm@cumin1004" [09:02:52] !log jmm@cumin1004 START - Cookbook sre.hosts.reimage for host pki1003.eqiad.wmnet with OS trixie [09:03:12] 06SRE, 06Infrastructure-Foundations, 10vm-requests, 13Patch-For-Review: Two VMs for the PKI infra - https://phabricator.wikimedia.org/T436903#12288745 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host pki1003.eqiad.wmnet with OS trixie [09:03:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:07:42] (03CR) 10Muehlenhoff: "I like the idea! But I had a closer look at the reimage cookbook and --force also controls a few other things than just the ack to proceed" [cookbooks] - 10https://gerrit.wikimedia.org/r/1334770 (owner: 10Ayounsi) [09:08:53] RESOLVED: FNMNotReported: FastNetMon metrics not reported - https://wikitech.wikimedia.org/wiki/Fastnetmon - https://w.wiki/8oU - https://alerts.wikimedia.org/?q=alertname%3DFNMNotReported [09:09:25] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host netflow5003.eqsin.wmnet with OS trixie [09:13:07] (03PS1) 10Trueg: WDQS: Improved resource limits for Qlever on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335771 (https://phabricator.wikimedia.org/T436295) [09:17:46] !log jmm@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on pki1003.eqiad.wmnet with reason: host reimage [09:18:37] (03CR) 10Muehlenhoff: "Sudo changes need to be approved in the weekly SRE IF meeting (the next one is Monday), I'll add it to the agenda and will followup when t" [puppet] - 10https://gerrit.wikimedia.org/r/1319853 (https://phabricator.wikimedia.org/T421952) (owner: 10Brouberol) [09:18:39] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm [09:23:43] (03PS2) 10Elukey: docker_registry: move away from Swift completely [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) [09:23:43] (03PS1) 10Elukey: docker_registry: add Prometheus config for all Distribution endpoints [puppet] - 10https://gerrit.wikimedia.org/r/1335775 (https://phabricator.wikimedia.org/T427175) [09:23:45] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pki1003.eqiad.wmnet with reason: host reimage [09:25:13] (03PS2) 10Elukey: docker_registry: add Prometheus config for all Distribution endpoints [puppet] - 10https://gerrit.wikimedia.org/r/1335775 (https://phabricator.wikimedia.org/T427175) [09:25:13] (03PS3) 10Elukey: docker_registry: move away from Swift completely [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) [09:25:54] (03CR) 10CI reject: [V:04-1] docker_registry: add Prometheus config for all Distribution endpoints [puppet] - 10https://gerrit.wikimedia.org/r/1335775 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [09:26:15] (03CR) 10CI reject: [V:04-1] docker_registry: move away from Swift completely [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) (owner: 10Elukey) [09:26:18] (03PS1) 10Trueg: WDQS: Do not keep any index backups on the PVC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335776 (https://phabricator.wikimedia.org/T436802) [09:26:33] (03CR) 10CI reject: [V:04-1] docker_registry: add Prometheus config for all Distribution endpoints [puppet] - 10https://gerrit.wikimedia.org/r/1335775 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [09:27:25] (03PS3) 10Elukey: docker_registry: add Prometheus config for all Distribution endpoints [puppet] - 10https://gerrit.wikimedia.org/r/1335775 (https://phabricator.wikimedia.org/T427175) [09:27:25] (03PS4) 10Elukey: docker_registry: move away from Swift completely [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) [09:28:07] (03CR) 10Elukey: "I noticed https://phabricator.wikimedia.org/T384321, so maybe this could be an occasion to move to a blackbox exporter. Thoughts Tiziano?" [puppet] - 10https://gerrit.wikimedia.org/r/1335775 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [09:28:38] (03PS2) 10Ayounsi: makevm: add --force to reimage cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1334770 [09:29:08] (03CR) 10Brouberol: "I think we can abandon this, as we've ensure the sorting order at the chart level itself https://gerrit.wikimedia.org/r/c/operations/deplo" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333897 (https://phabricator.wikimedia.org/T435634) (owner: 10Ladsgroup) [09:30:50] (03PS1) 10Marostegui: mariadb: Decommission db1182 [puppet] - 10https://gerrit.wikimedia.org/r/1335778 (https://phabricator.wikimedia.org/T434869) [09:31:02] (03CR) 10Brouberol: [C:03+1] kartotherian: remove duplicate key [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321523 (owner: 10Mvolz) [09:31:25] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1182: Decommissioning [09:32:51] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1182: Decommissioning [09:39:45] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pki1003.eqiad.wmnet with OS trixie [09:39:45] !log jmm@cumin1004 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host pki1003.eqiad.wmnet [09:39:53] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Two VMs for the PKI infra - https://phabricator.wikimedia.org/T436903#12288817 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host pki1003.eqiad.wmnet with OS trixie completed: - pki1003 (**PASS**) - Removed from... [09:41:53] !log marostegui@cumin1003 START - Cookbook sre.hosts.decommission for hosts db1182.eqiad.wmnet [09:42:23] (03CR) 10Btullis: [C:03+2] 7za: Use cgroups to determine the maximum RAM and number of threads [dumps] - 10https://gerrit.wikimedia.org/r/1334846 (https://phabricator.wikimedia.org/T436913) (owner: 10Btullis) [09:42:29] (03CR) 10Marostegui: [C:03+2] mariadb: Decommission db1182 [puppet] - 10https://gerrit.wikimedia.org/r/1335778 (https://phabricator.wikimedia.org/T434869) (owner: 10Marostegui) [09:44:09] (03Merged) 10jenkins-bot: 7za: Use cgroups to determine the maximum RAM and number of threads [dumps] - 10https://gerrit.wikimedia.org/r/1334846 (https://phabricator.wikimedia.org/T436913) (owner: 10Btullis) [09:45:28] !log marostegui@cumin1003 dbctl commit (dc=all): 'Remove db1182 from dbctl T434869', diff saved to https://phabricator.wikimedia.org/P96346 and previous config saved to /var/cache/conftool/dbconfig/20260904-094527-marostegui.json [09:45:31] T434869: decommission db1182.eqiad.wmnet - https://phabricator.wikimedia.org/T434869 [09:46:30] (03PS4) 10Elukey: docker_registry: migrate nrpe checks to alertmanager [puppet] - 10https://gerrit.wikimedia.org/r/1306351 (https://phabricator.wikimedia.org/T384321) (owner: 10Hnowlan) [09:46:43] !log marostegui@cumin1003 START - Cookbook sre.dns.netbox [09:50:13] (03CR) 10Elukey: [C:03+2] "Yep let's keep them!" [puppet] - 10https://gerrit.wikimedia.org/r/1306351 (https://phabricator.wikimedia.org/T384321) (owner: 10Hnowlan) [09:50:40] !log marostegui@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1182.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" [09:51:11] !log marostegui@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1182.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" [09:51:11] !log marostegui@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:51:12] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1182.eqiad.wmnet [09:52:23] FIRING: GnmiInterfaceCountersDrop: ... [09:52:28] pfw1-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=pfw1-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [09:53:17] !log marostegui@cumin1003 START - Cookbook sre.mysql.decommission [09:53:17] !log marostegui@cumin1003 END (ERROR) - Cookbook sre.mysql.decommission (exit_code=1) [09:53:32] !log marostegui@cumin1003 START - Cookbook sre.mysql.decommission [09:53:46] !log marostegui@cumin1003 START - Cookbook sre.hosts.decommission for hosts db1182.eqiad.wmnet [09:57:01] !log btullis@deploy1003 Started scap sync-world: Incorporating changes from https://gerrit.wikimedia.org/r/c/operations/dumps/+/1334846 to mediawiki-cli [09:57:22] FIRING: [7x] GnmiInterfaceCountersDrop: cloudsw1-e4-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [09:57:49] !log marostegui@cumin1003 START - Cookbook sre.dns.netbox [09:57:49] (03CR) 10Dreamy Jazz: "Not sure why this needs a wmf.18 backport? Is the MediaWiki core change also being backported?" [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335754 (https://phabricator.wikimedia.org/T436971) (owner: 10Krinkle) [09:59:55] (03PS4) 10Elukey: docker_registry: add Prometheus config for all Distribution endpoints [puppet] - 10https://gerrit.wikimedia.org/r/1335775 (https://phabricator.wikimedia.org/T427175) [09:59:55] (03PS5) 10Elukey: docker_registry: move away from Swift completely [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) [10:00:01] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [10:00:14] !log marostegui@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:00:15] !log marostegui@cumin1003 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts db1182.eqiad.wmnet [10:00:22] !log marostegui@cumin1003 Removing db1182 from zarcillo T434869 [10:00:26] T434869: decommission db1182.eqiad.wmnet - https://phabricator.wikimedia.org/T434869 [10:00:27] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) [10:00:29] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1182.eqiad.wmnet - https://phabricator.wikimedia.org/T434869#12288889 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1003 for hosts: `db1182.eqiad.wmnet` - db1182.eqiad.wmnet (**FAIL**) - //Missing... [10:00:32] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1182.eqiad.wmnet - https://phabricator.wikimedia.org/T434869#12288890 (10ops-monitoring-bot) db1182 has been deleted from zarcillo [10:00:35] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1182.eqiad.wmnet - https://phabricator.wikimedia.org/T434869#12288892 (10ops-monitoring-bot) db1182 has been decommissioned by Data Persistence [10:00:38] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1182.eqiad.wmnet - https://phabricator.wikimedia.org/T434869#12288893 (10ops-monitoring-bot) a:05Marostegui→03None This host is ready for DC-Ops to decommission [10:02:53] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 1 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [10:04:53] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [10:06:50] (03CR) 10Krinkle: "Yes, planning to on Monday." [extensions/CheckUser] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1335754 (https://phabricator.wikimedia.org/T436971) (owner: 10Krinkle) [10:12:27] RESOLVED: [7x] GnmiInterfaceCountersDrop: cloudsw1-e4-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [10:15:27] (03CR) 10Tiziano Fogli: [C:03+1] docker_registry: add Prometheus config for all Distribution endpoints [puppet] - 10https://gerrit.wikimedia.org/r/1335775 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [10:17:20] (03CR) 10Elukey: [C:03+2] docker_registry: add Prometheus config for all Distribution endpoints [puppet] - 10https://gerrit.wikimedia.org/r/1335775 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [10:22:53] FIRING: [15x] GnmiInterfaceCountersDrop: cloudsw1-b1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [10:23:07] RESOLVED: [7x] GnmiInterfaceCountersDrop: cloudsw1-e4-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [10:25:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.36% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:27:53] FIRING: [24x] GnmiInterfaceCountersDrop: cloudsw1-b1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [10:29:23] !log btullis@deploy1003 Finished scap sync-world: Incorporating changes from https://gerrit.wikimedia.org/r/c/operations/dumps/+/1334846 to mediawiki-cli (duration: 41m 14s) [10:30:16] (03CR) 10Clément Goubert: "We haven't decided on whether we're renaming or duplicating and deprecating yet, so for using python-webapp is fine." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [10:31:46] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1334856 (owner: 10L10n-bot) [10:35:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:43:40] (03PS2) 10JavierMonton: stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333854 (https://phabricator.wikimedia.org/T431555) [10:44:43] !log jmm@cumin1004 START - Cookbook sre.ganeti.makevm for new host pki2003.codfw.wmnet [10:44:45] !log jmm@cumin1004 START - Cookbook sre.dns.netbox [10:46:32] (03CR) 10Muehlenhoff: [C:03+1] "Looks good. I can test the "host exists" logic later with a harmless VM" [cookbooks] - 10https://gerrit.wikimedia.org/r/1334770 (owner: 10Ayounsi) [10:47:52] FIRING: [2x] GnmiInterfaceCountersDrop: lsw1-b3-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [10:49:30] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM pki2003.codfw.wmnet - jmm@cumin1004" [10:49:33] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM pki2003.codfw.wmnet - jmm@cumin1004" [10:49:33] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:49:33] !log jmm@cumin1004 START - Cookbook sre.dns.wipe-cache pki2003.codfw.wmnet on all recursors [10:49:37] !log jmm@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) pki2003.codfw.wmnet on all recursors [10:50:11] !log jmm@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM pki2003.codfw.wmnet - jmm@cumin1004" [10:50:15] !log jmm@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM pki2003.codfw.wmnet - jmm@cumin1004" [10:52:21] !log jmm@cumin1004 START - Cookbook sre.hosts.reimage for host pki2003.codfw.wmnet with OS trixie [10:52:33] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Two VMs for the PKI infra - https://phabricator.wikimedia.org/T436903#12289039 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1004 for host pki2003.codfw.wmnet with OS trixie [10:52:53] RESOLVED: [2x] GnmiInterfaceCountersDrop: lsw1-b3-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [10:54:43] (03PS1) 10Dpogorzelski: lw studio: add dedicated pgsql [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335793 [10:57:07] (03CR) 10CI reject: [V:04-1] lw studio: add dedicated pgsql [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335793 (owner: 10Dpogorzelski) [11:00:01] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260904T0700) [11:00:05] jelto, arnoldokoth, mutante, and arnaudb: GitLab version upgrades (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260904T1100). Please do the needful. [11:01:32] (03PS2) 10Dpogorzelski: lw studio: add dedicated pgsql [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335793 [11:03:33] (03CR) 10CI reject: [V:04-1] lw studio: add dedicated pgsql [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335793 (owner: 10Dpogorzelski) [11:05:25] 10SRE-tools, 06Infrastructure-Foundations: Upgrade BGPAlerter to 1.33 - https://phabricator.wikimedia.org/T354998#12289072 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff Our bgpalerter is on 1.33 [11:06:19] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a5-eqiad [11:06:40] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a5-eqiad [11:09:27] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12289091 (10MoritzMuehlenhoff) [11:10:27] (03CR) 10Clément Goubert: apache: Configure wikimedia.org redirect in apache, remove from redirects.dat (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1333298 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [11:10:33] (03PS3) 10Dpogorzelski: lw studio: add dedicated pgsql [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335793 [11:13:22] (03CR) 10Clément Goubert: [C:03+1] "Heck yeah \o/" [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) (owner: 10Elukey) [11:13:23] !log jmm@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on pki2003.codfw.wmnet with reason: host reimage [11:19:15] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pki2003.codfw.wmnet with reason: host reimage [11:20:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.1% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:25:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.1% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:32:55] !log jmm@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pki2003.codfw.wmnet with OS trixie [11:32:55] !log jmm@cumin1004 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host pki2003.codfw.wmnet [11:33:02] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Two VMs for the PKI infra - https://phabricator.wikimedia.org/T436903#12289192 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1004 for host pki2003.codfw.wmnet with OS trixie completed: - pki2003 (**PASS**) - Removed from... [11:33:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:36:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:36:33] !log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm [11:36:47] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12289200 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host cloudcephosd1055.eqiad.... [11:41:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:47:30] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a6-eqiad [11:47:48] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a6-eqiad [12:01:28] !log ayounsi@cumin1003 START - Cookbook sre.hosts.reimage for host netflow3004.esams.wmnet with OS trixie [12:01:45] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1182.eqiad.wmnet - https://phabricator.wikimedia.org/T434869#12289307 (10Jclark-ctr) a:03Jclark-ctr [12:07:52] (03PS3) 10Ayounsi: makevm: add --force to reimage cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1334770 [12:11:14] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a7-eqiad [12:11:35] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a7-eqiad [12:11:55] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Two VMs for the PKI infra - https://phabricator.wikimedia.org/T436903#12289349 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff VMs are created [12:12:49] (03PS1) 10Btullis: Add the new dse-k8s-worker nodes in eqiad to site.pp and preseed.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1335817 (https://phabricator.wikimedia.org/T436964) [12:13:57] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Q1:rack/setup/install dse-k8s-worker10[29-38] - https://phabricator.wikimedia.org/T436964#12289358 (10BTullis) [12:15:32] !log ayounsi@cumin1003 START - Cookbook sre.network.tls for network device ssw1-d1-codfw [12:15:49] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d1-codfw [12:16:28] (03PS2) 10Btullis: Add the new dse-k8s-worker nodes in eqiad to site.pp and preseed.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1335817 (https://phabricator.wikimedia.org/T436964) [12:16:41] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12289361 (10MoritzMuehlenhoff) [12:17:11] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12289362 (10MoritzMuehlenhoff) [12:18:36] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12289366 (10Geertivp) I am now a "member" without any rights. [12:18:53] FIRING: FNMNotReported: FastNetMon metrics not reported - https://wikitech.wikimedia.org/wiki/Fastnetmon - https://w.wiki/8oU - https://alerts.wikimedia.org/?q=alertname%3DFNMNotReported [12:19:50] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1182.eqiad.wmnet - https://phabricator.wikimedia.org/T434869#12289369 (10Jclark-ctr) 05Open→03Resolved [12:20:54] (03PS1) 10Muehlenhoff: Setup a syncrepl cluster on trixie/MDB [puppet] - 10https://gerrit.wikimedia.org/r/1335825 (https://phabricator.wikimedia.org/T331699) [12:21:54] (03CR) 10Ayounsi: "Tested in https://www.irccloud.com/pastebin/VHIXQUtF/, just need to test the --force now." [cookbooks] - 10https://gerrit.wikimedia.org/r/1334770 (owner: 10Ayounsi) [12:24:32] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335825 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [12:28:00] (03PS1) 10Muehlenhoff: New r/o LDAP replicas on Trixie/MDB [puppet] - 10https://gerrit.wikimedia.org/r/1335830 (https://phabricator.wikimedia.org/T331699) [12:28:54] !log ayounsi@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on netflow3004.esams.wmnet with reason: host reimage [12:29:06] (03PS2) 10Muehlenhoff: Setup a syncrepl cluster on trixie/MDB [puppet] - 10https://gerrit.wikimedia.org/r/1335825 (https://phabricator.wikimedia.org/T331699) [12:30:38] FIRING: CertAlmostExpired: gNMI TLS certificate for ssw1-d1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:31:23] (03PS1) 10Tiziano Fogli: bento: add container image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1335831 [12:33:57] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow3004.esams.wmnet with reason: host reimage [12:34:18] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335825 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [12:35:07] (03CR) 10Elukey: [C:03+1] "Left a nit for the ownership, the rest looks good!" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1335831 (owner: 10Tiziano Fogli) [12:35:54] (03CR) 10Elukey: [C:03+2] "After a chat with Arzhel we decided to merge, even if we weren't able to make it work on Junipers (they expose only the leaf, not caring a" [cookbooks] - 10https://gerrit.wikimedia.org/r/1334793 (https://phabricator.wikimedia.org/T375513) (owner: 10Elukey) [12:36:22] (03PS1) 10Btullis: Add the upstream ceph-squid repository bookworm-wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/1335835 (https://phabricator.wikimedia.org/T428445) [12:38:20] (03CR) 10Muehlenhoff: [C:03+1] "Anything using libkrb from MIT Kerberos will use the KDCs listed in krb.conf in round-robin, but it seems the Java clients we predominantl" [puppet] - 10https://gerrit.wikimedia.org/r/1335159 (https://phabricator.wikimedia.org/T435873) (owner: 10Bking) [12:39:11] (03CR) 10Ladsgroup: "Ah okay. Thanks!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333897 (https://phabricator.wikimedia.org/T435634) (owner: 10Ladsgroup) [12:39:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:39:45] (03Abandoned) 10Ladsgroup: turnilo: Order entries alphabetically [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333897 (https://phabricator.wikimedia.org/T435634) (owner: 10Ladsgroup) [12:39:51] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [12:41:26] (03CR) 10Gehel: "LGTM in principle, good that we configure the link to upstream repo. This is a new component, so the risk is super low." [puppet] - 10https://gerrit.wikimedia.org/r/1335835 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [12:44:08] 10ops-eqiad, 06SRE, 06DC-Ops: Broken disk on maps-test2001 - https://phabricator.wikimedia.org/T437036 (10MoritzMuehlenhoff) 03NEW [12:44:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:45:28] (03PS2) 10Tiziano Fogli: bento: add container image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1335831 [12:48:01] (03CR) 10Tiziano Fogli: [C:03+2] bento: add container image (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1335831 (owner: 10Tiziano Fogli) [12:48:05] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] bento: add container image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1335831 (owner: 10Tiziano Fogli) [12:48:53] RESOLVED: FNMNotReported: FastNetMon metrics not reported - https://wikitech.wikimedia.org/wiki/Fastnetmon - https://w.wiki/8oU - https://alerts.wikimedia.org/?q=alertname%3DFNMNotReported [12:49:40] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host netflow3004.esams.wmnet with OS trixie [12:56:14] !log ayounsi@cumin1003 START - Cookbook sre.hosts.reimage for host netflow1002.eqiad.wmnet with OS trixie [13:03:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-esams (185.15.59.139) - group Confed_esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:05:36] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b1-eqiad [13:06:06] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b1-eqiad [13:12:32] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a4-eqiad [13:12:46] !log ayounsi@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on netflow1002.eqiad.wmnet with reason: host reimage [13:15:29] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a4-eqiad [13:17:23] FIRING: GnmiInterfaceCountersDrop: ... [13:17:23] cr2-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr2-esams:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [13:18:11] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow1002.eqiad.wmnet with reason: host reimage [13:22:04] (03CR) 10Bking: [C:03+2] matomo: Restore GeoIP functionality [puppet] - 10https://gerrit.wikimedia.org/r/1335153 (https://phabricator.wikimedia.org/T431608) (owner: 10Bking) [13:27:10] FIRING: BFDdown: BFD session down between cr1-esams and 185.15.59.138 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:28:34] (03CR) 10Ladsgroup: [C:03+2] Tidy up resource and rounding leftovers [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333248 (owner: 10Ladsgroup-claude) [13:29:01] (03PS1) 10Muehlenhoff: Additional necessary settings for the openldap::rw_mdb role [puppet] - 10https://gerrit.wikimedia.org/r/1335860 (https://phabricator.wikimedia.org/T331699) [13:31:14] (03PS2) 10Muehlenhoff: Additional necessary settings for the openldap::rw_mdb role [puppet] - 10https://gerrit.wikimedia.org/r/1335860 (https://phabricator.wikimedia.org/T331699) [13:32:10] RESOLVED: [2x] BFDdown: BFD session down between cr1-codfw and fe80::7ee2:ca07:dde:4ba1 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:33:14] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations, 13Patch-For-Review: Move Docker images under the /v2/releng prefix to S3 - https://phabricator.wikimedia.org/T432829#12289814 (10elukey) 05Open→03Resolved a:03elukey On a second thought I preferred/decided to avoid adding the test-platfor... [13:33:36] 06SRE, 10Wikimedia-Mailing-lists: Mailing list logging in/ownership issue - https://phabricator.wikimedia.org/T436830#12289819 (10Ladsgroup) Your email (none of yours) is not listed as owner, maybe someone has accidentally removed you. Please ask for it to be added back. [13:34:59] (03Merged) 10jenkins-bot: Tidy up resource and rounding leftovers [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333248 (owner: 10Ladsgroup-claude) [13:36:03] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host netflow1002.eqiad.wmnet with OS trixie [13:46:32] (03CR) 10Elukey: [C:03+1] Additional necessary settings for the openldap::rw_mdb role [puppet] - 10https://gerrit.wikimedia.org/r/1335860 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [13:46:39] (03PS1) 10Jaime Nuche: releases-jenkins: do not use HTTP proxy outside of wmnet [puppet] - 10https://gerrit.wikimedia.org/r/1335868 [13:47:21] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12289888 (10MoritzMuehlenhoff) [13:52:27] (03CR) 10Cathal Mooney: [C:03+1] "I'm not terribly familiar with the spicerack/confctl options but it all looks sane. One thing I wonder about the results, will this cease " [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 (owner: 10Ssingh) [13:53:51] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b2-eqiad [13:54:17] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b2-eqiad [13:56:17] (03PS1) 10Muehlenhoff: Record LDAP access for jalter [puppet] - 10https://gerrit.wikimedia.org/r/1335871 [13:56:35] (03CR) 10Atsuko: [C:03+1] Add the new dse-k8s-worker nodes in eqiad to site.pp and preseed.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1335817 (https://phabricator.wikimedia.org/T436964) (owner: 10Btullis) [13:56:54] 06SRE, 10LDAP-Access-Requests: Grant Access to wmf, for WMF staff/contractors nda group for gsduser - https://phabricator.wikimedia.org/T435852#12289903 (10GSduser) I lack access to the data I need. [13:57:23] FIRING: [8x] GnmiInterfaceCountersDrop: cr2-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:01:06] (03CR) 10Muehlenhoff: [C:03+2] Record LDAP access for jalter [puppet] - 10https://gerrit.wikimedia.org/r/1335871 (owner: 10Muehlenhoff) [14:02:22] FIRING: [9x] GnmiInterfaceCountersDrop: cr1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:04:31] !log ayounsi@cumin1003 START - Cookbook sre.hosts.reimage for host netflow1003.eqiad.wmnet with OS trixie [14:06:58] (03CR) 10Btullis: Druid: Add alerts for memory and I/O stalls (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1335090 (https://phabricator.wikimedia.org/T436968) (owner: 10Bking) [14:07:23] FIRING: [15x] GnmiInterfaceCountersDrop: cr1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:08:35] (03CR) 10Btullis: [C:03+2] Add the new dse-k8s-worker nodes in eqiad to site.pp and preseed.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1335817 (https://phabricator.wikimedia.org/T436964) (owner: 10Btullis) [14:09:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:12:23] FIRING: [14x] GnmiInterfaceCountersDrop: cr1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:14:51] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:17:23] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS trixie [14:17:59] !log ayounsi@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on netflow1003.eqiad.wmnet with reason: host reimage [14:18:01] (03CR) 10Arnaudb: [C:03+1] "lgtm, thanks for the patch!" [puppet] - 10https://gerrit.wikimedia.org/r/1335868 (owner: 10Jaime Nuche) [14:21:14] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b3-eqiad [14:21:32] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad [14:23:52] (03PS1) 10Tiziano Fogli: prometheus/rules_ops: fix a typo in instance:gnmic_target_up expr [puppet] - 10https://gerrit.wikimedia.org/r/1335885 [14:24:49] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow1003.eqiad.wmnet with reason: host reimage [14:25:45] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2196 crashed - https://phabricator.wikimedia.org/T437015#12290030 (10Jhancock.wm) @Marostegui I'll get this one added to the thread and start updating firmwares. [14:26:53] !log elukey@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS trixie [14:26:58] (03CR) 10Scott French: [C:03+1] "Thank you, Luca! Very exciting :)" [puppet] - 10https://gerrit.wikimedia.org/r/1333237 (https://phabricator.wikimedia.org/T435499) (owner: 10Elukey) [14:27:39] (03CR) 10Tiziano Fogli: "We're currently getting some `PrometheusRuleEvaluationFailures` alerts, and looking at the logs, they seem to be related to the `instance:" [puppet] - 10https://gerrit.wikimedia.org/r/1335885 (owner: 10Tiziano Fogli) [14:28:01] !log cgoubert@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1339.eqiad.wmnet [14:28:05] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1339.eqiad.wmnet [14:28:41] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1339.eqiad.wmnet [14:29:49] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [14:30:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.04% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:31:42] cgoubert@cumin2003 renumber-node (PID 3363369) is awaiting input [14:31:58] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1339.eqiad.wmnet with OS trixie [14:32:20] !log cgoubert@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1339 [14:32:21] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1339 [14:37:22] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b4-eqiad [14:37:23] FIRING: [17x] GnmiInterfaceCountersDrop: cr1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:37:44] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b4-eqiad [14:38:23] 06SRE, 10LDAP-Access-Requests: Grant Access to wmf, for WMF staff/contractors nda group for gsduser - https://phabricator.wikimedia.org/T435852#12290077 (10Aklapper) Which data access are you missing? We cannot easily answer this for you. Please see https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access... [14:38:30] !log andrew@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephosd1045.eqiad.wmnet [14:40:08] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host netflow1003.eqiad.wmnet with OS trixie [14:40:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.04% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:42:00] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcephosd1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [14:42:23] RESOLVED: [18x] GnmiInterfaceCountersDrop: cr1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:43:26] (03CR) 10Btullis: "You have two other changes included in this CR, which you haven't mentioned in the commit message." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335793 (owner: 10Dpogorzelski) [14:44:43] (03CR) 10Jaime Nuche: "What I do right now when I deploy a releases Jenkins outside of prod (for ex. locally) is to remove the proxy config manually from the web" [puppet] - 10https://gerrit.wikimedia.org/r/1335868 (owner: 10Jaime Nuche) [14:45:32] (03CR) 10Scott French: "Thanks, Jasmine!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335403 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [14:46:26] !log andrew@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd1045.eqiad.wmnet [14:49:54] (03PS1) 10Muehlenhoff: cumin: Fix firmware sync [puppet] - 10https://gerrit.wikimedia.org/r/1335895 [14:51:02] (03CR) 10Scott French: [C:03+1] "Thanks, Simon!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [14:51:14] (03CR) 10Atsuko: [C:03+1] Druid: Add alerts for memory and I/O stalls [alerts] - 10https://gerrit.wikimedia.org/r/1335090 (https://phabricator.wikimedia.org/T436968) (owner: 10Bking) [14:51:40] (03CR) 10Bking: "Excellent point, and Ben mentioned this as well. I will update this change (and its comment) to move `krb1004` to the top of the list, and" [puppet] - 10https://gerrit.wikimedia.org/r/1335159 (https://phabricator.wikimedia.org/T435873) (owner: 10Bking) [14:54:51] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:55:19] (03PS3) 10Bking: Kerberos: Present krb1004 as first choice to clients [puppet] - 10https://gerrit.wikimedia.org/r/1335159 (https://phabricator.wikimedia.org/T435873) [14:56:44] (03PS4) 10Dpogorzelski: lw studio: add dedicated pgsql [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335793 [14:56:44] (03PS1) 10Dpogorzelski: admin_ng: add liftwing-studio namespace to dse-k8s-eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335898 [14:56:45] (03PS1) 10Dpogorzelski: admin_ng: add liftwing-studio to the ceph-rbd CSI tenant namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335899 [14:59:54] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335159 (https://phabricator.wikimedia.org/T435873) (owner: 10Bking) [15:03:46] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1335159 (https://phabricator.wikimedia.org/T435873) (owner: 10Bking) [15:06:00] (03CR) 10Scott French: apache: Configure wikimedia.org redirect in apache, remove from redirects.dat (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1333298 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [15:09:57] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b5-eqiad [15:10:17] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b5-eqiad [15:10:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for ssw1-d1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:10:52] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module galera [puppet] - 10https://gerrit.wikimedia.org/r/1335901 (https://phabricator.wikimedia.org/T435225) [15:15:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:21:53] (03CR) 10JHathaway: [C:03+1] cumin: Fix firmware sync [puppet] - 10https://gerrit.wikimedia.org/r/1335895 (owner: 10Muehlenhoff) [15:22:30] (03CR) 10Muehlenhoff: [C:03+2] cumin: Fix firmware sync [puppet] - 10https://gerrit.wikimedia.org/r/1335895 (owner: 10Muehlenhoff) [15:23:42] (03CR) 10Scott French: "Thanks, Denisse!" [alerts] - 10https://gerrit.wikimedia.org/r/1335636 (https://phabricator.wikimedia.org/T437010) (owner: 10Andrea Denisse) [15:25:23] FIRING: [5x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:30:23] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:40:23] FIRING: GnmiInterfaceCountersDrop: ... [15:40:23] lsw1-d6-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-d6-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:41:10] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module openstack [puppet] - 10https://gerrit.wikimedia.org/r/1335917 (https://phabricator.wikimedia.org/T435225) [15:41:49] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile wmcs [puppet] - 10https://gerrit.wikimedia.org/r/1335918 (https://phabricator.wikimedia.org/T435225) [15:42:19] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module base [puppet] - 10https://gerrit.wikimedia.org/r/1335919 (https://phabricator.wikimedia.org/T435225) [15:43:11] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [15:43:18] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile debmonitor [puppet] - 10https://gerrit.wikimedia.org/r/1335920 (https://phabricator.wikimedia.org/T435225) [15:43:47] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile postfix [puppet] - 10https://gerrit.wikimedia.org/r/1335921 (https://phabricator.wikimedia.org/T435225) [15:43:51] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1335919 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [15:47:19] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add missing dns entries lsw1-b5-eqiad - cmooney@cumin1004" [15:49:13] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module mailman3 [puppet] - 10https://gerrit.wikimedia.org/r/1335927 (https://phabricator.wikimedia.org/T435225) [15:49:28] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module phabricator [puppet] - 10https://gerrit.wikimedia.org/r/1335928 (https://phabricator.wikimedia.org/T435225) [15:49:47] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add missing dns entries lsw1-b5-eqiad - cmooney@cumin1004" [15:49:47] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:50:35] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile etcd [puppet] - 10https://gerrit.wikimedia.org/r/1335929 (https://phabricator.wikimedia.org/T435225) [15:51:46] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module query_service [puppet] - 10https://gerrit.wikimedia.org/r/1335930 (https://phabricator.wikimedia.org/T435225) [15:52:27] !log cgoubert@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1339.eqiad.wmnet with OS trixie [15:52:28] !log cgoubert@cumin2003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1339.eqiad.wmnet [15:55:16] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1335920 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [16:00:00] 06SRE, 10Internet-Archive, 06Traffic, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12290437 (10BCornwall) Given that removal of the "dead host" flag from the Archive team made it work again it appears that's our cause. The question is then whether we're sendi... [16:05:29] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b6-eqiad [16:05:51] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b6-eqiad [16:08:32] 10SRE-swift-storage, 10Thumbor: Gradually drop all thumbnails as a one-off clean up - https://phabricator.wikimedia.org/T379942#12290460 (10Ladsgroup) [16:10:48] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1335921 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:15:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:22:38] 06SRE, 10SRE-Access-Requests: Requesting access to production for derenrich - https://phabricator.wikimedia.org/T436965#12290531 (10HSwan-WMF) Approved. Thank you! [16:24:05] (03CR) 10Btullis: [C:03+1] admin_ng: add liftwing-studio namespace to dse-k8s-eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335898 (owner: 10Dpogorzelski) [16:24:18] (03CR) 10Btullis: [C:03+1] admin_ng: add liftwing-studio to the ceph-rbd CSI tenant namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335899 (owner: 10Dpogorzelski) [16:24:36] (03CR) 10Btullis: [C:03+1] lw studio: add dedicated pgsql [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335793 (owner: 10Dpogorzelski) [16:25:33] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module backup [puppet] - 10https://gerrit.wikimedia.org/r/1335952 (https://phabricator.wikimedia.org/T435225) [16:27:02] (03PS1) 10JHathaway: Puppet 8: Replace unscoped legacy facts in role mariadb [puppet] - 10https://gerrit.wikimedia.org/r/1335953 (https://phabricator.wikimedia.org/T435225) [16:29:07] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in profile debmonitor [puppet] - 10https://gerrit.wikimedia.org/r/1335920 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [16:29:15] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in module base [puppet] - 10https://gerrit.wikimedia.org/r/1335919 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [16:31:36] (03PS4) 10DLynch: editcheck-headless: Add deployment for the technical pilot [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) [16:31:49] (03CR) 10Scott French: [C:03+1] "Thanks, Jesse!" [puppet] - 10https://gerrit.wikimedia.org/r/1335929 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [16:32:25] (03CR) 10DLynch: editcheck-headless: Add deployment for the technical pilot (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [16:33:00] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b7-eqiad [16:33:44] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-b7-eqiad [16:34:16] 06SRE, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290602 (10BLiviero-WMF) [16:35:39] !log jgiannelos@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [16:36:18] !log jgiannelos@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [16:36:19] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [16:37:00] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [16:40:39] (03PS5) 10DLynch: editcheck-headless: Add deployment for the technical pilot [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332764 (https://phabricator.wikimedia.org/T434109) [16:41:14] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290634 (10thcipriani) Bullseye is LTS EoL as of Monday, so they're starting to dismantle/rearrange the apt repo it seems. Ideally, w... [16:48:26] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in profile postfix [puppet] - 10https://gerrit.wikimedia.org/r/1335921 (https://phabricator.wikimedia.org/T435225) [16:48:32] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335921 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [16:49:18] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in profile etcd [puppet] - 10https://gerrit.wikimedia.org/r/1335929 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [16:51:42] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace unscoped legacy facts in profile postfix [puppet] - 10https://gerrit.wikimedia.org/r/1335921 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [16:52:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [16:52:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 2.706% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:53:47] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module backup [puppet] - 10https://gerrit.wikimedia.org/r/1335952 (https://phabricator.wikimedia.org/T435225) [16:53:52] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335952 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [16:54:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 0% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:57:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [16:57:15] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-ext releases routed via main (k8s) 2.274s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [16:58:51] 10ops-codfw, 06SRE, 06DC-Ops: Broken disk on maps-test2001 - https://phabricator.wikimedia.org/T437036#12290684 (10Jhancock.wm) @MoritzMuehlenhoff I don't have an exact match for the failed drive, but i have a larger one that could be over-provisioned. Would that work? [17:02:07] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in role mariadb [puppet] - 10https://gerrit.wikimedia.org/r/1335953 (https://phabricator.wikimedia.org/T435225) [17:02:13] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335953 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [17:05:03] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:06:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:06:31] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-internal-scholarly_443: Servers wdqs1027.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [17:08:31] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [17:10:13] FIRING: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [17:11:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:12:18] !incidents [17:12:19] 8324 (ACKED) PHPFPMTooBusy sre (mw-api-ext main eqiad) [17:12:19] 8325 (UNACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [17:12:19] 8322 (RESOLVED) db2196 (paged)/MariaDB Replica SQL: x1 (paged) [17:12:19] 8323 (RESOLVED) db2196 (paged)/MariaDB Replica Lag: x1 (paged) [17:12:19] 8321 (RESOLVED) db2196 (paged)/MariaDB Replica IO: x1 (paged) [17:12:20] 8320 (RESOLVED) Host db2196 (paged) [17:12:20] 8318 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main eqiad) [17:12:21] 8319 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [17:12:23] !ack [17:12:24] 8325 (ACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [17:15:13] RESOLVED: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [17:15:31] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-internal-scholarly_443: Servers wdqs1027.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [17:16:31] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [17:18:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:19:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 0.6826% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:22:15] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-ext releases routed via main (k8s) 1.646s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:23:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:27:02] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b8-eqiad [17:27:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 24.27% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:27:22] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b8-eqiad [17:32:15] RESOLVED: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [17:36:01] 10ops-codfw, 06SRE, 06DC-Ops: Broken disk on maps-test2001 - https://phabricator.wikimedia.org/T437036#12290833 (10MoritzMuehlenhoff) Yes, that would work, thanks! [17:39:29] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12290837 (10cmooney) Just an update on where we are at here. I had some time over the last day or two to work on the connectivity and configuration for thes... [17:40:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.11% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:41:31] (03PS9) 10Andrew Bogott: Add cloudvps-project-usage.py [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) [17:41:41] (03CR) 10Andrew Bogott: Add cloudvps-project-usage.py (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1331749 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [17:42:52] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290843 (10dancy) Current production scap targets: ` ┌──────────┬─────────┬───────────────────────────────────────────� [17:44:12] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290845 (10dduvall) Noting that there are a number of images in operations/docker-images/production-images that depend on bullseye (d... [17:45:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.04% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:45:20] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333249 (owner: 10Ladsgroup-claude) [17:45:29] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333250 (owner: 10Ladsgroup-claude) [17:45:44] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333251 (owner: 10Ladsgroup-claude) [17:45:55] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333252 (owner: 10Ladsgroup-claude) [17:46:14] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 (owner: 10Ladsgroup-claude) [17:46:20] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333254 (owner: 10Ladsgroup-claude) [17:46:45] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290846 (10MoritzMuehlenhoff) Judging from what I saw on Debian IRC it looks like the removal of bullseye-security was accidental, th... [17:53:01] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Attempt to use a VM as Kerberos replica - https://phabricator.wikimedia.org/T435873#12290851 (10bking) Slight change to the above plan, as both Ben and Moritz pointed out that the best way to force more traff... [17:55:03] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:00:51] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290893 (10MoritzMuehlenhoff) Or specifically for scap maybe just disable CI for Bullseye for now? The remaining aqs/restbase/session... [18:06:26] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290910 (10dancy) Here's the full list of bullseye targets as of today: ` aqs1022.eqiad.wmnet aqs1023.eqiad.wmnet aqs1024.eqiad... [18:09:54] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [18:12:59] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module query_service [puppet] - 10https://gerrit.wikimedia.org/r/1335930 (https://phabricator.wikimedia.org/T435225) [18:13:06] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335930 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [18:14:19] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module phabricator [puppet] - 10https://gerrit.wikimedia.org/r/1335928 (https://phabricator.wikimedia.org/T435225) [18:14:28] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335928 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [18:15:48] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module mailman3 [puppet] - 10https://gerrit.wikimedia.org/r/1335927 (https://phabricator.wikimedia.org/T435225) [18:15:56] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1335927 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [18:17:22] (03PS2) 10JHathaway: Puppet 8: Replace unscoped legacy facts in module galera [puppet] - 10https://gerrit.wikimedia.org/r/1335901 (https://phabricator.wikimedia.org/T435225) [18:17:47] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290930 (10MoritzMuehlenhoff) >>! In T437069#12290910, @dancy wrote: > Here's the full list of bullseye targets as of today: We're t... [18:18:24] !log bking@clouddumps100[12] `systemctl reset-failed` to quash alerts until https://w.wiki/UBje . The systemd timer should try again tomorrow [18:18:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:23:36] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290932 (10dancy) Remaining beta cluster bullseye targets: ` bullseye (8 targets) deployment-deploy04.deployment-prep.eqiad1.wikime... [18:24:56] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12290942 (10dancy) >>! In T437069#12290893, @MoritzMuehlenhoff wrote: > Or specifically for scap maybe just disable CI for Bullseye fo... [18:25:05] PROBLEM - Host maps-test2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:27:35] RECOVERY - Host maps-test2001 is UP: PING OK - Packet loss = 0%, RTA = 33.01 ms [18:28:00] 10ops-codfw, 06SRE, 06DC-Ops: Broken disk on maps-test2001 - https://phabricator.wikimedia.org/T437036#12290946 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm disk replacement has been completed. [18:33:43] ACKNOWLEDGEMENT - MD RAID on maps-test2001 is CRITICAL: CRITICAL: State: degraded, Active: 3, Working: 4, Failed: 0, Spare: 0 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T437082 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [18:33:51] 10ops-codfw, 06SRE, 06DC-Ops: Degraded RAID on maps-test2001 - https://phabricator.wikimedia.org/T437082 (10ops-monitoring-bot) 03NEW [18:55:06] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:01:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:07:23] !log jhancock@cumin1003 START - Cookbook sre.dns.netbox [19:11:13] !log jhancock@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2013 to codfw - jhancock@cumin1003" [19:11:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:11:18] !log jhancock@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding sretest2013 to codfw - jhancock@cumin1003" [19:11:18] !log jhancock@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [19:11:37] !log jhancock@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host sretest2013 [19:11:55] !log jhancock@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host sretest2013 [19:12:23] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [19:13:08] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12291052 (10Jhancock.wm) [19:13:12] !log jhancock@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [19:19:57] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [19:20:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:25:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:26:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:27:04] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [19:27:51] (03PS22) 10Herron: sre.opensearch.roll-restart-reboot: include checklist items [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (https://phabricator.wikimedia.org/T435265) [19:31:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.41% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:33:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:37:07] (03PS4) 10Bking: Druid: Add alerts for memory and I/O stalls [alerts] - 10https://gerrit.wikimedia.org/r/1335090 (https://phabricator.wikimedia.org/T436968) [19:37:11] (03CR) 10Bking: Druid: Add alerts for memory and I/O stalls (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1335090 (https://phabricator.wikimedia.org/T436968) (owner: 10Bking) [19:40:13] FIRING: ProbeDown: Service registry1005:443 has failed probes (http_docker_registry_discovery_wmnet_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:40:37] FIRING: GnmiInterfaceCountersDrop: ... [19:40:38] lsw1-d6-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-d6-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [19:41:32] (03CR) 10Ladsgroup: "I try to deploy this next week. Unless Hugh thinks it's not ready." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333155 (owner: 10Hnowlan) [19:45:13] RESOLVED: ProbeDown: Service registry1005:443 has failed probes (http_docker_registry_discovery_wmnet_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:48:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:51:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:58:45] (03CR) 10RLazarus: [C:03+1] mesh: Support Host splits in configuration 1.16.0, networkpolicy 1.3.0 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331858 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [20:11:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.72% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:15:38] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [20:31:18] 06SRE, 10SRE-Access-Requests: Requesting access to Stat host stat1010 for Jose Aleman - https://phabricator.wikimedia.org/T436298#12291319 (10JAATPH) 05Open→03Resolved [20:40:42] !log jhathaway@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker1339.eqiad.wmnet with OS trixie [20:42:20] !log jhathaway@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1339.eqiad.wmnet with OS trixie [21:00:46] (03PS4) 10Scott French: modules: Prepare mesh.configuration / networkpolicy minor version bump [deployment-charts] - 10https://gerrit.wikimedia.org/r/1328738 (https://phabricator.wikimedia.org/T427666) [21:00:46] (03PS6) 10Scott French: mesh: Support Host splits in configuration 1.16.0, networkpolicy 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331858 (https://phabricator.wikimedia.org/T427666) [21:03:06] (03PS1) 10Gmodena: wdqs: extend envoy rq_time histogram buckets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1336094 (https://phabricator.wikimedia.org/T428631) [21:06:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:11:00] (03CR) 10Scott French: "Thanks for the review!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331858 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [21:11:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:13:25] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:19:02] (03CR) 10Bking: [C:03+2] Druid: Add alerts for memory and I/O stalls [alerts] - 10https://gerrit.wikimedia.org/r/1335090 (https://phabricator.wikimedia.org/T436968) (owner: 10Bking) [21:30:21] !log jhathaway@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker1339.eqiad.wmnet with OS trixie [21:34:36] (03Abandoned) 10DLynch: editcheck-headless: Add chart and deployment for the technical pilot [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326332 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [21:36:33] FIRING: KubernetesCalicoDown: wikikube-worker1339.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1339.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [21:37:04] (03CR) 10Scott French: [C:03+2] modules: Prepare mesh.configuration / networkpolicy minor version bump [deployment-charts] - 10https://gerrit.wikimedia.org/r/1328738 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [21:39:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:39:25] (03Merged) 10jenkins-bot: modules: Prepare mesh.configuration / networkpolicy minor version bump [deployment-charts] - 10https://gerrit.wikimedia.org/r/1328738 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [21:40:07] (03CR) 10Scott French: [C:03+2] mesh: Support Host splits in configuration 1.16.0, networkpolicy 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331858 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [21:42:15] !log jhathaway@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1339.eqiad.wmnet with reason: host reimage [21:42:37] (03Merged) 10jenkins-bot: mesh: Support Host splits in configuration 1.16.0, networkpolicy 1.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1331858 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [21:44:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:45:33] (03PS1) 10JHathaway: http boot: don't log error message [software/spicerack] - 10https://gerrit.wikimedia.org/r/1336116 [21:48:30] !log jhathaway@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1339.eqiad.wmnet with reason: host reimage [21:49:47] (03PS1) 10Zabe: Move production reads for commons link tables to x4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) [21:55:20] 10SRE-swift-storage: Unable to undelete - https://phabricator.wikimedia.org/T437114#12291634 (10Nemoralis) [21:55:48] 10SRE-swift-storage: Unknown error while undeleting a file on Wikimedia Commons - https://phabricator.wikimedia.org/T437114#12291641 (10Nemoralis) [21:56:43] 10SRE-swift-storage: Unknown error while undeleting a file on Wikimedia Commons - https://phabricator.wikimedia.org/T437114#12291642 (10Nemoralis) [22:07:11] !log jhathaway@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1339.eqiad.wmnet with OS trixie [22:09:54] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [22:26:10] (03CR) 10Ladsgroup: "maybe it's a bit of crazy idea but use mt_rand to move only 1% of requests to it. Deploy, check everything and fix anything that breaks qu" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [22:26:42] (03CR) 10Ladsgroup: Move production reads for commons link tables to x4 (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [22:27:04] (03CR) 10BCornwall: "Turns out this is triggering alerts:" [puppet] - 10https://gerrit.wikimedia.org/r/1329663 (owner: 10CDanis) [22:27:15] (03CR) 10BCornwall: "Marking unresolved" [puppet] - 10https://gerrit.wikimedia.org/r/1329663 (owner: 10CDanis) [22:42:01] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12291725 (10Jhancock.wm) @ssingh got it racked and provisioned. all yours. feel free to close to task unless you seem something out of place. [22:54:20] 06SRE, 10SRE-Access-Requests: Requesting access to production for derenrich - https://phabricator.wikimedia.org/T436965#12291740 (10thcipriani) >>! In T436965#12288621, @Volans wrote: > @derenrich that's the key's fingerprint, we need the actual public key, you can find the public key in the `.pub` file create... [22:55:06] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [23:00:50] (03CR) 10JHathaway: "@cgoubert@wikimedia.org I looked into the issue with wikikube-worker1339, the error message is expected as we try different boot options t" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1336116 (owner: 10JHathaway) [23:14:31] 06SRE, 10SRE-Access-Requests: Requesting access to production for derenrich - https://phabricator.wikimedia.org/T436965#12291780 (10derenrich) Ah sorry. Here is the public key ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF66vmTGSCShUC1HkopdFG8dtfPFOJQz+P6PVRypvDqV derenrich@wmf3489 [23:17:55] (03PS1) 10Southparkfan: LabsServices: adjust Marhoid URI to use service RR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336153 (https://phabricator.wikimedia.org/T436465) [23:18:15] (03PS2) 10Southparkfan: LabsServices: adjust Mathoid URI to use service RR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336153 (https://phabricator.wikimedia.org/T436465) [23:37:49] (03CR) 10Andrea Denisse: sre.opensearch.roll-restart-reboot: include checklist items (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (https://phabricator.wikimedia.org/T435265) (owner: 10Herron) [23:40:38] FIRING: GnmiInterfaceCountersDrop: ... [23:40:38] lsw1-d6-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-d6-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [23:49:39] RESOLVED: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [23:49:51] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown