[00:17:12] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [00:24:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:29:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:30:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.14% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:35:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:40:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:51:14] (03PS1) 10Codename Noreste: eswiki: Add abusefilter-access-protected-vars to abusefilter user group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342825 (https://phabricator.wikimedia.org/T436652) [01:00:05] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1342392 (owner: 10TrainBranchBot) [01:11:03] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1342826 [01:11:03] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1342826 (owner: 10TrainBranchBot) [01:18:36] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1342826 (owner: 10TrainBranchBot) [01:20:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:23:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:26:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:28:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:29:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.56% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:39:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:49:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438282#12336349 (10phaultfinder) [02:00:55] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:08:53] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 57s) [02:12:12] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:17:12] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:32:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:37:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:38:40] FIRING: [4x] ProbeDown: Service registry1004:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:16:37] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:18:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [04:23:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:45:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:50:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:07:04] (03PS1) 10Giuseppe Lavagetto: gVisor: disable on bookworm in containerd [puppet] - 10https://gerrit.wikimedia.org/r/1342961 (https://phabricator.wikimedia.org/T436649) [05:07:07] (03PS1) 10Giuseppe Lavagetto: gVisor: use kvm as a platform when hardware virtualization is available [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) [05:07:09] (03PS1) 10Giuseppe Lavagetto: kubernetes::node: add gvisor label based on hardware virtualization [puppet] - 10https://gerrit.wikimedia.org/r/1342963 (https://phabricator.wikimedia.org/T438287) [05:07:49] (03PS1) 10Giuseppe Lavagetto: runtimeClasses: add gVisor-kvm RuntimeClass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342964 (https://phabricator.wikimedia.org/T438287) [05:07:52] (03PS1) 10Giuseppe Lavagetto: shellbox-timeline: run under kvm in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342965 (https://phabricator.wikimedia.org/T438287) [05:26:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:00:04] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260918T0600) [06:14:35] (03CR) 10Marostegui: mediabackups: Perform backups via systemd units (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) (owner: 10CWilliams) [06:14:38] (03CR) 10Marostegui: [C:03+1] mediabackups: Perform backups via systemd units [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) (owner: 10CWilliams) [06:17:12] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:21:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:25:07] (03PS1) 10Marostegui: installserver: Remove duplicate [puppet] - 10https://gerrit.wikimedia.org/r/1342969 [06:26:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:28:23] (03CR) 10Marostegui: [C:03+2] installserver: Remove duplicate [puppet] - 10https://gerrit.wikimedia.org/r/1342969 (owner: 10Marostegui) [06:38:40] FIRING: [4x] ProbeDown: Service registry1004:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [06:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:51:55] (03PS4) 10Slyngshede: mw-web: upsize for single-DC serving [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) [06:52:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:52:27] (03PS5) 10Slyngshede: mw-web: upsize for single-DC serving [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) [06:53:13] (03CR) 10Slyngshede: "Rechecking the numbers and talking to joe, we've decided that 600 replicas seems like the safer option." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [06:58:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.55% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:58:26] (03CR) 10Giuseppe Lavagetto: [C:03+1] mw-web: upsize for single-DC serving [deployment-charts] - 10https://gerrit.wikimedia.org/r/1332725 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260918T0700) [07:03:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.55% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:09:08] (03PS1) 10Filippo Giunchedi: prometheus: report metrics from automount units [puppet] - 10https://gerrit.wikimedia.org/r/1342971 (https://phabricator.wikimedia.org/T438062) [07:12:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:16:37] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:20:08] (03PS5) 10CWilliams: mediabackups: Perform backups via systemd units [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) [07:22:29] 06SRE, 10LDAP-Access-Requests: Request for LDAP NDA Access for CentralNotice Metrics for Yahya - https://phabricator.wikimedia.org/T436294#12336629 (10MoritzMuehlenhoff) >>! In T436294#12334060, @tappof wrote: > User @Yahya has been added to NDA ldap group. The tracking entry in data.yaml is missing, reopenin... [07:22:37] 06SRE, 10LDAP-Access-Requests: Request for LDAP NDA Access for CentralNotice Metrics for Yahya - https://phabricator.wikimedia.org/T436294#12336630 (10MoritzMuehlenhoff) 05Resolved→03Open [07:24:48] 06SRE, 10LDAP-Access-Requests: Request for LDAP NDA Access for CentralNotice Metrics for Yahya - https://phabricator.wikimedia.org/T436294#12336638 (10tappof) Sorry, I'm going to add him. Thank you! [07:25:44] (03CR) 10Brouberol: "The fact that this was not merged has raised some questions in https://wikimedia.slack.com/archives/C01DFMX6QLB/p1789654093386779?thread_t" [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) (owner: 10Ssingh) [07:25:49] (03CR) 10Brouberol: [C:03+2] admin: add anilk to analytics-privatedata-users, remove anil [puppet] - 10https://gerrit.wikimedia.org/r/1339129 (https://phabricator.wikimedia.org/T437611) (owner: 10Ssingh) [07:27:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:35:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.04% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:42:26] (03PS6) 10CWilliams: mediabackups: Perform backups via systemd units [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) [07:45:41] (03CR) 10CWilliams: [C:03+2] mediabackups: Perform backups via systemd units [puppet] - 10https://gerrit.wikimedia.org/r/1342701 (https://phabricator.wikimedia.org/T438012) (owner: 10CWilliams) [07:46:14] 06SRE, 10observability, 06Traffic: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12336680 (10dkertesz) The behavior reported in this ticket is still happening; hourly reloads correspond to hourly dips in `haproxy_frontend_http_requests_total`, for example. Persistent s... [07:47:32] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 21 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342825 (https://phabricator.wikimedia.org/T436652) (owner: 10Codename Noreste) [08:00:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:00:39] (03CR) 10JMeybohm: [C:03+2] kubelet: Add missing evictionHard imagefs.inodesFree [puppet] - 10https://gerrit.wikimedia.org/r/1338831 (https://phabricator.wikimedia.org/T437539) (owner: 10JMeybohm) [08:00:50] (03CR) 10JMeybohm: [C:03+2] kubelet: Start image garbage collection at 80% imagefs usage [puppet] - 10https://gerrit.wikimedia.org/r/1338832 (https://phabricator.wikimedia.org/T437539) (owner: 10JMeybohm) [08:01:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:02:50] (03PS1) 10Tiziano Fogli: admin/data: add yahya [puppet] - 10https://gerrit.wikimedia.org/r/1343007 (https://phabricator.wikimedia.org/T436294) [08:06:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:06:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.14% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:16:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:19:12] (03CR) 10Btullis: [C:03+1] Revert "Only serve opensearch-{ipoid,ttmserver} traffic from dse-k8s-eqiad" [dns] - 10https://gerrit.wikimedia.org/r/1342669 (owner: 10Brouberol) [08:21:04] (03CR) 10Brouberol: [C:03+2] Revert "Only serve opensearch-{ipoid,ttmserver} traffic from dse-k8s-eqiad" [dns] - 10https://gerrit.wikimedia.org/r/1342669 (owner: 10Brouberol) [08:21:31] !log brouberol@dns1004 START - running authdns-update [08:21:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [08:23:51] !log brouberol@dns1004 END - running authdns-update [08:26:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:28:55] (03CR) 10Brouberol: [C:03+1] dumps: re-add mirror PDApps [puppet] - 10https://gerrit.wikimedia.org/r/1342785 (https://phabricator.wikimedia.org/T437896) (owner: 10Xcollazo) [08:32:23] (03PS5) 10Ryan Kemper: kafka: converge topic config from hieradata [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) [08:37:14] (03PS4) 10Federico Ceratto: wmnet: update CNAME records for DB masters to codfw [dns] - 10https://gerrit.wikimedia.org/r/1319815 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [08:38:13] (03CR) 10Federico Ceratto: "Rebased" [dns] - 10https://gerrit.wikimedia.org/r/1319815 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [08:44:51] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1343007 (https://phabricator.wikimedia.org/T436294) (owner: 10Tiziano Fogli) [08:46:18] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343010 [08:47:40] (03CR) 10Marostegui: "Unless there's an emergency I don't think we will be doing any DB switchovers before the DC switchover so we can probably give this a last" [dns] - 10https://gerrit.wikimedia.org/r/1319815 (https://phabricator.wikimedia.org/T433363) (owner: 10Slyngshede) [08:48:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:51:13] (03CR) 10Brouberol: kafka: converge topic config from hieradata (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [08:51:50] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1342775 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall) [08:55:13] (03PS1) 10Ozge: ml-services: Increase llm namespace cpu quota from 150 to 200 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343012 (https://phabricator.wikimedia.org/T438448) [08:55:35] (03PS2) 10Ozge: ml-services: Increase llm namespace cpu quota from 150 to 200 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343012 (https://phabricator.wikimedia.org/T438448) [08:57:24] (03CR) 10Filippo Giunchedi: "> Hi," [puppet] - 10https://gerrit.wikimedia.org/r/1305183 (https://phabricator.wikimedia.org/T429630) (owner: 10MVernon) [09:03:16] (03CR) 10Brouberol: [C:03+1] ml-services: Increase llm namespace cpu quota from 150 to 200 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343012 (https://phabricator.wikimedia.org/T438448) (owner: 10Ozge) [09:04:13] (03PS3) 10Filippo Giunchedi: hieradata: organize wmcs cluster into separate cloud clusters [puppet] - 10https://gerrit.wikimedia.org/r/1338132 (https://phabricator.wikimedia.org/T437272) [09:04:14] (03CR) 10AikoChou: [C:03+1] ml-services: Increase llm namespace cpu quota from 150 to 200 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343012 (https://phabricator.wikimedia.org/T438448) (owner: 10Ozge) [09:06:09] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Increase llm namespace cpu quota from 150 to 200 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343012 (https://phabricator.wikimedia.org/T438448) (owner: 10Ozge) [09:06:21] (03PS4) 10Filippo Giunchedi: hieradata: organize wmcs cluster into separate cloud clusters [puppet] - 10https://gerrit.wikimedia.org/r/1338132 (https://phabricator.wikimedia.org/T437272) [09:07:19] (03PS5) 10Filippo Giunchedi: hieradata: organize wmcs cluster into separate cloud clusters [puppet] - 10https://gerrit.wikimedia.org/r/1338132 (https://phabricator.wikimedia.org/T437272) [09:07:57] (03PS1) 10Elukey: sre.hardware.upgrade-firmware: remove dead code [cookbooks] - 10https://gerrit.wikimedia.org/r/1343013 [09:08:19] (03CR) 10Filippo Giunchedi: "This is ready to be merged next week following phab feedback" [puppet] - 10https://gerrit.wikimedia.org/r/1338132 (https://phabricator.wikimedia.org/T437272) (owner: 10Filippo Giunchedi) [09:08:54] !log brouberol@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [09:09:22] FIRING: GnmiInterfaceCountersDrop: ... [09:09:22] ssw1-d8-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=ssw1-d8-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [09:09:40] !log brouberol@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [09:10:07] (03PS1) 10Muehlenhoff: kernel: Blocklist ipsec authentication headers [puppet] - 10https://gerrit.wikimedia.org/r/1343014 [09:14:22] RESOLVED: GnmiInterfaceCountersDrop: ... [09:14:22] ssw1-d8-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=ssw1-d8-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [09:18:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:18:52] !log clone db1270:x4 from db1155:x4 lag will appear on x4 [09:18:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:18:57] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 21 hosts with reason: cloning db1270 [09:19:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:19:27] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [09:20:19] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [09:21:13] (03CR) 10Elukey: kafka: converge topic config from hieradata (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [09:21:52] (03PS1) 10Marostegui: db1270: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1343017 (https://phabricator.wikimedia.org/T436920) [09:22:16] PROBLEM - MariaDB Replica IO: x4 on an-redacteddb1001 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db1155.eqiad.wmnet:3364 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db1155.eqiad.wmnet (111 Connection refused) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [09:22:25] ^expected [09:22:53] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet with reason: cloning [09:26:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:28:56] (03CR) 10Marostegui: [C:03+2] db1270: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1343017 (https://phabricator.wikimedia.org/T436920) (owner: 10Marostegui) [09:30:03] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [09:30:15] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [09:43:10] o/away afk! lunch [09:43:26] err wrong chan :D [09:44:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:47:45] 06SRE, 06Infrastructure-Foundations: Disable TUN/TAP outside of virtualisation roles - https://phabricator.wikimedia.org/T438459 (10MoritzMuehlenhoff) 03NEW [09:49:51] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 3 NOOP 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1342636 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [09:54:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.42% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:02:38] (03PS1) 10Btullis: ceph: Allow the administration hosts through the server firewall [puppet] - 10https://gerrit.wikimedia.org/r/1343021 (https://phabricator.wikimedia.org/T435608) [10:17:27] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:17:49] (03PS1) 10Btullis: k8s: Let kube-proxy detect local Pod traffic by interface name [puppet] - 10https://gerrit.wikimedia.org/r/1343023 (https://phabricator.wikimedia.org/T429773) [10:17:52] (03PS1) 10Btullis: hieradata: Detect local Pod traffic by interface name on dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1343024 (https://phabricator.wikimedia.org/T429773) [10:19:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:19:40] (03PS2) 10Btullis: ceph: Allow the administration hosts through the server firewall [puppet] - 10https://gerrit.wikimedia.org/r/1343021 (https://phabricator.wikimedia.org/T435608) [10:20:14] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343021 (https://phabricator.wikimedia.org/T435608) (owner: 10Btullis) [10:20:18] RECOVERY - MariaDB Replica IO: x4 on an-redacteddb1001 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [10:20:46] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343023 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [10:24:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:26:30] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343024 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [10:38:40] FIRING: [4x] ProbeDown: Service registry1004:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:39:58] (03PS1) 10Muehlenhoff: kernel: Blocklist PPPOE [puppet] - 10https://gerrit.wikimedia.org/r/1343027 [10:43:11] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343014 (owner: 10Muehlenhoff) [10:43:15] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343027 (owner: 10Muehlenhoff) [10:44:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:45:54] (03PS2) 10Btullis: k8s: Let kube-proxy detect local Pod traffic by interface name [puppet] - 10https://gerrit.wikimedia.org/r/1343023 (https://phabricator.wikimedia.org/T429773) [10:45:57] (03PS2) 10Btullis: hieradata: Detect local Pod traffic by interface name on dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1343024 (https://phabricator.wikimedia.org/T429773) [10:54:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.04% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260918T0700) [11:00:05] jelto, arnoldokoth, mutante, and arnaudb: How many deployers does it take to do GitLab version upgrades deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260918T1100). [11:09:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:16:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:27:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:47:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:54:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:54:48] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1342694 (https://phabricator.wikimedia.org/T301640) (owner: 10Majavah) [12:13:50] !log jclark@cumin1004 START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [12:14:52] !log jclark@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [12:17:04] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12337578 (10Jclark-ctr) @klausman Can you update Site.pp file I am blocked from imaging till it is added [12:17:26] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12337579 (10Jclark-ctr) [12:19:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:20:47] (03PS1) 10Klausman: site.pp: Add ml-serve1016 in setup state [puppet] - 10https://gerrit.wikimedia.org/r/1343038 [12:21:38] (03CR) 10Btullis: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1343038 (owner: 10Klausman) [12:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [12:22:12] (03CR) 10Klausman: [V:03+2 C:03+2] site.pp: Add ml-serve1016 in setup state [puppet] - 10https://gerrit.wikimedia.org/r/1343038 (owner: 10Klausman) [12:22:53] (03CR) 10Elukey: [C:03+1] Revert "Stop building a Bullseye base image" [puppet] - 10https://gerrit.wikimedia.org/r/1342715 (owner: 10Muehlenhoff) [12:23:16] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12337592 (10klausman) Done: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1343038 (forgot Bug: line on the commit message, so explicitly adding comment here). [12:27:23] (03CR) 10Elukey: "Saw the change passing by, IIRC we had a different recipe for those jumbo hosts:" [puppet] - 10https://gerrit.wikimedia.org/r/1343038 (owner: 10Klausman) [12:28:07] (03CR) 10Elukey: "Horrible paste sorry, it is verbatim from preseed.yaml" [puppet] - 10https://gerrit.wikimedia.org/r/1343038 (owner: 10Klausman) [12:30:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.14% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:30:55] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12337610 (10MoritzMuehlenhoff) [12:31:01] (03PS1) 10Klausman: preseed: Add ml-serve1016 to receive correct partitioning [puppet] - 10https://gerrit.wikimedia.org/r/1343043 (https://phabricator.wikimedia.org/T438161) [12:31:06] (03CR) 10Klausman: [V:03+2 C:03+2] "Thanks! Created https://gerrit.wikimedia.org/r/c/operations/puppet/+/1343043 to add 1016 to the right preseed recipe." [puppet] - 10https://gerrit.wikimedia.org/r/1343038 (owner: 10Klausman) [12:31:40] (03CR) 10Muehlenhoff: [C:03+2] Revert "Stop building a Bullseye base image" [puppet] - 10https://gerrit.wikimedia.org/r/1342715 (owner: 10Muehlenhoff) [12:32:37] (03PS1) 10AKhatun: article-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343044 (https://phabricator.wikimedia.org/T437000) [12:37:18] 06SRE, 10vm-requests: codfw: 1 VM requested for ceph-admin - https://phabricator.wikimedia.org/T438038#12337620 (10BTullis) 05Open→03Resolved [12:40:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:40:29] (03CR) 10AKhatun: article-feature-counts: add deployment chart files (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343044 (https://phabricator.wikimedia.org/T437000) (owner: 10AKhatun) [12:43:01] (03CR) 10Klausman: [C:03+2] preseed: Add ml-serve1016 to receive correct partitioning [puppet] - 10https://gerrit.wikimedia.org/r/1343043 (https://phabricator.wikimedia.org/T438161) (owner: 10Klausman) [12:52:20] (03CR) 10Brouberol: kafka: converge topic config from hieradata (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [12:54:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:55:40] (03CR) 10Brouberol: [C:03+1] "Nice!" [puppet] - 10https://gerrit.wikimedia.org/r/1343021 (https://phabricator.wikimedia.org/T435608) (owner: 10Btullis) [12:55:43] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for VolkerE - https://phabricator.wikimedia.org/T437634#12337737 (10tappof) [13:02:33] (03CR) 10Btullis: [C:03+2] ceph: Allow the administration hosts through the server firewall [puppet] - 10https://gerrit.wikimedia.org/r/1343021 (https://phabricator.wikimedia.org/T435608) (owner: 10Btullis) [13:03:42] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for VolkerE - https://phabricator.wikimedia.org/T437634#12337758 (10tappof) ` $ sudo manage_principals.py create 'volker-e' --email=volker.e@wikimedia.org Principal successfully created. Make sure to update dat... [13:03:54] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343023 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [13:04:03] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343024 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [13:04:14] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host registry1004.eqiad.wmnet with OS trixie [13:05:31] (03PS1) 10Muehlenhoff: Remove cluster::management role from cumin1003 [puppet] - 10https://gerrit.wikimedia.org/r/1343055 (https://phabricator.wikimedia.org/T427897) [13:09:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:12:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:15:40] 07sre-alert-triage, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Alert in need of triage: AlertLintProblem (instance localhost:9123) - https://phabricator.wikimedia.org/T430139#12337853 (10Gehel) [13:16:59] 10ops-codfw, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Q1:rack/setup/install cirrussearch21[16-20] - https://phabricator.wikimedia.org/T436287#12337887 (10Gehel) [13:17:11] (03PS1) 10CWilliams: mediabackups: Run mediabackup processes via systemd [puppet] - 10https://gerrit.wikimedia.org/r/1343056 (https://phabricator.wikimedia.org/T438012) [13:17:12] FIRING: [2x] JobUnavailable: Reduced availability for job docker-registry in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:17:17] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Q1:rack/setup/install cirrussearch11[26-30] - https://phabricator.wikimedia.org/T436285#12337889 (10Gehel) [13:18:01] !log elukey@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on registry1004.eqiad.wmnet with reason: host reimage [13:21:52] 06SRE, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Site: EQIAD VM request for Kerberos - https://phabricator.wikimedia.org/T438229#12338001 (10Gehel) [13:22:12] FIRING: [2x] JobUnavailable: Reduced availability for job docker-registry in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:22:35] 06SRE, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: install1005 running out of disk due to squid log volume from an-worker* webproxy workload - https://phabricator.wikimedia.org/T435555#12338012 (10Gehel) [13:22:49] 07sre-alert-triage, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Alert in need of triage: AlertLintProblem (instance localhost:9123) - https://phabricator.wikimedia.org/T430139#12338018 (10bking) 05Open→03Resolved a:03bking This has been addressed by gerrit change ID `Ic21bd45c7... [13:23:02] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry1004.eqiad.wmnet with reason: host reimage [13:25:58] (03CR) 10JHathaway: [C:03+1] kernel: Blocklist ipsec authentication headers [puppet] - 10https://gerrit.wikimedia.org/r/1343014 (owner: 10Muehlenhoff) [13:26:10] 06SRE, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 4 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12338073 (10Gehel) [13:26:20] (03CR) 10JHathaway: [C:03+1] kernel: Blocklist PPPOE [puppet] - 10https://gerrit.wikimedia.org/r/1343027 (owner: 10Muehlenhoff) [13:26:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:26:52] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12338092 (10Gehel) [13:28:07] 10SRE-SLO, 10observability, 10Wikidata, 06Wikidata Platform Team, and 3 others: Update WDQS SLOs to reflect graph split changes - https://phabricator.wikimedia.org/T393966#12338122 (10Gehel) [13:28:32] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12338134 (10Gehel) [13:29:02] (03CR) 10JHathaway: [C:03+1] sre.hardware.upgrade-firmware: remove dead code [cookbooks] - 10https://gerrit.wikimedia.org/r/1343013 (owner: 10Elukey) [13:32:20] 10ops-eqiad, 06SRE, 10Continuous-Integration-Infrastructure, 06DC-Ops, and 2 others: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12338238 (10fgiunchedi) a:05Andrew→03fgiunchedi [13:34:00] (03PS1) 10Muehlenhoff: Record LDAP access for ashhan [puppet] - 10https://gerrit.wikimedia.org/r/1343060 [13:37:03] (03PS1) 10Btullis: ceph: Add a profile for the administration host tools [puppet] - 10https://gerrit.wikimedia.org/r/1343061 (https://phabricator.wikimedia.org/T435608) [13:37:11] (03CR) 10Muehlenhoff: [C:03+2] Record LDAP access for ashhan [puppet] - 10https://gerrit.wikimedia.org/r/1343060 (owner: 10Muehlenhoff) [13:37:29] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry1004.eqiad.wmnet with OS trixie [13:38:46] (03PS2) 10Btullis: ceph: Add a profile for the administration host tools [puppet] - 10https://gerrit.wikimedia.org/r/1343061 (https://phabricator.wikimedia.org/T435608) [13:39:23] (03PS2) 10Tiziano Fogli: admin: add volker-e to analytics-privatedata-users (krb) [puppet] - 10https://gerrit.wikimedia.org/r/1339865 (https://phabricator.wikimedia.org/T437634) (owner: 10Ssingh) [13:47:21] (03PS1) 10DLynch: mw.DesktopArticleTarget: if source education is enabled suppress welcome [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343065 (https://phabricator.wikimedia.org/T434249) [13:49:45] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1339865 (https://phabricator.wikimedia.org/T437634) (owner: 10Ssingh) [13:51:21] (03CR) 10MVernon: "> Another option I can see for swift in the pontoon case is to have one/two high level hiera control variables e.g. "list of backends" to " [puppet] - 10https://gerrit.wikimedia.org/r/1305183 (https://phabricator.wikimedia.org/T429630) (owner: 10MVernon) [13:53:11] I need an emergency deploy for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/1343065 -- context is T434249, are SRE ok with a deployment? (cc: thcipriani, jnuche, federico3, _joe_). I can deploy myself. [13:53:12] T434249: Design user experience for helping newcomers discover source editing (desktop, en.wiki) - https://phabricator.wikimedia.org/T434249 [13:54:04] <_joe_> Kemayo: lemme read about context, but otherwise yes [13:54:17] 10ops-eqiad, 06SRE, 06DC-Ops, 10ServiceOps-Upgrades-Hardware, 06ServiceOps (Next quarter): Q1:rack/setup/install conf101[0-2] - https://phabricator.wikimedia.org/T435426#12338352 (10MLechvien-WMF) [13:54:47] (03PS3) 10Tiziano Fogli: admin: add derenrich to analytics-privatedata-users (krb) [puppet] - 10https://gerrit.wikimedia.org/r/1339911 (https://phabricator.wikimedia.org/T437668) (owner: 10Ssingh) [13:54:58] <_joe_> Kemayo: the reason for the emergency isn't clear from the commit message, or the task [13:55:04] <_joe_> care to elaborate? [13:55:08] (03CR) 10CI reject: [V:04-1] mw.DesktopArticleTarget: if source education is enabled suppress welcome [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343065 (https://phabricator.wikimedia.org/T434249) (owner: 10DLynch) [13:55:32] (03CR) 10Tiziano Fogli: [C:03+2] admin: add volker-e to analytics-privatedata-users (krb) [puppet] - 10https://gerrit.wikimedia.org/r/1339865 (https://phabricator.wikimedia.org/T437634) (owner: 10Ssingh) [13:56:21] _joe_: I was just reopening the task with it, but sure: basically, the ticket was implementing an enwiki RfC for a behavior change, and a bit of it wasn't implemented how enwiki expected because of a wording ambiguity. Fixing it is a tiny change, so we'd like to not have this stewing over the weekend. [13:56:51] <_joe_> yeah I saw the code in the meantime, go ahead once it passes CI [13:57:15] 10ops-eqiad, 06SRE, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12338381 (10Jclark-ctr) @Eevans I got notice of parts being delivered. When would you like to schedule? [13:57:24] _joe_: Will do, thanks. [13:59:00] (03PS4) 10Tiziano Fogli: admin: add derenrich to analytics-privatedata-users (krb) [puppet] - 10https://gerrit.wikimedia.org/r/1339911 (https://phabricator.wikimedia.org/T437668) (owner: 10Ssingh) [13:59:23] Kemayo: no objection from RelEng either [14:00:20] (03CR) 10DLynch: "recheck" [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343065 (https://phabricator.wikimedia.org/T434249) (owner: 10DLynch) [14:01:35] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for VolkerE - https://phabricator.wikimedia.org/T437634#12338408 (10tappof) 05Open→03Resolved Patch merged. Access granted. Growthbook permissions can be self-served through IDM. [14:02:02] That said, the CI failure looks like it might be a node-version issue entirely unrelated to the patch, since the error is "ReferenceError: require is not defined in ES module scope, you can use import instead". [14:02:44] (03CR) 10Xcollazo: "Thanks @brouberol@wikimedia.org. I have no merge access to puppet. If you are ok with the patch, can you please merge?" [puppet] - 10https://gerrit.wikimedia.org/r/1342785 (https://phabricator.wikimedia.org/T437896) (owner: 10Xcollazo) [14:03:09] I'm immediately suspicious of several patches for T438085 having just merged. [14:03:09] T438085: Upgrade all CI jobs for WMF-deployed projects from Node 24 to Node 26 - https://phabricator.wikimedia.org/T438085 [14:03:54] James_F: is this something where it's going to fix itself, or is this a more complicated backport / force-merge situation? [14:04:17] (03CR) 10Brouberol: [C:03+2] dumps: re-add mirror PDApps [puppet] - 10https://gerrit.wikimedia.org/r/1342785 (https://phabricator.wikimedia.org/T437896) (owner: 10Xcollazo) [14:07:03] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Add config to build node26 images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1342602 (https://phabricator.wikimedia.org/T437509) (owner: 10Muehlenhoff) [14:07:28] Kemayo: That’s most odd. I specifically tested api-testing and it worked on Node 26. [14:07:47] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1339911 (https://phabricator.wikimedia.org/T437668) (owner: 10Ssingh) [14:08:05] (03CR) 10Tiziano Fogli: [C:03+2] admin: add derenrich to analytics-privatedata-users (krb) [puppet] - 10https://gerrit.wikimedia.org/r/1339911 (https://phabricator.wikimedia.org/T437668) (owner: 10Ssingh) [14:08:51] James_F: Something about the backport to a previous branch perhaps? Older version of yargs being run on a newer node than it expects? [14:08:56] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to analytics-privatedata-users level 3 for derenrich - https://phabricator.wikimedia.org/T437668#12338441 (10tappof) 05Open→03Resolved a:03tappof Patch merged. Access granted. [14:09:08] Kemayo: The version of api-testing in master and wmf/* is the same. [14:09:43] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users, growthbook-customelevatedaccess for Anil - https://phabricator.wikimedia.org/T437611#12338444 (10tappof) 05Open→03Resolved [14:11:19] Kemayo: Might just need a fresh `npm install` on the correct version of Node, but that’s very irritating if so. [14:15:10] 06SRE, 10SRE-Access-Requests: Requesting access to Superset Dashboard for Hany EL Mokadem - https://phabricator.wikimedia.org/T438034#12338478 (10tappof) [14:15:35] James_F: it does look like yargs on master incidentally got updated from 17.3.1 to 17.7.3 yesterday by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1339693 [14:16:15] Aha, Yeah 728d76c5096ab706b169face101d5b07f0725506 [14:16:27] 06SRE, 10SRE-Access-Requests: Requesting access to Superset Dashboard for Hany EL Mokadem - https://phabricator.wikimedia.org/T438034#12338481 (10tappof) [14:16:33] Let’s just backport that to wmf.20, it’s cleanest? [14:16:50] Or I can make a bespoke wmf.20 patch for fixing yargs? [14:18:37] That package-lock one looks like a pain to backport, though I guess it's plausible nothing else has touched it since then. [14:18:52] I’ll do a bespoke, then. [14:20:45] (03PS2) 10Tiziano Fogli: admin/data: add yahya [puppet] - 10https://gerrit.wikimedia.org/r/1343007 (https://phabricator.wikimedia.org/T436294) [14:21:03] FFS, it’s taking ages to clone all of wmf.20. [14:21:57] mediawiki core being slow to clone, how unexpected. ;_; [14:22:09] It’s not core, is the 200 extensions. [14:22:22] I have a very up-to-date core locally, unsurprisingly. :-) [14:22:40] (03CR) 10Tiziano Fogli: [C:03+2] admin/data: add yahya [puppet] - 10https://gerrit.wikimedia.org/r/1343007 (https://phabricator.wikimedia.org/T436294) (owner: 10Tiziano Fogli) [14:23:18] 06SRE, 10LDAP-Access-Requests, 13Patch-For-Review: Request for LDAP NDA Access for CentralNotice Metrics for Yahya - https://phabricator.wikimedia.org/T436294#12338527 (10tappof) 05Open→03Resolved Patch merged. [14:23:36] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1305183 (https://phabricator.wikimedia.org/T429630) (owner: 10MVernon) [14:26:44] (03PS1) 10Jforrester: Re-generate package-lock.json for wmf.20 [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343086 (https://phabricator.wikimedia.org/T438085) [14:26:55] Kemayo: ^^ Sorry for the slowness. [14:27:11] James_F: Nothing to be done about it [14:27:36] (03PS2) 10CWilliams: mediabackups: Run mediabackup processes via systemd [puppet] - 10https://gerrit.wikimedia.org/r/1343056 (https://phabricator.wikimedia.org/T438012) [14:38:03] Kemayo: OK, other than Wikibase’s disaster-zone of flakiness that passes CI. Are you OK to deploy it? [14:38:41] FIRING: [3x] ProbeDown: Service registry1005:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:39:30] James_F: Sure, I can include it. [14:39:38] (03PS1) 10Btullis: network: Add the second Pod range for dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1343090 (https://phabricator.wikimedia.org/T429773) [14:40:59] (03PS2) 10DLynch: mw.DesktopArticleTarget: if source education is enabled suppress welcome [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343065 (https://phabricator.wikimedia.org/T434249) [14:43:13] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12338611 (10VRiley-WMF) We are still awaiting dells response. [14:43:55] James_F: And I can confirm that the apitests run for my patch passes now that it depends-on yours. [14:48:55] (03PS1) 10Aghirelli: eventschemas: add api-platform-schemas repository [puppet] - 10https://gerrit.wikimedia.org/r/1343092 (https://phabricator.wikimedia.org/T438208) [14:49:23] !log jclark@cumin1004 START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie [14:49:38] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12338664 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1004 for host ml-serve1016.eqiad.wmnet with OS trixie [14:50:24] (03PS2) 10Aghirelli: eventschemas: add api-platform-schemas repository [puppet] - 10https://gerrit.wikimedia.org/r/1343092 (https://phabricator.wikimedia.org/T438208) [14:51:03] (03CR) 10DLynch: "recheck" [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343086 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [14:52:13] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip for thumbor toolhub wikifeeds zotero codfw [puppet] - 10https://gerrit.wikimedia.org/r/1342185 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [14:52:33] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip for thumbor toolhub wikifeeds zotero eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1342186 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [14:55:33] (03PS1) 10Btullis: dse-k8s: allow Pod egress to the second eqiad Pod range [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343097 (https://phabricator.wikimedia.org/T429773) [14:55:49] (03CR) 10CI reject: [V:04-1] Re-generate package-lock.json for wmf.20 [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343086 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [14:56:11] (03CR) 10DLynch: "recheck" [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343086 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [14:56:32] My earlier recheck jumped the gun, alas. [14:56:47] Meh. [14:57:16] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12338712 (10VRiley-WMF) proceeding with trying to boot into 6.1.176 [14:58:22] FIRING: GnmiInterfaceCountersDrop: ... [14:58:22] ssw1-d8-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=ssw1-d8-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:58:47] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343090 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [15:00:04] James_F: Wikibase continues to be problematic. [15:00:28] Kemayo: The normal kind of flakiness? [15:01:01] (We shouldn’t be running the selenium tests on wmf/ anyway, I deleted them and someone put them back.) [15:01:03] James_F: Yeah, it's random cypress tests failing. [15:01:32] Kemayo: More worryingly, https://integration.wikimedia.org/ci/job/mediawiki-node26/230/console is stuck again? [15:02:14] James_F: Very unhelpfully stuck, yeah. Just stalled out with no obviously relevant messages. [15:02:54] Gotta love jest. [15:03:13] Kemayo: I can roll back CI to Node 24, but I don’t have confidence that that’s the issue. [15:03:22] RESOLVED: GnmiInterfaceCountersDrop: ... [15:03:22] ssw1-d8-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=ssw1-d8-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [15:04:48] James_F: It progressed slightly, and it did timeout on exactly the same test as last time ("UserLookup › should update menu items based on the API response"), so it's at least reproducible. [15:05:22] FWIW, mediawiki-node26 is green on MW-core master: https://integration.wikimedia.org/ci/job/mediawiki-node26/220/ [15:05:41] So it’s almost certainly just flakiness. [15:06:53] Kemayo: We could C+2 and see if that helps (the submit pipeline uses different dependencies sometimes) [15:07:19] James_F: I don't mind giving it a shot. [15:08:27] (03PS1) 10Ameisenigel: Disable wgMFCustomSiteModules on German Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343100 (https://phabricator.wikimedia.org/T403380) [15:08:53] (03PS5) 10JHathaway: WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 [15:08:55] (03CR) 10Jforrester: [C:03+2] Re-generate package-lock.json for wmf.20 [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343086 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [15:08:58] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342382 (owner: 10JHathaway) [15:12:19] (03PS5) 10Majavah: dynamicproxy: Do not provision backup timer on trixie [puppet] - 10https://gerrit.wikimedia.org/r/1342694 (https://phabricator.wikimedia.org/T301640) [15:12:40] (03CR) 10CI reject: [V:04-1] WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 (owner: 10JHathaway) [15:13:11] (03CR) 10Majavah: [C:03+2] "merging without the parent to unbreak the new hosts on trixie, old hosts are still there to take backups" [puppet] - 10https://gerrit.wikimedia.org/r/1342694 (https://phabricator.wikimedia.org/T301640) (owner: 10Majavah) [15:14:28] Sigh, no, still fails. [15:14:39] Let’s roll back CI to see if that helps. [15:14:43] James_F: Differently though. This time it wasn't a cypress failure. [15:15:03] Yeah, but it’s blocking you. [15:16:20] I'd say we could always force-merge the VE patch since we're confident it's a CI issue... but presumably we're going to hit the same problem for everyone doing backports next week until .20 is done with. [15:16:29] Yeah. [15:16:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:16:53] First things first, I’m re-dropping the browser tests from wmf/. [15:17:33] (03PS1) 10Tiziano Fogli: logstash: drop noisy logs related to vue-router multiple registrations [puppet] - 10https://gerrit.wikimedia.org/r/1343107 (https://phabricator.wikimedia.org/T438387) [15:18:10] (03CR) 10Herron: [C:03+1] logstash: drop noisy logs related to vue-router multiple registrations [puppet] - 10https://gerrit.wikimedia.org/r/1343107 (https://phabricator.wikimedia.org/T438387) (owner: 10Tiziano Fogli) [15:22:53] (03PS6) 10JHathaway: WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 [15:23:03] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342382 (owner: 10JHathaway) [15:23:26] FIRING: [4x] ProbeDown: Service registry1004:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:25:12] (03CR) 10Btullis: [C:03+2] ceph: Add a profile for the administration host tools [puppet] - 10https://gerrit.wikimedia.org/r/1343061 (https://phabricator.wikimedia.org/T435608) (owner: 10Btullis) [15:25:47] (03CR) 10Tiziano Fogli: [C:03+2] logstash: drop noisy logs related to vue-router multiple registrations [puppet] - 10https://gerrit.wikimedia.org/r/1343107 (https://phabricator.wikimedia.org/T438387) (owner: 10Tiziano Fogli) [15:27:30] (03CR) 10CI reject: [V:04-1] WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 (owner: 10JHathaway) [15:28:12] Kemayo: OK, CI is now running and hopefully it’ll merge soon. [15:30:11] James_F: Do we still need to backport the package-lock bump if CI is back to node24? [15:32:27] Kemayo: I think it’s safer to do that than not. [15:32:39] That said, I think it's stalling again. [15:32:43] Though I note that the mediawiki-node24 is… yeah. [15:32:57] Maybe the issue was nothing to do with the CI change. [15:34:20] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12338894 (10VRiley-WMF) When I try to reboot it, it seems to want to default boot into a "SystemRescue" but, monitoring it and choosing the normal Linux OS, it does seem to boot successfully now? But,... [15:34:29] (03PS3) 10DLynch: mw.DesktopArticleTarget: if source education is enabled suppress welcome [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343065 (https://phabricator.wikimedia.org/T434249) [15:35:26] James_F: ^ removed the depends-on from that to see whether it passes without the package-lock while CI's on node24. [15:35:56] * James_F nods. [15:36:08] I’m debugging locally what causes it to be so slow, but work find locally. [15:36:24] (03CR) 10CI reject: [V:04-1] mw.DesktopArticleTarget: if source education is enabled suppress welcome [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343065 (https://phabricator.wikimedia.org/T434249) (owner: 10DLynch) [15:36:53] (03CR) 10Klausman: [C:03+1] dse-k8s: allow Pod egress to the second eqiad Pod range [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343097 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [15:37:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:37:23] James_F: Well, it failed, but it failed because the quibble job is still running on node 26 according to the log. [15:37:50] (03PS1) 10Raymond Ndibe: toollabs-images: Add deprecation notes to bullseye-based images' dockerfile templates [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1343113 (https://phabricator.wikimedia.org/T438356) [15:39:25] That’s odd. [15:39:48] James_F: assuming I'm interpreting https://integration.wikimedia.org/ci/job/quibble-apitests-only-vendor-php83/40481/console correctly, at least. [15:39:58] (03Abandoned) 10Raymond Ndibe: toollabs-images: Add deprecation notes to bullseye-based images' dockerfile templates [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1343113 (https://phabricator.wikimedia.org/T438356) (owner: 10Raymond Ndibe) [15:40:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 15.87% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:42:47] (03CR) 10FNegri: [C:03+1] "We were discussing this just yesterday with @dcaro@wikimedia.org and @rolisaemeka-ctr@wikimedia.org. I think it's ok to remove these files" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1342772 (https://phabricator.wikimedia.org/T400258) (owner: 10Majavah) [15:43:24] Kemayo:: Oh, yes, five different changes at once. :-( [15:44:56] Kemayo: In that case, let’s try to fix the mediawiki-nodexx jobs so we can land the package-lock one. [15:45:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.55% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:45:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 15.77% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:45:50] (03CR) 10CI reject: [V:04-1] Re-generate package-lock.json for wmf.20 [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343086 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [15:51:06] (03CR) 10Majavah: "> We have some people still using those images, so I wonder if we might need an emergency rebuild in case we find a critical bug in one of" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1342772 (https://phabricator.wikimedia.org/T400258) (owner: 10Majavah) [15:52:40] (03CR) 10JMeybohm: [C:04-1] "This is a good find actually, we got lucky nobody tried to use it apart from those who implemented it in first place (and used the correct" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338960 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [15:57:05] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: Q1:rack/setup/install 10 new ceph nodes - https://phabricator.wikimedia.org/T438213#12338994 (10BTullis) [15:59:39] James_F: in terms of smoking guns for that job on that package-log commit... compared to the versions on master, the .20 commit has bumped nwsapi from 2.2.22 to 2.2.27. Since that's actually jest-related, it's very suspicious. [15:59:48] *package-lock [16:00:08] (03PS2) 10Jforrester: Re-generate package-lock.json for wmf.20 [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343086 (https://phabricator.wikimedia.org/T438085) [16:00:17] Kemayo: Just pushed with that fix. [16:01:28] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12339033 (10MoritzMuehlenhoff) Ok, thanks. Please leave the current kernel running, I'll have a closer look next week over the serial console [16:01:49] Kemayo: Upstream report, already fixed but not released: https://github.com/dperini/nwsapi/issues/215 [16:02:12] Fun! [16:03:18] Oy beh. [16:03:38] Kemayo: Landing https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1343118 in master would be appreciated to avoid this going bang for anyone else. [16:03:47] Is there a problem with CI, I was searching for a phabricator ticket on failing wikibase tests but I could not find something. See https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/1343006 for an example with failing tests [16:04:39] physikerwelt: Hmm, that’s odd. We’re fixing this for the wmf/ branch, but that should already be fixed for master. [16:05:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:06:24] physikerwelt: Try now, it should be fixed. [16:07:20] (03CR) 10Jforrester: [C:03+2] Re-generate package-lock.json for wmf.20 [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343086 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [16:08:09] James_F: That, too, failed CI for wikibase reasons. [16:09:14] (03CR) 10CI reject: [V:04-1] Re-generate package-lock.json for wmf.20 [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343086 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [16:09:35] !log jclark@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1016.eqiad.wmnet with OS trixie [16:09:43] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12339062 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1004 for host ml-serve1016.eqiad.wmnet with OS trixie executed with errors: - m... [16:09:45] Yeah, fingers crossed this last one is the charm. [16:12:12] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:18] (03PS1) 10Marco Fossati: Let AA measure eligible readers w/o beta opt-in [extensions/MultimediaViewer] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343122 (https://phabricator.wikimedia.org/T437076) [16:14:28] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 21 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343122 (https://phabricator.wikimedia.org/T437076) (owner: 10Marco Fossati) [16:15:18] 06SRE, 10SRE-swift-storage, 10Pontoon, 06Traffic: /etc/envoy/envoy.yaml empty on pontoon hosts (and thus envoy doesn't work) - https://phabricator.wikimedia.org/T438309#12339088 (10Dzahn) I think I have seen this behaviour with the empty envoy file before and then restarting everything somehow fixed it. B... [16:15:41] (03PS1) 10Btullis: ceph-csi-rbd: rebase the chart on upstream v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343123 (https://phabricator.wikimedia.org/T407166) [16:15:43] (03PS1) 10Btullis: ceph-csi-cephfs: rebase the chart on upstream v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343124 (https://phabricator.wikimedia.org/T407166) [16:16:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for fasw1-f5a-codfw.mgmt.codfw.wmnet is going to expire in 6d 23h 52m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:17:12] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:28] (03Merged) 10jenkins-bot: Re-generate package-lock.json for wmf.20 [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343086 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [16:17:30] Kemayo: It landed! Do we want to ship the VE change? [16:18:56] (03CR) 10DLynch: "recheck" [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343065 (https://phabricator.wikimedia.org/T434249) (owner: 10DLynch) [16:18:58] (03CR) 10Bking: [C:03+1] network: Add the second Pod range for dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1343090 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [16:19:21] James_F: let me see where Peter lands on it first. [16:19:26] Ack. [16:21:46] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [16:21:51] <_joe_> James_F: can we also ship https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PersonalDashboard/+/1343088 ? [16:21:59] <_joe_> the logspam is killing logstash [16:22:28] (03PS2) 10Btullis: network: Add the second Pod range for dse-k8s-eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1343090 (https://phabricator.wikimedia.org/T429773) [16:22:57] _joe_: I’d really like the team responsible to acknowledge the change before we deploy it, just in case it breaks everything in a way that their CI testing doesn't spot. [16:23:11] ack, I'm escalating [16:23:12] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343090 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [16:23:35] <_joe_> I mean the alternative is a hotpatch that drops that error message if name is vue.router [16:23:55] Yeah, which isn’t great but is not going to break prod silently over the weekend. [16:24:16] <_joe_> ok it makes sense, I'll create that patch [16:24:22] <3 [16:24:26] <_joe_> what's the current branch? I will make the patch for that branch [16:24:37] wmf.20 [16:24:53] Sorry, “wmf/1.47.0-wmf.20”. [16:25:16] James_F: who is the team responsible? [16:27:17] ah, the service catalogue ofc :) [16:27:23] (03CR) 10JMeybohm: [C:04-1] "Please sync with @dpogorzelski@wikimedia.org and take a look at his approach in If5e8fb83613dce744cfc0692a0f283fe6f07a4f6 which seems less" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338751 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [16:27:37] hnowlan: Moderator Tools; I posted to them in Slack. [16:27:41] thank you! [16:29:18] (03CR) 10JMeybohm: "This is a kind-of-duplicate of Ib02044c7d16a2599a81dc5df4c1f7e26b38b2259 where I've asked @abran@wikimedia.org to sync up with you on the " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342612 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [16:29:20] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12339123 (10Dzahn) Here is some documentation on how to setup SSH to jump via the bastion hosts. https://wikitech.wikimedia.org/wiki/SRE/Production_access#Setting_up_your_SSH_config [16:29:25] (03PS1) 10Giuseppe Lavagetto: ResourceLoader: hotfix for current logspam over the weekend [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343127 (https://phabricator.wikimedia.org/T438387) [16:29:28] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12339127 (10Dzahn) a:05Dzahn→03None [16:29:52] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12339130 (10Dzahn) a:03LPetty-WMF [16:30:05] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12339132 (10Dzahn) 05Open→03In progress [16:30:19] (03CR) 10Jforrester: [C:03+2] ResourceLoader: hotfix for current logspam over the weekend [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343127 (https://phabricator.wikimedia.org/T438387) (owner: 10Giuseppe Lavagetto) [16:30:53] (03PS3) 10Zabe: Switch local copy of vue-router to a local name so it doesn't clash with MW's [extensions/PersonalDashboard] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343126 (https://phabricator.wikimedia.org/T438387) [16:31:15] <_joe_> Kemayo: should I wait for your change as well? Or should we ship this now? [16:31:22] (03Abandoned) 10Zabe: Switch local copy of vue-router to a local name so it doesn't clash with MW's [extensions/PersonalDashboard] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343126 (https://phabricator.wikimedia.org/T438387) (owner: 10Zabe) [16:32:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.42% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:32:55] _joe_: ship yours now, I’m still waiting on product. [16:33:36] _joe_: oh, actually, just got the go ahead. They can be thrown together. [16:33:37] (03CR) 10TrainBranchBot: [C:03+2] "Approved by oblivian@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343127 (https://phabricator.wikimedia.org/T438387) (owner: 10Giuseppe Lavagetto) [16:33:51] <_joe_> heh one second to late [16:35:32] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12339165 (10LPetty-WMF) Hi @Dzahn thanks for the detailed notes, I think Im OK now. I was working with @Cklimas and am now able get a shell at both `releases1003.eqiad.wmnet` and `bast... [16:36:04] (03Merged) 10jenkins-bot: ResourceLoader: hotfix for current logspam over the weekend [core] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343127 (https://phabricator.wikimedia.org/T438387) (owner: 10Giuseppe Lavagetto) [16:37:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.42% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:37:46] !log oblivian@deploy1003 Started scap sync-world: Backport for [[gerrit:1343127|ResourceLoader: hotfix for current logspam over the weekend (T438387)]] [16:37:49] T438387: Noisy log: normalized_message: ResourceLoader duplicate registration warning. Another module has already been registered as vue-router - https://phabricator.wikimedia.org/T438387 [16:38:31] (03PS1) 10JHathaway: nginx: fix specs [puppet] - 10https://gerrit.wikimedia.org/r/1343128 [16:39:22] (03Restored) 10Jforrester: Switch local copy of vue-router to a local name so it doesn't clash with MW's [extensions/PersonalDashboard] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343126 (https://phabricator.wikimedia.org/T438387) (owner: 10Zabe) [16:39:51] (03CR) 10Jforrester: "We might still want to back-port this too." [extensions/PersonalDashboard] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343126 (https://phabricator.wikimedia.org/T438387) (owner: 10Zabe) [16:41:34] (03CR) 10Dreamy Jazz: nlwiki: enable SecurePoll local elections (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) (owner: 10Novem Linguae) [16:42:01] !log oblivian@deploy1003 oblivian: Backport for [[gerrit:1343127|ResourceLoader: hotfix for current logspam over the weekend (T438387)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.41% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:42:41] !log oblivian@deploy1003 oblivian: Continuing with deployment [16:46:01] (03PS2) 10Btullis: ceph-csi-rbd: rebase the chart on upstream v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343123 (https://phabricator.wikimedia.org/T407166) [16:46:01] (03PS2) 10Btullis: ceph-csi-cephfs: rebase the chart on upstream v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343124 (https://phabricator.wikimedia.org/T407166) [16:46:01] (03PS1) 10Btullis: admin_ng: pin the ceph-csi charts, and let codfw go first [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343130 (https://phabricator.wikimedia.org/T407166) [16:47:32] (03CR) 10Novem Linguae: nlwiki: enable SecurePoll local elections (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) (owner: 10Novem Linguae) [16:49:22] !log oblivian@deploy1003 Finished scap sync-world: Backport for [[gerrit:1343127|ResourceLoader: hotfix for current logspam over the weekend (T438387)]] (duration: 11m 36s) [16:49:25] T438387: Noisy log: normalized_message: ResourceLoader duplicate registration warning. Another module has already been registered as vue-router - https://phabricator.wikimedia.org/T438387 [16:50:59] I will go ahead with mine now. [16:51:34] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343065 (https://phabricator.wikimedia.org/T434249) (owner: 10DLynch) [16:52:17] (03PS5) 10Zabe: Switch local copy of vue-router to a local name so it doesn't clash with MW's [extensions/PersonalDashboard] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343126 (https://phabricator.wikimedia.org/T438387) [16:52:17] (03CR) 10Zabe: "@jsherman@wikimedia.org I rebase the patch. Should work like this." [extensions/PersonalDashboard] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343126 (https://phabricator.wikimedia.org/T438387) (owner: 10Zabe) [16:52:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438282#12339242 (10Jhancock.wm) a:03Jhancock.wm balanced power. last alert was at 11:43 local time. waiting to see if it is resolved. [16:55:13] (03PS3) 10Btullis: ceph-csi-rbd: rebase the chart on upstream v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343123 (https://phabricator.wikimedia.org/T407166) [16:55:13] (03PS3) 10Btullis: ceph-csi-cephfs: rebase the chart on upstream v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343124 (https://phabricator.wikimedia.org/T407166) [16:56:25] FIRING: [8x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:57:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:59:16] (03Merged) 10jenkins-bot: mw.DesktopArticleTarget: if source education is enabled suppress welcome [extensions/VisualEditor] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343065 (https://phabricator.wikimedia.org/T434249) (owner: 10DLynch) [16:59:31] !log kemayo@deploy1003 Started scap sync-world: Backport for [[gerrit:1343065|mw.DesktopArticleTarget: if source education is enabled suppress welcome (T434249)]] [16:59:35] T434249: Design user experience for helping newcomers discover source editing (desktop, en.wiki) - https://phabricator.wikimedia.org/T434249 [16:59:54] Zabe: thanks! I'll get in line for backporting [16:59:58] (03PS1) 10Btullis: install_server: Add a UEFI partman recipe for cephosd servers [puppet] - 10https://gerrit.wikimedia.org/r/1343133 (https://phabricator.wikimedia.org/T438213) [17:02:36] (03CR) 10Jsn.sherman: [C:03+1] "LGTM! Will backport this as soon as the lane is open." [extensions/PersonalDashboard] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343126 (https://phabricator.wikimedia.org/T438387) (owner: 10Zabe) [17:03:18] JSherman: If there’s no need to emergency back-port over the weekend, we could also wait for Monday? [17:03:51] !log kemayo@deploy1003 kemayo: Backport for [[gerrit:1343065|mw.DesktopArticleTarget: if source education is enabled suppress welcome (T434249)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:04:31] !log kemayo@deploy1003 kemayo: Continuing with deployment [17:05:25] James_F: it's midday friday for me, so I'm happy to backport right after Kemayo: if SRE is okay with it. [17:05:45] !log Created `securepoll_log` on `nlwiki` main DB cluster for T434045 [17:05:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:05:49] T434045: Enable SecurePoll on nl.wikipedia.org - https://phabricator.wikimedia.org/T434045 [17:06:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for fasw1-f5a-codfw.mgmt.codfw.wmnet is going to expire in 6d 23h 2m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:06:25] FIRING: [8x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:07:38] (03CR) 10Dreamy Jazz: nlwiki: enable SecurePoll local elections (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) (owner: 10Novem Linguae) [17:08:04] (03PS3) 10Dreamy Jazz: nlwiki: enable SecurePoll local elections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) (owner: 10Novem Linguae) [17:08:56] !log kemayo@deploy1003 Finished scap sync-world: Backport for [[gerrit:1343065|mw.DesktopArticleTarget: if source education is enabled suppress welcome (T434249)]] (duration: 09m 26s) [17:09:01] T434249: Design user experience for helping newcomers discover source editing (desktop, en.wiki) - https://phabricator.wikimedia.org/T434249 [17:09:20] JSherman: I'm done if you want to go. [17:09:38] do I need an official sre go ahead from somebody? [17:10:25] JSherman: For anything on days like today there's technically an IRC message template you're supposed to fill in, from: https://wikitech.wikimedia.org/wiki/Deployments/Emergencies [17:11:04] kemayo: thanks! [17:11:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:13:36] (03CR) 10Dreamy Jazz: [C:03+1] nlwiki: enable SecurePoll local elections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) (owner: 10Novem Linguae) [17:14:59] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 21 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) (owner: 10Novem Linguae) [17:16:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:16:25] FIRING: [8x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:17:13] (03CR) 10Dreamy Jazz: [C:03+1] eswiki: Add abusefilter-access-protected-vars to abusefilter user group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1342825 (https://phabricator.wikimedia.org/T436652) (owner: 10Codename Noreste) [17:17:46] (03CR) 10Atsuko: [C:03+1] "Looks good! TIL the use-case for "~"" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343130 (https://phabricator.wikimedia.org/T407166) (owner: 10Btullis) [17:19:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:22:21] (03CR) 10Novem Linguae: nlwiki: enable SecurePoll local elections (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337580 (https://phabricator.wikimedia.org/T434045) (owner: 10Novem Linguae) [17:23:04] Okay, I did the request over in releng and tyler is having a look for approval; cc herron: [17:24:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:24:36] (03CR) 10Atsuko: [C:03+1] "LGTM, partman config is not my forte, but it should be safe to merge and try" [puppet] - 10https://gerrit.wikimedia.org/r/1343133 (https://phabricator.wikimedia.org/T438213) (owner: 10Btullis) [17:25:12] ack thanks JSherman [17:25:33] JSherman: I affirmed it's ok with me in -releng. Though reading scrollback here, it looks like the emergency part may already be covered by other changes? [17:27:52] bb, [17:29:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.42% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:33:33] taking a pause. I suppose my question is: if it doesn't warrant an emergency backport, does it warrant a backport at all, or should the local rename patch for PersonalDashboard just ride the next train? [17:34:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:36:08] JSherman: I believe adding it to next train is the right thing with the other mitigations that rolled out. [17:36:25] FIRING: [8x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:36:47] Basing that on the last message on the task: https://phabricator.wikimedia.org/T438387#12339274 [17:38:04] we'll need to get changes in before wmf.21 so we can drop the resource loader logging change. [17:38:14] * thcipriani adds as actual train blocker [17:39:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 18.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:44:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:46:25] FIRING: [8x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:50:41] PROBLEM - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1194 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [17:50:43] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1194 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T438526 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [17:52:01] thcipriani: herron: James_F: okay, I'm officially cooling my jets. thanks for being patient with me as I bumbled my way through here. [17:52:47] see an UBN and fire up jets is a fine response :) [17:53:01] no worries! thanks JSherman [18:01:44] (03CR) 10CDanis: [C:03+1] nginx: fix specs [puppet] - 10https://gerrit.wikimedia.org/r/1343128 (owner: 10JHathaway) [18:01:57] (03CR) 10JHathaway: [C:03+2] nginx: fix specs [puppet] - 10https://gerrit.wikimedia.org/r/1343128 (owner: 10JHathaway) [18:06:25] FIRING: [8x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:08:48] FIRING: PuppetFailure: Puppet has failed on krb1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:26:25] FIRING: [8x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:33:47] (03CR) 10RLazarus: "Does this also need to be added to profile::swift::proxy::private_container_list? https://gerrit.wikimedia.org/r/plugins/gitiles/operation" [puppet] - 10https://gerrit.wikimedia.org/r/1339694 (https://phabricator.wikimedia.org/T437403) (owner: 10Zabe) [18:34:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [18:37:56] (03CR) 10Zabe: Add config for conductwiki (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1292346 (https://phabricator.wikimedia.org/T426984) (owner: 10Ladsgroup) [18:39:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [18:39:55] (03CR) 10Zabe: "That list appears to be unused, see comment at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1292346." [puppet] - 10https://gerrit.wikimedia.org/r/1339694 (https://phabricator.wikimedia.org/T437403) (owner: 10Zabe) [18:41:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.59% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [18:42:34] (03CR) 10RLazarus: [C:03+1] "Lol, amazing. Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1339694 (https://phabricator.wikimedia.org/T437403) (owner: 10Zabe) [18:43:27] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343140 [18:44:18] (03PS7) 10JHathaway: WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 [18:46:25] FIRING: [9x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:48:19] (03CR) 10CI reject: [V:04-1] WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 (owner: 10JHathaway) [18:51:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [18:52:09] (03PS1) 10RLazarus: hieradata: Remove profile::swift::proxy::private_container_list [puppet] - 10https://gerrit.wikimedia.org/r/1343143 (https://phabricator.wikimedia.org/T334488) [18:52:51] (03PS2) 10RLazarus: hieradata: Remove profile::swift::proxy::private_container_list [puppet] - 10https://gerrit.wikimedia.org/r/1343143 (https://phabricator.wikimedia.org/T334488) [19:03:48] RESOLVED: PuppetFailure: Puppet has failed on krb1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:18:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.11% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:23:41] FIRING: [4x] ProbeDown: Service registry1004:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:36:25] FIRING: [9x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:41:43] 06SRE, 06RoadToWiki, 06Traffic, 10WMF-General-or-Unknown: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12340191 (10Chuiimuii_ofc) RoadToWiki board hygiene: moved this task to **Completed** so the board matches Phabricator status (resolved/clos... [19:43:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.8% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:46:25] FIRING: [9x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:58:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.17% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:03:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:09:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:17:12] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:21:47] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [20:28:03] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1194 - https://phabricator.wikimedia.org/T438526#12340407 (10VRiley-WMF) a:03VRiley-WMF [20:34:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:36:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:41:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:51:48] FIRING: PuppetFailure: Puppet has failed on krb1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [20:52:44] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1194 - https://phabricator.wikimedia.org/T438526#12340506 (10VRiley-WMF) opened a ticked with Dell for a replacement drive SR231912612. [20:54:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:06:25] FIRING: [9x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:06:38] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for fasw1-f5a-codfw.mgmt.codfw.wmnet is going to expire in 6d 19h 2m 7s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:09:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:16:25] FIRING: [9x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:16:54] (03PS1) 10JHathaway: ssh server: fix match indentation [puppet] - 10https://gerrit.wikimedia.org/r/1343156 [21:18:05] (03CR) 10CI reject: [V:04-1] ssh server: fix match indentation [puppet] - 10https://gerrit.wikimedia.org/r/1343156 (owner: 10JHathaway) [21:21:03] (03PS2) 10JHathaway: ssh server: fix match indentation [puppet] - 10https://gerrit.wikimedia.org/r/1343156 [21:23:20] (03PS8) 10JHathaway: WIP: stdlib [puppet] - 10https://gerrit.wikimedia.org/r/1342382 [21:29:17] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343156 (owner: 10JHathaway) [21:29:22] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342382 (owner: 10JHathaway) [21:30:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 15.43% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [21:36:25] FIRING: [9x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:43:36] (03CR) 10Jdlrobson: [C:03+1] Disable wgMFCustomSiteModules on German Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343100 (https://phabricator.wikimedia.org/T403380) (owner: 10Ameisenigel) [21:46:25] FIRING: [9x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:55:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [22:01:48] RESOLVED: PuppetFailure: Puppet has failed on krb1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [22:13:24] (03PS1) 10RLazarus: mwscript_k8s: Call expanduser() on input paths [puppet] - 10https://gerrit.wikimedia.org/r/1343171 (https://phabricator.wikimedia.org/T415835) [22:14:23] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12340906 (10Dzahn) Hi @LPetty-WMF ah, great! That's good to hear. Yea, you can totally test this beforehand. You can make a test file and upload it into your home directory on the ser... [22:15:02] 06SRE, 10SRE-Access-Requests: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12340911 (10Dzahn) 05In progress→03Resolved Optimistically calling it resolved. Feel free to keep commenting here if you run into any issues or have questions. [22:29:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [22:29:53] FIRING: SessionStoreErrorRateHigh: Session storage error rates (5xx) in eqiad are elevated #page - TODO - https://grafana.wikimedia.org/d/000001590/sessionstore?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSessionStoreErrorRateHigh [22:30:11] looking [22:30:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 18.46% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [22:31:07] o/ [22:31:23] cassandra seems healthy afaict [22:32:05] looks like there was some phpfpmtoobusies earlier today? [22:32:15] something at the sessionstore level -- on a hunch I'm going to peek at the hosts urandom was nervous about the other day [22:32:31] ok [22:32:38] PHPFPMTooBusy has been flapping on and off for the last couple weeks, I wouldn't read into it right now [22:32:43] ok [22:33:37] if it is something hardwarey, we can always depool sessionstore from eqiad, but I'd like to know it isn't something trafficky first obviously [22:35:01] https://grafana.wikimedia.org/goto/s9k764?orgId=default sessionstore1005 CPU goes to zero at 22:25, and the other sessionstore1* hosts don't. that's not a smoking gun but it's suspicious [22:37:50] yeah I see disk errors [22:37:54] yeah I just got there too [22:37:56] let's depool [22:38:11] ack [22:38:15] the host or the dc, what do you think? [22:38:33] I would think the host would be sufficient, but I don't know details [22:38:33] depooling the host takes us back to that "no redundancy" state urandom was concerned about, going into the weekend [22:38:40] ok [22:38:47] the dc it is! [22:39:10] yeah I guess so -- shame about the latency, but let's stabilize first and we can always reconsider [22:39:26] want to do it or shall I? :) [22:39:32] go forit [22:40:48] check me? # confctl --object-type discovery select 'dnsdisc=sessionstore,name=eqiad' set/pooled=false [22:41:17] lgtm! [22:41:21] !log rzl@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=sessionstore,name=eqiad [22:41:27] FWIW, the other 5 hosts don't have disk errors in recent syslog like 1005 does [22:41:49] cool, I'd cross-checked with one but not all of them [22:42:04] we'll need a dcops ticket too I guess [22:42:35] ttl on that dnsdisc record will be up at 22:46, for the record [22:43:55] I'll make the ticket [22:43:56] previously on battlestar galactica: https://phabricator.wikimedia.org/T437915 [22:44:07] I guess we don't even need a new one, if that one's still open [22:44:21] heh [22:48:39] edit failures are trending back down, which is what I most wanted to see https://grafana.wikimedia.org/goto/sfrmtq?orgId=default [22:49:53] RESOLVED: SessionStoreErrorRateHigh: Session storage error rates (5xx) in eqiad are elevated #page - TODO - https://grafana.wikimedia.org/d/000001590/sessionstore?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSessionStoreErrorRateHigh [22:49:59] the alert for 5xxs in eqiad should clear shortly too-- damn :) [22:50:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [22:50:17] -- and I expect it'll be good for worker saturation. we'll run a tiny bit hotter because of the roundtrip, but probably not perceptibly [22:50:20] damn! [22:50:34] got to type faster if I want to look like I know what's coming [22:51:23] 10ops-eqiad, 06SRE, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12341019 (10BBlack) We depooled all sessionstore in eqiad today, as `sessionstore1005` was misbehaving and causing paging alerts for `FIRING: SessionStoreErrorRateHigh:` as we... [22:51:28] https://phabricator.wikimedia.org/T437915#12341019 [22:51:40] 👍 [22:52:21] so now that we have some breathing room, we can decide whether depooling the machine or the DC was the right call -- happy to revisit but from u.random's post at the top of that ticket, this sounds like what he would probably do [22:53:06] next week of course everything *else* will move to codfw anyway, so it won't be cross-dc for long [22:53:15] no, based on the ticket desc, depooling a single node sounds like a terrible idea [22:54:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [22:54:32] oh lol he's out of office all next week too [22:55:02] hopefully somebody else in data persistence can take a look then. I think this is the right friday afternoon state, at least [22:55:21] anything else you think we need to do here? [22:57:26] (ha, I was going to ping k.wakuofori on the task to let him know it needs attention before u.random is back, but they're *both* out on Monday. I'll do that anyway and leave a note in their team channel too, and hope for the best) [22:58:43] 10ops-eqiad, 06SRE, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12341070 (10RLazarus) And cc @KOfori for awareness -- I see @Eevans is OOO until the 28th, this will probably need attention before he comes back. [23:09:21] ack, thanks [23:23:41] FIRING: [4x] ProbeDown: Service registry1004:5001 has failed probes (http_docker_registry_health_ip4) - https://wikitech.wikimedia.org/wiki/Docker - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:30:02] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1342820 (owner: 10TrainBranchBot) [23:30:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 17.25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [23:36:25] FIRING: [9x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:40:55] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1343178 [23:40:55] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1343178 (owner: 10TrainBranchBot) [23:46:25] FIRING: [9x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:46:40] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1343178 (owner: 10TrainBranchBot) [23:55:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [23:57:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.7% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy