[00:00:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [00:23:28] (03CR) 10Samwilson: [C:03+1] "Looks good, but I've not figured out how to test this locally." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333249 (owner: 10Ladsgroup-claude) [00:39:56] (03CR) 10Samwilson: [C:04-1] "Looks good, just one small issue." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333250 (owner: 10Ladsgroup-claude) [00:48:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 18.32% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:53:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:13:40] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:27:42] (03PS1) 10Samwilson: Set development log level to 'debug' [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337196 [01:28:20] (03CR) 10CI reject: [V:04-1] Set development log level to 'debug' [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337196 (owner: 10Samwilson) [01:41:23] (03PS2) 10Samwilson: Set development log level to 'debug' [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337196 [01:41:49] (03CR) 10CI reject: [V:04-1] Set development log level to 'debug' [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337196 (owner: 10Samwilson) [02:00:17] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:00:43] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 00m 25s) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:00:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:12:25] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:13:44] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260906T0700) [08:00:06] PROBLEM - Host wikikube-worker1140 is DOWN: PING CRITICAL - Packet loss = 71%, RTA = 2560.68 ms [08:00:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:00:48] RECOVERY - Host wikikube-worker1140 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [08:12:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:47:15] (03CR) 10Ladsgroup: [C:03+2] "ah, I thought you're doing the puppet/mediawiki-config way. I fix it." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333249 (owner: 10Ladsgroup-claude) [08:51:15] (03PS1) 10Ladsgroup: Review access change [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337307 [08:52:27] (03CR) 10Ladsgroup: [C:03+2] "This should fix it I think I685c27d184a9eb" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333249 (owner: 10Ladsgroup-claude) [08:53:36] (03Merged) 10jenkins-bot: Replace python-memcached with pymemcache [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333249 (owner: 10Ladsgroup-claude) [08:54:57] (03PS6) 10Ladsgroup: Call ImageMagick 7's magick instead of the deprecated convert [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333250 (owner: 10Ladsgroup-claude) [08:55:48] (03CR) 10Ladsgroup: Call ImageMagick 7's magick instead of the deprecated convert (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333250 (owner: 10Ladsgroup-claude) [09:12:46] 06SRE, 10Wikimedia-Mailing-lists: Create new mailing lists: wikidata-admins@lists.wikimedia.org - https://phabricator.wikimedia.org/T435638#12292978 (10Jony) @Ladsgroup and @Quiddity, I believe there were no objection in the noticeboard, and second admin email has been added. [09:13:40] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:36:36] 06SRE, 10Wikimedia-Mailing-lists: Create new mailing lists: wikidata-admins@lists.wikimedia.org - https://phabricator.wikimedia.org/T435638#12292988 (10Ladsgroup) 05Stalled→03Resolved a:03Ladsgroup https://lists.wikimedia.org/postorius/lists/wikidata-admins.lists.wikimedia.org I added myself too (no... [09:42:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 16.67% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:49:20] (03PS1) 10Zabe: Migrate querying categorylinks to virtual domain [extensions/UploadWizard] (wmf/1.47.0-wmf.18) - 10https://gerrit.wikimedia.org/r/1337310 (https://phabricator.wikimedia.org/T405812) [09:55:31] (03CR) 10Samwilson: [C:03+1] Call ImageMagick 7's magick instead of the deprecated convert (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333250 (owner: 10Ladsgroup-claude) [10:02:07] (03CR) 10Ladsgroup: [C:03+2] Call ImageMagick 7's magick instead of the deprecated convert (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333250 (owner: 10Ladsgroup-claude) [10:03:04] (03Merged) 10jenkins-bot: Call ImageMagick 7's magick instead of the deprecated convert [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333250 (owner: 10Ladsgroup-claude) [10:07:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:07:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:09:22] (03PS5) 10Ladsgroup-claude: Clean up requirements and add a dependency bump path [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333251 [10:09:23] (03PS7) 10Ladsgroup-claude: Add a unit test layer that runs without the container [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333252 [10:09:23] (03PS5) 10Ladsgroup-claude: Replace flake8 with ruff and apply its fixes [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333253 [10:09:23] (03PS4) 10Ladsgroup-claude: Apply ruff format [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1333254 [10:09:24] (03PS1) 10Ladsgroup-claude: [WIP] Run the Swift result storage read off the event loop [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337312 (https://phabricator.wikimedia.org/T431767) [10:09:26] (03PS1) 10Ladsgroup-claude: [WIP] Stop blocking the loop on the failure-counter memcached write [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337313 (https://phabricator.wikimedia.org/T431767) [10:09:30] (03PS1) 10Ladsgroup-claude: [WIP] Read originals off the event loop in the file loader [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337314 (https://phabricator.wikimedia.org/T431767) [10:09:34] (03PS1) 10Ladsgroup-claude: [WIP] Collect timing headers on the context instead of the request handler [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337315 (https://phabricator.wikimedia.org/T431767) [10:09:38] (03PS1) 10Ladsgroup-claude: [WIP] Give thumbor's engine thread pool a worker [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337316 (https://phabricator.wikimedia.org/T431767) [10:09:42] (03PS1) 10Ladsgroup-claude: [WIP] Run engine.load() on the engine thread pool too [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337317 (https://phabricator.wikimedia.org/T431767) [10:32:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:50:23] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:31:08] (03PS2) 10Zabe: Move production reads for commons link tables to x4 for 1% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) [11:31:49] (03CR) 10Zabe: "I like it. Simply to implement and should do the desired job for our application." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [11:32:49] (03PS1) 10Zabe: Move production reads for commons link tables to x4 for all of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 [11:33:13] (03PS2) 10Zabe: Move production reads for commons link tables to x4 for all of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337319 (https://phabricator.wikimedia.org/T437108) [11:36:39] (03PS3) 10Zabe: Move production reads for commons link tables to x4 for 1% of requests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) [12:10:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 14.53% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:12:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:30:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:34:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:39:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:49:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:54:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:13:40] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:07:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:12:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.04% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:25:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:30:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:30:32] PROBLEM - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1200 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 2 UGood : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [14:30:34] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1200 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 2 UGood : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T437170 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [14:30:45] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T437170 (10ops-monitoring-bot) 03NEW [14:50:38] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:13:40] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:22:14] PROBLEM - Host cp1109 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 2780.25 ms [17:22:44] RECOVERY - Host cp1109 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [18:49:52] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:50:38] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:59:12] (03CR) 10Ladsgroup: [C:03+1] "Before deploying this, we can have this without the 1% in mw-experimental and play with it. I can try it tomorrow "morning". Most importan" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1336120 (https://phabricator.wikimedia.org/T437108) (owner: 10Zabe) [20:08:20] (03PS1) 10Gergő Tisza: profile::mediawiki::php: Add unserialize_callback_func php.ini var [puppet] - 10https://gerrit.wikimedia.org/r/1337377 (https://phabricator.wikimedia.org/T389402) [20:12:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:39:24] (03PS1) 10Ladsgroup: Enable thumb.wikimedia.org everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337379 (https://phabricator.wikimedia.org/T427465) [20:45:02] (03CR) 10Ladsgroup: "So Sam and rest of MMPT can also merge patches" [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337307 (owner: 10Ladsgroup) [21:13:40] FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:15:24] (03CR) 10Gmodena: [C:03+1] WDQS: Do not keep any index backups on the PVC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335776 (https://phabricator.wikimedia.org/T436802) (owner: 10Trueg) [22:50:07] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [22:50:38] FIRING: [8x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:36:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1047:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1047 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [23:41:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1047:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1047 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown