[00:04:07] RECOVERY - snapshot of x1 in codfw on backupmon1001 is OK: Last snapshot for x1 at codfw (db2197) taken on 2026-09-25 23:11:43 (414 GiB, +1.4 %) https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [00:12:49] RECOVERY - snapshot of s7 in codfw on backupmon1001 is OK: Last snapshot for s7 at codfw (db2198) taken on 2026-09-25 23:05:21 (1016 GiB, +0.0 %) https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [00:43:15] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 637.41 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:52:41] PROBLEM - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1200 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [00:52:43] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1200 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T439299 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [00:52:52] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439299 (10ops-monitoring-bot) 03NEW [01:08:59] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345244 [01:08:59] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345244 (owner: 10TrainBranchBot) [01:09:57] RECOVERY - MariaDB Replica Lag: s4 on db1265 is OK: OK slave_sql_lag Replication lag: 0.27 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:17:28] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345244 (owner: 10TrainBranchBot) [01:51:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:00:29] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:03:57] PROBLEM - MariaDB Replica Lag: s3 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 624.26 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:06:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:07:43] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:25:15] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.37 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [03:00:37] (03PS7) 10Andrea Denisse: oncall: Add Klaxon to the aux clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344072 (https://phabricator.wikimedia.org/T438897) [03:09:21] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs2008 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:10:13] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2008 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [03:24:25] (03PS1) 10Ladsgroup: Disable Score exec [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345252 (https://phabricator.wikimedia.org/T439297) [03:24:31] (03PS2) 10JHathaway: stdlib: upgrade to v10.1.0 [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) [03:24:40] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [03:24:43] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345252 (https://phabricator.wikimedia.org/T439297) (owner: 10Ladsgroup) [03:25:44] (03Merged) 10jenkins-bot: Disable Score exec [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345252 (https://phabricator.wikimedia.org/T439297) (owner: 10Ladsgroup) [03:26:29] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1345252|Disable Score exec (T439297 T438443)]] [03:30:42] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1345252|Disable Score exec (T439297 T438443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [03:31:49] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [03:37:31] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345252|Disable Score exec (T439297 T438443)]] (duration: 11m 01s) [03:50:57] RECOVERY - MariaDB Replica Lag: s3 on db1265 is OK: OK slave_sql_lag Replication lag: 0.22 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [05:51:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:06:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:19:06] (03PS2) 10Giuseppe Lavagetto: runtimeClasses: add gVisor-kvm RuntimeClass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342964 (https://phabricator.wikimedia.org/T438287) [06:19:06] (03PS2) 10Giuseppe Lavagetto: shellbox-timeline: run under kvm in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342965 (https://phabricator.wikimedia.org/T438287) [06:19:06] (03PS1) 10Giuseppe Lavagetto: shellbox-score: enable gVisor [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345257 [06:19:33] (03PS2) 10Giuseppe Lavagetto: shellbox-score: enable gVisor [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345257 [06:22:58] (03CR) 10Rsilvola: [C:03+1] shellbox-score: enable gVisor [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345257 (owner: 10Giuseppe Lavagetto) [06:24:29] (03CR) 10Giuseppe Lavagetto: [C:03+2] shellbox-score: enable gVisor [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345257 (owner: 10Giuseppe Lavagetto) [06:27:00] (03Merged) 10jenkins-bot: shellbox-score: enable gVisor [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345257 (owner: 10Giuseppe Lavagetto) [06:27:32] !log oblivian@deploy1003 helmfile [codfw] START helmfile.d/services/shellbox: apply [06:28:08] !log oblivian@deploy1003 helmfile [codfw] DONE helmfile.d/services/shellbox: apply [06:29:21] !log oblivian@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: apply [06:29:29] !log oblivian@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: apply [06:30:17] !log oblivian@deploy1003 helmfile [eqiad] START helmfile.d/services/shellbox: apply [06:30:57] !log oblivian@deploy1003 helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [06:32:57] (03PS1) 10Giuseppe Lavagetto: Revert "Disable Score exec" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345258 [06:34:34] (03PS1) 10Ladsgroup: Empty commit to trigger rebuild [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1345259 (https://phabricator.wikimedia.org/T439297) [06:41:01] (03CR) 10Rsilvola: [C:03+2] Empty commit to trigger rebuild [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1345259 (https://phabricator.wikimedia.org/T439297) (owner: 10Ladsgroup) [06:49:43] (03Merged) 10jenkins-bot: Empty commit to trigger rebuild [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1345259 (https://phabricator.wikimedia.org/T439297) (owner: 10Ladsgroup) [07:14:17] (03PS1) 10Ladsgroup: thumbor: Bump to the most recent version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345261 (https://phabricator.wikimedia.org/T439297) [07:30:37] (03PS1) 10Giuseppe Lavagetto: thumbor: update version to 2026-09-26-065005-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345262 [07:37:52] (03CR) 10Giuseppe Lavagetto: [C:03+2] thumbor: update version to 2026-09-26-065005-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345262 (owner: 10Giuseppe Lavagetto) [07:40:14] (03Merged) 10jenkins-bot: thumbor: update version to 2026-09-26-065005-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345262 (owner: 10Giuseppe Lavagetto) [07:42:05] !log oblivian@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [07:42:11] !log oblivian@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [07:44:00] !log oblivian@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [07:46:04] !log oblivian@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [07:50:29] !log oblivian@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [07:52:28] !log oblivian@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [07:55:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by oblivian@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345258 (owner: 10Giuseppe Lavagetto) [07:56:39] (03Merged) 10jenkins-bot: Revert "Disable Score exec" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345258 (owner: 10Giuseppe Lavagetto) [07:56:52] !log oblivian@deploy1003 Started scap sync-world: Backport for [[gerrit:1345258|Revert "Disable Score exec"]] [08:00:54] !log oblivian@deploy1003 oblivian: Backport for [[gerrit:1345258|Revert "Disable Score exec"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:02:11] !log oblivian@deploy1003 oblivian: Continuing with deployment [08:07:45] !log oblivian@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345258|Revert "Disable Score exec"]] (duration: 10m 53s) [09:51:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:06:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:38:20] FIRING: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 89.88467111111112 times in the last 10 minutes due to memory usage (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [11:43:20] RESOLVED: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 23.333333333333336 times in the last 10 minutes due to memory usage (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [11:48:35] (03CR) 10Phuedx: [C:03+1] [beta eventgate] Migrate urls to deployment-eventgate05 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345154 (https://phabricator.wikimedia.org/T429497) (owner: 10TChin) [12:48:27] (03CR) 10Tryvix1509: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345052 (https://phabricator.wikimedia.org/T439159) (owner: 10Ameisenigel) [12:49:22] (03CR) 10Tryvix1509: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344216 (https://phabricator.wikimedia.org/T386776) (owner: 10Ameisenigel) [13:01:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:17:22] FIRING: CertAlmostExpired: gNMI TLS certificate for ssw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:32:22] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:37:22] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:50:40] (03PS3) 10JHathaway: stdlib: upgrade to v10.1.0 [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) [13:50:53] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [13:54:29] (03CR) 10CI reject: [V:04-1] stdlib: upgrade to v10.1.0 [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [14:06:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:35:15] 06SRE, 06serviceops-deprecated, 07Wikimedia-Performance-recommendation: Evaluate using igbinary for MW php-apcu at WMF - https://phabricator.wikimedia.org/T225074#12366409 (10Krinkle) @MGoncalves-WMF Two questions: Is this with PHP 8.5 on Debian Bookworm? (i.e. latest MediaWiki-Docker `dev/bookworm-php85-fp... [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:29:49] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [16:30:15] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [16:30:16] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [16:30:46] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [16:37:18] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [16:37:20] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [16:37:21] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [16:37:25] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [16:44:25] PROBLEM - Kafka broker TLS certificate validity on kafka-logging1003 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [16:44:49] PROBLEM - Kafka Broker Server on kafka-logging1003 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [16:45:41] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:37:22] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:06:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:07:33] PROBLEM - Host wikikube-worker2280 is DOWN: PING CRITICAL - Packet loss = 100% [18:07:57] FIRING: ProbeDown: Service mw-web-next:4454 has failed probes (http_mw-web-next_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#mw-web-next:4454 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:09:23] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - mw-web-next_4454: Servers wikikube-worker2033.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2136.codfw.wmn [18:09:23] kube-worker2185.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2314.codfw.wmnet, wikikube-worker20 [18:09:23] .wmnet, wikikube-worker2060.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2160.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2002.codfw.wmnet, wikikube-wor https://wikitech.wikimedia.org/wiki/PyBal [18:09:33] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - mw-web-next_4454: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2262.codfw.wmnet, wikikube-worker2033.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2136.codfw.wmn [18:09:33] kube-worker2276.codfw.wmnet, wikikube-worker2312.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker21 [18:09:33] .wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-wor https://wikitech.wikimedia.org/wiki/PyBal [18:10:01] RECOVERY - Host wikikube-worker2280 is UP: PING OK - Packet loss = 0%, RTA = 30.44 ms [18:11:33] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:12:23] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:12:57] RESOLVED: ProbeDown: Service mw-web-next:4454 has failed probes (http_mw-web-next_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#mw-web-next:4454 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:13:27] FIRING: [3x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:17:58] * volans looking [18:23:10] (03PS1) 10Krinkle: trafficserver: Document and test normalize-path [puppet] - 10https://gerrit.wikimedia.org/r/1345282 (https://phabricator.wikimedia.org/T425216) [19:24:03] 06SRE, 06serviceops-deprecated, 07Wikimedia-Performance-recommendation: Evaluate using igbinary for MW php-apcu at WMF - https://phabricator.wikimedia.org/T225074#12366494 (10Krinkle) A few APCU read/write rates from https://grafana.wikimedia.org/d/U7JT--knk/mediawiki-on-k8s (APCU inserts, and APCU hits): |... [21:13:15] PROBLEM - MariaDB Replica Lag: x1 on db2201 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 308.25 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:13:15] PROBLEM - MariaDB Replica Lag: x1 on db2197 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 308.26 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:13:50] PROBLEM - MariaDB Replica Lag: x1 #page on db2249 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 325.96 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:13:50] PROBLEM - MariaDB Replica Lag: x1 #page on db2191 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 318.01 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:14:18] !ack [21:14:19] All incidents are already acked. [21:16:35] !incidents [21:16:35] 8604 (ACKED) db2249 (paged)/MariaDB Replica Lag: x1 (paged) [21:16:35] 8605 (ACKED) db2191 (paged)/MariaDB Replica Lag: x1 (paged) [21:16:36] 8603 (RESOLVED) ProbeDown sre (10.2.1.6 ip4 mw-web-next:4454 probes/service http_mw-web-next_ip4 codfw) [21:16:36] 8602 (RESOLVED) Manual (paged) for SRE managers by LSobanski (lsobanski@wikimedia.org): Test page 2 [21:16:36] 8601 (RESOLVED) Manual (paged) for SRE managers by LSobanski (lsobanski@wikimedia.org): Test page [21:23:15] RECOVERY - MariaDB Replica Lag: x1 on db2201 is OK: OK slave_sql_lag Replication lag: 0.11 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:23:15] RECOVERY - MariaDB Replica Lag: x1 on db2197 is OK: OK slave_sql_lag Replication lag: 0.12 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:23:50] RECOVERY - MariaDB Replica Lag: x1 #page on db2191 is OK: OK slave_sql_lag Replication lag: 0.25 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:24:10] !incidents [21:24:11] 8604 (ACKED) db2249 (paged)/MariaDB Replica Lag: x1 (paged) [21:24:11] 8605 (RESOLVED) db2191 (paged)/MariaDB Replica Lag: x1 (paged) [21:24:11] 8603 (RESOLVED) ProbeDown sre (10.2.1.6 ip4 mw-web-next:4454 probes/service http_mw-web-next_ip4 codfw) [21:24:11] 8602 (RESOLVED) Manual (paged) for SRE managers by LSobanski (lsobanski@wikimedia.org): Test page 2 [21:24:11] 8601 (RESOLVED) Manual (paged) for SRE managers by LSobanski (lsobanski@wikimedia.org): Test page [21:26:08] !log krinkle@deploy1003 Started deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 [21:26:18] !log krinkle@deploy1003 Finished deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 (duration: 00m 09s) [21:37:22] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:39:44] (03PS1) 10Zabe: wikibase: Set virtual-wikibase-entityusage to x1 for test wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345298 (https://phabricator.wikimedia.org/T439319) [21:39:50] RECOVERY - MariaDB Replica Lag: x1 #page on db2249 is OK: OK slave_sql_lag Replication lag: 0.04 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:02:05] PROBLEM - SSH on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [22:02:09] PROBLEM - librenms.wikimedia.org requires authentication on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [22:02:49] PROBLEM - librenms.wikimedia.org tls expiry on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [22:04:01] (03PS1) 10Akaza24: plugins: Add wm-group-repos plugin for group repository access view [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1345299 (https://phabricator.wikimedia.org/T433223) [22:06:09] (03PS1) 10Akaza24: plugins: Add wm-group-repos plugin for group repository access view [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1345299 (https://phabricator.wikimedia.org/T433223) [22:06:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:08:39] RECOVERY - librenms.wikimedia.org tls expiry on netmon2002 is OK: OK - Certificate librenms.wikimedia.org will expire on Mon 09 Nov 2026 02:18:41 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [22:08:55] RECOVERY - SSH on netmon2002 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [22:08:59] RECOVERY - librenms.wikimedia.org requires authentication on netmon2002 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 701 bytes in 0.138 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [22:10:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:11:49] PROBLEM - librenms.wikimedia.org tls expiry on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [22:12:05] PROBLEM - SSH on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [22:12:09] PROBLEM - librenms.wikimedia.org requires authentication on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [22:13:27] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:25:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:33:39] RECOVERY - librenms.wikimedia.org tls expiry on netmon2002 is OK: OK - Certificate librenms.wikimedia.org will expire on Mon 09 Nov 2026 02:18:41 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [22:33:55] RECOVERY - SSH on netmon2002 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [22:33:59] RECOVERY - librenms.wikimedia.org requires authentication on netmon2002 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 701 bytes in 0.136 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [22:45:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:38:52] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1345304 [23:38:52] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1345304 (owner: 10TrainBranchBot) [23:39:57] FIRING: CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:47:46] (03CR) 10CI reject: [V:04-1] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1345304 (owner: 10TrainBranchBot)