[00:04:56] (03PS8) 10BryanDavis: httpbb: Add test suite for testwiki [puppet] - 10https://gerrit.wikimedia.org/r/1323829 (https://phabricator.wikimedia.org/T428972) [00:04:57] (03PS2) 10BryanDavis: httpbb: sort managed directory list [puppet] - 10https://gerrit.wikimedia.org/r/1326939 [00:11:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [00:13:28] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:13:56] (03CR) 10BryanDavis: httpbb: Add test suite for testwiki (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1323829 (https://phabricator.wikimedia.org/T428972) (owner: 10BryanDavis) [00:28:37] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:28:40] (03CR) 10Scott French: [C:03+1] "Thanks, Bryan!" [puppet] - 10https://gerrit.wikimedia.org/r/1323829 (https://phabricator.wikimedia.org/T428972) (owner: 10BryanDavis) [00:35:41] (03PS1) 10Cwhite: logstash: enable passing authentication headers to dashboards [puppet] - 10https://gerrit.wikimedia.org/r/1326956 (https://phabricator.wikimedia.org/T350516) [00:40:04] (03PS1) 10Cwhite: beta-logs: enable security plugin [puppet] - 10https://gerrit.wikimedia.org/r/1326957 (https://phabricator.wikimedia.org/T350516) [00:40:51] (03PS5) 10Cwhite: centralserver: add utility to delete oldest files based on disk usage [puppet] - 10https://gerrit.wikimedia.org/r/1321669 [00:41:13] (03PS6) 10Cwhite: centralserver: add utility to delete oldest files based on disk usage [puppet] - 10https://gerrit.wikimedia.org/r/1321669 (https://phabricator.wikimedia.org/T434690) [00:45:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:01:21] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 44%, RTA = 4848.56 ms [01:03:21] PROBLEM - PyBal IPVS diff check on lvs1019 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [01:04:19] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 0.15 ms [01:04:48] 10SRE-swift-storage, 06Commons, 06DBA, 10media-backups, and 5 others: old file revisions missing of File:A_Warm_Shade_of_Ivory_-_Henry_Mancini_album_cover.jpg - https://phabricator.wikimedia.org/T428406#12229289 (10Asukite) The following images also appear to be missing, is this possibly related? https://... [01:08:21] RECOVERY - PyBal IPVS diff check on lvs1019 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [01:11:21] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1326962 [01:11:21] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1326962 (owner: 10TrainBranchBot) [01:21:01] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1326962 (owner: 10TrainBranchBot) [02:00:42] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:02:40] 10SRE-swift-storage, 06Commons, 06DBA, 10media-backups, and 5 others: old file revisions missing of File:A_Warm_Shade_of_Ivory_-_Henry_Mancini_album_cover.jpg - https://phabricator.wikimedia.org/T428406#12229312 (10RLazarus) (Drive-by comment here, happened to see @Asukite's message in passing.) @Asukite... [02:07:31] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 48s) [02:08:13] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:09] PROBLEM - Host db2207 #page is DOWN: PING CRITICAL - Packet loss = 100% [02:13:13] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:55] PROBLEM - MariaDB Replica IO: s2 on db2197 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2207.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2207.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:14:56] PROBLEM - MariaDB Replica IO: s2 #page on db2175 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2207.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2207.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:14:57] PROBLEM - MariaDB Replica IO: s2 #page on db2226 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2207.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2207.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:14:58] PROBLEM - MariaDB Replica IO: s2 #page on db2189 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2207.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2207.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:14:59] PROBLEM - MariaDB Replica IO: s2 #page on db2204 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2207.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2207.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:15:00] PROBLEM - MariaDB Replica IO: s2 #page on db2238 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2207.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2207.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:15:00] PROBLEM - MariaDB Replica IO: s2 #page on db2225 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2207.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2207.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:15:43] RECOVERY - Host db2207 #page is UP: PING OK - Packet loss = 0%, RTA = 31.68 ms [02:15:57] PROBLEM - mysqld processes on db2207 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [02:15:59] PROBLEM - MariaDB Replica SQL: s2 #page on db2207 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:16:00] PROBLEM - MariaDB Replica IO: s2 #page on db2207 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:16:01] PROBLEM - MariaDB Events s2 on db2207 is CRITICAL: CRITICAL - Failed to query events: ERROR 2002 (HY000): Cant connect to local server through socket /run/mysqld/mysqld.sock (2) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [02:16:01] PROBLEM - pt-heartbeat-wikimedia process on db2207 is CRITICAL: PROCS CRITICAL: 0 processes with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [02:16:12] !incidients [02:16:16] !incidents [02:16:16] 8256 (ACKED) db2175 (paged)/MariaDB Replica IO: s2 (paged) [02:16:16] 8257 (ACKED) db2226 (paged)/MariaDB Replica IO: s2 (paged) [02:16:17] 8258 (ACKED) db2189 (paged)/MariaDB Replica IO: s2 (paged) [02:16:17] 8259 (ACKED) db2204 (paged)/MariaDB Replica IO: s2 (paged) [02:16:17] 8260 (ACKED) db2238 (paged)/MariaDB Replica IO: s2 (paged) [02:16:17] 8261 (ACKED) db2225 (paged)/MariaDB Replica IO: s2 (paged) [02:16:17] 8262 (UNACKED) db2207 (paged)/MariaDB Replica SQL: s2 (paged) [02:16:18] 8263 (UNACKED) db2207 (paged)/MariaDB Replica IO: s2 (paged) [02:16:18] 8255 (RESOLVED) Host db2207 (paged) [02:16:19] 8253 (RESOLVED) Host cr1-magru [02:16:19] 8251 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr2-eqsin:9804 Peering: BBIX (01130-SG1-X-I-0001) xe-0/1/6 gnmi eqsin) [02:16:27] !ack 8262 [02:16:27] 8262 (ACKED) db2207 (paged)/MariaDB Replica SQL: s2 (paged) [02:16:33] !ack 8263 [02:16:33] 8263 (ACKED) db2207 (paged)/MariaDB Replica IO: s2 (paged) [02:16:41] PROBLEM - MariaDB Event Scheduler s2 on db2207 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [02:16:42] PROBLEM - MariaDB read only s2 #page on db2207 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [02:17:39] !incidents [02:17:39] 8256 (ACKED) db2175 (paged)/MariaDB Replica IO: s2 (paged) [02:17:40] 8257 (ACKED) db2226 (paged)/MariaDB Replica IO: s2 (paged) [02:17:40] 8258 (ACKED) db2189 (paged)/MariaDB Replica IO: s2 (paged) [02:17:40] 8259 (ACKED) db2204 (paged)/MariaDB Replica IO: s2 (paged) [02:17:40] 8260 (ACKED) db2238 (paged)/MariaDB Replica IO: s2 (paged) [02:17:41] 8261 (ACKED) db2225 (paged)/MariaDB Replica IO: s2 (paged) [02:17:41] 8262 (ACKED) db2207 (paged)/MariaDB Replica SQL: s2 (paged) [02:17:41] 8263 (ACKED) db2207 (paged)/MariaDB Replica IO: s2 (paged) [02:17:41] 8264 (ACKED) db2207 (paged)/MariaDB read only s2 (paged) [02:17:42] 8255 (RESOLVED) Host db2207 (paged) [02:17:42] 8253 (RESOLVED) Host cr1-magru [02:17:43] 8251 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr2-eqsin:9804 Peering: BBIX (01130-SG1-X-I-0001) xe-0/1/6 gnmi eqsin) [02:18:05] !log denisse@cumin1003 START - Cookbook sre.mysql.depool depool db2207: Depooling replica [02:18:06] !log denisse@cumin1003 END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2207: Depooling replica [02:18:13] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:19:57] PROBLEM - MariaDB Replica Lag: s2 #page on db2175 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 601.54 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:19:58] PROBLEM - MariaDB Replica Lag: s2 #page on db2189 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 603.40 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:19:59] PROBLEM - MariaDB Replica Lag: s2 #page on db2226 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 605.07 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:20:00] PROBLEM - MariaDB Replica Lag: s2 #page on db2204 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 605.37 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:20:01] PROBLEM - MariaDB Replica Lag: s2 #page on db2225 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 605.41 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:20:02] PROBLEM - MariaDB Replica Lag: s2 #page on db2238 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 607.10 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:20:09] !ack [02:20:10] 8265 (ACKED) db2189 (paged)/MariaDB Replica Lag: s2 (paged) [02:20:10] 8266 (ACKED) db2175 (paged)/MariaDB Replica Lag: s2 (paged) [02:20:10] 8267 (ACKED) db2226 (paged)/MariaDB Replica Lag: s2 (paged) [02:20:11] 8268 (ACKED) db2204 (paged)/MariaDB Replica Lag: s2 (paged) [02:20:11] 8269 (ACKED) db2225 (paged)/MariaDB Replica Lag: s2 (paged) [02:20:11] 8270 (ACKED) db2238 (paged)/MariaDB Replica Lag: s2 (paged) [02:22:59] PROBLEM - MariaDB Replica Lag: s2 #page on db2207 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:23:08] !ack [02:23:09] 8271 (ACKED) db2207 (paged)/MariaDB Replica Lag: s2 (paged) [02:29:49] !log ryankemper@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1186.eqiad.wmnet with OS bookworm [02:32:09] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host an-worker1186.eqiad.wmnet with OS bookworm [02:33:41] RECOVERY - MariaDB Event Scheduler s2 on db2207 is OK: Version 10.11.16-MariaDB-log, Uptime 41s, read_only: True, event_scheduler: True, 2379.86 QPS, connection latency: 0.026346s, query latency: 0.001292s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [02:33:41] RECOVERY - MariaDB read only s2 #page on db2207 is OK: Version 10.11.16-MariaDB-log, Uptime 41s, read_only: True, event_scheduler: True, 2293.69 QPS, connection latency: 0.026375s, query latency: 0.001242s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [02:33:55] RECOVERY - mysqld processes on db2207 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [02:33:59] RECOVERY - MariaDB Events s2 on db2207 is OK: OK - All 2 events in ops database are ENABLED https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [02:33:59] RECOVERY - MariaDB Replica SQL: s2 #page on db2207 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:34:00] RECOVERY - MariaDB Replica IO: s2 #page on db2207 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:34:55] RECOVERY - MariaDB Replica IO: s2 on db2197 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:34:57] RECOVERY - MariaDB Replica IO: s2 #page on db2175 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:34:59] RECOVERY - MariaDB Replica IO: s2 #page on db2226 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:35:00] RECOVERY - MariaDB Replica IO: s2 #page on db2189 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:35:01] RECOVERY - MariaDB Replica IO: s2 #page on db2204 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:35:01] RECOVERY - MariaDB Replica Lag: s2 #page on db2207 is OK: OK slave_sql_lag Replication lag: 0.16 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:35:02] RECOVERY - MariaDB Replica IO: s2 #page on db2225 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:35:02] RECOVERY - MariaDB Replica IO: s2 #page on db2238 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:36:27] (03PS1) 10Gerrit maintenance bot: mariadb: Promote db2204 to s2 master [puppet] - 10https://gerrit.wikimedia.org/r/1326967 (https://phabricator.wikimedia.org/T435270) [02:36:56] RECOVERY - MariaDB Replica Lag: s2 #page on db2175 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:36:58] RECOVERY - MariaDB Replica Lag: s2 #page on db2189 is OK: OK slave_sql_lag Replication lag: 0.13 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:37:00] RECOVERY - pt-heartbeat-wikimedia process on db2207 is OK: PROCS OK: 1 process with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [02:37:00] RECOVERY - MariaDB Replica Lag: s2 #page on db2225 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:37:01] RECOVERY - MariaDB Replica Lag: s2 #page on db2204 is OK: OK slave_sql_lag Replication lag: 0.07 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:37:01] RECOVERY - MariaDB Replica Lag: s2 #page on db2226 is OK: OK slave_sql_lag Replication lag: 0.12 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:37:04] RECOVERY - MariaDB Replica Lag: s2 #page on db2238 is OK: OK slave_sql_lag Replication lag: 0.03 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:37:45] FIRING: CirrusConsumerFetchErrorRate: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s): fetch error rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerFetchErrorRate [02:39:40] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s2 T435270 [02:39:45] T435270: Switchover s2 master (db2207 -> db2204) - https://phabricator.wikimedia.org/T435270 [02:39:52] !log marostegui@cumin1003 dbctl commit (dc=all): 'Set db2204 with weight 0 T435270', diff saved to https://phabricator.wikimedia.org/P96179 and previous config saved to /var/cache/conftool/dbconfig/20260819-023951-marostegui.json [02:42:45] RESOLVED: CirrusConsumerFetchErrorRate: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s): fetch error rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerFetchErrorRate [02:43:15] (03CR) 10Marostegui: [C:03+2] mariadb: Promote db2204 to s2 master [puppet] - 10https://gerrit.wikimedia.org/r/1326967 (https://phabricator.wikimedia.org/T435270) (owner: 10Gerrit maintenance bot) [02:43:39] !log Starting s2 codfw failover from db2207 to db2204 - T435270 [02:43:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:44:05] !log marostegui@cumin1003 dbctl commit (dc=all): 'Promote db2204 to s2 primary T435270', diff saved to https://phabricator.wikimedia.org/P96180 and previous config saved to /var/cache/conftool/dbconfig/20260819-024403-marostegui.json [02:46:28] !log marostegui@cumin1003 dbctl commit (dc=all): 'Depool db2207 T435270', diff saved to https://phabricator.wikimedia.org/P96181 and previous config saved to /var/cache/conftool/dbconfig/20260819-024627-marostegui.json [02:46:34] T435270: Switchover s2 master (db2207 -> db2204) - https://phabricator.wikimedia.org/T435270 [02:46:35] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1186.eqiad.wmnet with reason: host reimage [02:48:36] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 2 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [02:54:31] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1186.eqiad.wmnet with reason: host reimage [03:16:40] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1186.eqiad.wmnet with OS bookworm [03:18:37] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:51:26] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [04:11:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [04:13:09] (03CR) 10Cwhite: [C:03+1] team-sre: add o11y team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319828 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [04:13:46] (03CR) 10Cwhite: [C:03+1] team-sre: make CertAlmostExpired team-aware [alerts] - 10https://gerrit.wikimedia.org/r/1311834 (https://phabricator.wikimedia.org/T414662) (owner: 10Hnowlan) [04:28:37] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:33:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 19 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325532 (https://phabricator.wikimedia.org/T434748) (owner: 10Danielyepezgarces) [04:45:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:08:32] (03CR) 10Giuseppe Lavagetto: [C:03+1] "Overall LGTM; only one optional comment on the login method composition." [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1325982 (https://phabricator.wikimedia.org/T434957) (owner: 10Dduvall) [05:25:09] (03CR) 10Giuseppe Lavagetto: [C:04-1] "Overall I like the split; I think that the more formal structure for the factory method overcomplicates things a bit though." [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1326398 (https://phabricator.wikimedia.org/T434957) (owner: 10Dduvall) [05:41:29] (03PS1) 10Marostegui: mariadb: Replace s2 codfw candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1326979 (https://phabricator.wikimedia.org/T435279) [05:44:50] (03CR) 10Marostegui: [C:03+2] mariadb: Replace s2 codfw candidate master [puppet] - 10https://gerrit.wikimedia.org/r/1326979 (https://phabricator.wikimedia.org/T435279) (owner: 10Marostegui) [05:47:41] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [05:51:03] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db2209: db2209 repool [05:54:16] (03PS1) 10Marostegui: db1284: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326981 [05:55:11] (03CR) 10Marostegui: [C:03+2] db1284: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326981 (owner: 10Marostegui) [05:55:31] 06SRE, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): sre.hosts.move-vlan crashes during init for hosts absent from PuppetDB - https://phabricator.wikimedia.org/T434737#12229570 (10RKemper) 05Open→03In progress [05:55:38] (03CR) 10Elukey: [C:03+1] Move the insetup role report to cumin2003 [puppet] - 10https://gerrit.wikimedia.org/r/1326828 (owner: 10Muehlenhoff) [05:55:48] 06SRE, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): sre.hosts.move-vlan crashes during init for hosts absent from PuppetDB - https://phabricator.wikimedia.org/T434737#12229573 (10RKemper) 05In progress→03Resolved [05:55:56] (03CR) 10Elukey: [C:03+1] Remove cluster management role from cumin2002 [puppet] - 10https://gerrit.wikimedia.org/r/1326829 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [05:56:40] (03PS1) 10Marostegui: instances.yaml: Add db1284 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1326982 (https://phabricator.wikimedia.org/T407942) [05:57:42] (03PS2) 10Marostegui: instances.yaml: Add db1284 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1326982 (https://phabricator.wikimedia.org/T407942) [05:59:14] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2048.codfw.wmnet [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T0600) [06:00:22] (03CR) 10Muehlenhoff: [C:03+2] Remove cumin2002 from the homer peer list of cumin1003 [puppet] - 10https://gerrit.wikimedia.org/r/1326847 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [06:00:40] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1284 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1326982 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [06:03:23] jouncebot: nowandnext [06:03:23] For the next 0 hour(s) and 56 minute(s): MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T0600) [06:03:24] In 0 hour(s) and 56 minute(s): UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T0700) [06:03:54] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2048.codfw.wmnet [06:06:11] PROBLEM - Host dse-k8s-etcd2002 is DOWN: PING CRITICAL - Packet loss = 100% [06:06:24] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1284 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96184 and previous config saved to /var/cache/conftool/dbconfig/20260819-060621-marostegui.json [06:06:30] T407942: Productionize db12[65-90] - https://phabricator.wikimedia.org/T407942 [06:06:46] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1284: Pool back [06:08:42] (03PS1) 10Marostegui: db1280: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326983 (https://phabricator.wikimedia.org/T407942) [06:09:27] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2048.codfw.wmnet [06:10:07] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2048.codfw.wmnet [06:10:07] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host zuul1002.eqiad.wmnet [06:10:33] (03CR) 10Marostegui: [C:03+2] db1280: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326983 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [06:10:39] RECOVERY - Host dse-k8s-etcd2002 is UP: PING OK - Packet loss = 0%, RTA = 30.79 ms [06:12:31] (03PS1) 10Marostegui: instances.yaml: Add db1280 [puppet] - 10https://gerrit.wikimedia.org/r/1326984 (https://phabricator.wikimedia.org/T407942) [06:14:14] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1002.eqiad.wmnet [06:14:26] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host zuul1003.eqiad.wmnet [06:15:44] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1280 [puppet] - 10https://gerrit.wikimedia.org/r/1326984 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [06:17:44] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1284 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96186 and previous config saved to /var/cache/conftool/dbconfig/20260819-061743-marostegui.json [06:17:50] T407942: Productionize db12[65-90] - https://phabricator.wikimedia.org/T407942 [06:18:12] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1280: Pool back [06:18:16] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1003.eqiad.wmnet [06:18:28] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:18:41] !log arnaudb@cumin1003 START - Cookbook sre.hosts.reboot-single for host zuul1001.eqiad.wmnet [06:19:05] (03PS1) 10Marostegui: db1277: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326985 (https://phabricator.wikimedia.org/T407942) [06:20:26] (03CR) 10Marostegui: [C:03+2] db1277: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1326985 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [06:22:49] !log arnaudb@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1001.eqiad.wmnet [06:23:07] (03PS1) 10Slyngshede: [WIP] Start of thumb.wikimedia.org in text [puppet] - 10https://gerrit.wikimedia.org/r/1298820 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [06:23:45] (03PS2) 10Slyngshede: Start of thumb.wikimedia.org in text [puppet] - 10https://gerrit.wikimedia.org/r/1298820 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [06:24:58] (03PS1) 10Marostegui: instances.yaml: Add db1277 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1326987 (https://phabricator.wikimedia.org/T407942) [06:26:19] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1277 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1326987 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [06:26:50] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org [06:27:26] (03PS1) 10Marostegui: db1277: Add note [puppet] - 10https://gerrit.wikimedia.org/r/1326988 [06:28:16] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add db1277 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96190 and previous config saved to /var/cache/conftool/dbconfig/20260819-062815-marostegui.json [06:28:21] T407942: Productionize db12[65-90] - https://phabricator.wikimedia.org/T407942 [06:29:14] (03CR) 10Marostegui: [C:03+2] db1277: Add note [puppet] - 10https://gerrit.wikimedia.org/r/1326988 (owner: 10Marostegui) [06:29:47] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1277: Pool back [06:32:58] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org [06:33:45] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet [06:34:07] 06SRE, 06Content-Platform-Team, 10MediaWiki-extensions-PageAssessments: pageassessments-cleanup alerts tagged with defunct team - https://phabricator.wikimedia.org/T434959#12229615 (10JMeybohm) 05Open→03Resolved a:03JMeybohm [06:34:23] (03PS1) 10Slyngshede: Update to CAS 7.3.8.1 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1326990 [06:36:14] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: db2209 repool [06:36:50] (03CR) 10JMeybohm: [C:03+2] maintenance: Change owner of purge_loginnotify to PSI [puppet] - 10https://gerrit.wikimedia.org/r/1326937 (owner: 10RLazarus) [06:37:34] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet [06:38:03] 06SRE, 10SRE-Access-Requests: Requesting access to 'restricted' for nicholusmuwonge - https://phabricator.wikimedia.org/T435168#12229618 (10JMeybohm) 05Stalled→03Open [06:39:43] (03PS1) 10JMeybohm: Add nicholusmuwonge to restricted group [puppet] - 10https://gerrit.wikimedia.org/r/1326991 (https://phabricator.wikimedia.org/T435168) [06:41:04] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12229622 (10JMeybohm) 05Stalled→03Open [06:41:10] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet [06:41:44] (03CR) 10JMeybohm: [C:03+2] Add nicholusmuwonge to restricted group [puppet] - 10https://gerrit.wikimedia.org/r/1326991 (https://phabricator.wikimedia.org/T435168) (owner: 10JMeybohm) [06:41:51] (03CR) 10Muehlenhoff: [C:03+2] Remove the acmechief config for mirrors.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1307378 (https://phabricator.wikimedia.org/T416707) (owner: 10Muehlenhoff) [06:43:15] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to 'restricted' for nicholusmuwonge - https://phabricator.wikimedia.org/T435168#12229641 (10JMeybohm) 05Open→03Resolved a:03JMeybohm Group membership has been added and will populate within the next 30min across the infra. Please f... [06:44:38] (03PS1) 10Marostegui: mariadb: Productionize db1276 [puppet] - 10https://gerrit.wikimedia.org/r/1326992 (https://phabricator.wikimedia.org/T407942) [06:45:12] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet [06:45:40] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1276 [puppet] - 10https://gerrit.wikimedia.org/r/1326992 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [06:49:16] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1222.eqiad.wmnet onto db1276.eqiad.wmnet [06:49:20] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1222: Depool db1222.eqiad.wmnet to then clone it to db1276.eqiad.wmnet - marostegui@cumin1003 [06:49:35] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Depool db1222.eqiad.wmnet to then clone it to db1276.eqiad.wmnet - marostegui@cumin1003 [06:51:55] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1284: Pool back [06:52:56] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12229671 (10JMeybohm) Hi @Bethany and thanks for the clarification. Your account is member of the `wmf` LDAP as well as the `analytics-privatedata-users` shell group a... [06:54:21] !log installing php8.2 security updates [06:54:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:55:38] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org [06:58:24] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for eamedina - https://phabricator.wikimedia.org/T435200#12229686 (10JMeybohm) [06:59:47] (03PS1) 10JMeybohm: Add eamedina to analytics-privatedata-users (level 1) [puppet] - 10https://gerrit.wikimedia.org/r/1326996 (https://phabricator.wikimedia.org/T435200) [07:00:04] Amir1, urbanecm, and awight: Time to do the UTC morning backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T0700). [07:00:04] No Gerrit patches in the queue for this window AFAICS. [07:00:41] (03CR) 10CI reject: [V:04-1] Add eamedina to analytics-privatedata-users (level 1) [puppet] - 10https://gerrit.wikimedia.org/r/1326996 (https://phabricator.wikimedia.org/T435200) (owner: 10JMeybohm) [07:01:20] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org [07:02:21] (03CR) 10Slyngshede: [V:03+2 C:03+2] Update to CAS 7.3.8.1 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1326990 (owner: 10Slyngshede) [07:02:55] (03PS2) 10JMeybohm: Add eamedina to analytics-privatedata-users (level 1) [puppet] - 10https://gerrit.wikimedia.org/r/1326996 (https://phabricator.wikimedia.org/T435200) [07:03:13] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:03:21] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1280: Pool back [07:04:25] (03CR) 10JMeybohm: [C:03+2] Add eamedina to analytics-privatedata-users (level 1) [puppet] - 10https://gerrit.wikimedia.org/r/1326996 (https://phabricator.wikimedia.org/T435200) (owner: 10JMeybohm) [07:05:28] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [07:07:44] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org [07:08:13] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:10:28] RESOLVED: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [07:13:04] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for eamedina - https://phabricator.wikimedia.org/T435200#12229717 (10JMeybohm) 05Open→03Resolved a:03JMeybohm `analytics-privatedata-users` level 1 access has been granted and will populate across the... [07:13:24] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9272/co" [puppet] - 10https://gerrit.wikimedia.org/r/1298820 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [07:14:43] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org [07:14:55] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1277: Pool back [07:15:03] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9273/co" [puppet] - 10https://gerrit.wikimedia.org/r/1298820 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [07:16:58] FIRING: [2x] KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [07:17:06] (03PS3) 10Slyngshede: P:trafficserver::backend map thumb to swift backend [puppet] - 10https://gerrit.wikimedia.org/r/1298820 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [07:17:13] RESOLVED: [2x] KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [07:17:18] !log installing imagemagick security updates [07:17:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:18:13] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:18:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:24:48] 06SRE, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 3 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12229735 (10RKemper) [07:26:17] 06SRE, 10hCaptcha, 06Product Safety and Integrity, 10ServiceOps-Mediawiki, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12229743 (10MLechvien-WMF) [07:26:30] 06SRE, 10hCaptcha, 06Product Safety and Integrity, 10ServiceOps-Mediawiki, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12229745 (10MLechvien-WMF) p:05Triage→03Medium [07:28:16] (03PS3) 10Hnowlan: team-sre: add o11y team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319828 (https://phabricator.wikimedia.org/T432376) [07:33:00] (03CR) 10Hnowlan: [C:03+2] team-sre: add o11y team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319828 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [07:35:21] !log mvernon@cumin2003 START - Cookbook sre.swift.convert-disks for host ms-be1064 [07:35:28] (03Merged) 10jenkins-bot: team-sre: add o11y team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319828 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [07:35:46] (03PS3) 10Hnowlan: team-sre: use traffic team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319829 (https://phabricator.wikimedia.org/T432376) [07:36:56] !log mvernon@cumin2003 START - Cookbook sre.swift.convert-disks for host ms-be1068 [07:39:04] (03PS1) 10Gkyziridis: ml-services: Deploy latest changes in liftwing-openapi-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327001 [07:41:20] (03CR) 10Hnowlan: [C:03+2] team-sre: use traffic team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319829 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [07:42:54] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1222: Pool db1222.eqiad.wmnet in after cloning [07:43:16] (03Merged) 10jenkins-bot: team-sre: use traffic team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319829 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [07:45:41] !log extend the disk of ldap-rw2001 by 80G T331699 [07:45:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:45:46] T331699: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699 [07:48:03] (03Abandoned) 10Arnaudb: Revert^2 "gitlab: lower TTL for CNAMEs" [dns] - 10https://gerrit.wikimedia.org/r/1310135 (owner: 10Dzahn) [07:48:05] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1055.eqiad.wmnet with OS trixie [07:49:06] (03CR) 10Arnaudb: [C:03+2] gitlab: discard firewall throttling on the primary [puppet] - 10https://gerrit.wikimedia.org/r/1325844 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [07:50:21] (03CR) 10MSantos: [C:03+1] Hand PageAssesment ownership to Content-Platform-Team [puppet] - 10https://gerrit.wikimedia.org/r/1326808 (https://phabricator.wikimedia.org/T434959) (owner: 10JMeybohm) [07:51:26] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [07:53:13] RESOLVED: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:58:47] !log arnaudb@dns1006 START - running authdns-update [07:59:26] I ^C it, will submit the change first: ) [07:59:35] (03CR) 10Arnaudb: [C:03+2] gitlab: point gitlab.wikimedia.org at the CDN [dns] - 10https://gerrit.wikimedia.org/r/1324281 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [07:59:49] !log arnaudb@dns1006 START - running authdns-update [08:00:04] andre and brennen: Time to snap out of that daydream and deploy MediaWiki train - Utc-0+Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T0800). [08:00:06] o/ [08:01:37] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1172.eqiad.wmnet with OS bookworm [08:01:41] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1173.eqiad.wmnet with OS bookworm [08:01:45] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1174.eqiad.wmnet with OS bookworm [08:01:56] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1207.eqiad.wmnet with OS bookworm [08:01:58] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1208.eqiad.wmnet with OS bookworm [08:02:01] !log arnaudb@dns1006 END - running authdns-update [08:03:39] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1055.eqiad.wmnet with reason: host reimage [08:04:10] (03PS1) 10TrainBranchBot: group1 to 1.47.0-wmf.16 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327003 (https://phabricator.wikimedia.org/T430835) [08:04:14] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by aklapper@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327003 (https://phabricator.wikimedia.org/T430835) (owner: 10TrainBranchBot) [08:07:20] !log cmooney@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pfw1-codfw with reason: upgrade pfw1a-codfw and pfw1b-codfw pair [08:07:38] PROBLEM - Gitlab SSH healthcheck git daemon on gitlab.wikimedia.org is CRITICAL: connect to address gitlab.wikimedia.org and port 22: Connection refused https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [08:07:41] (03Merged) 10jenkins-bot: group1 to 1.47.0-wmf.16 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327003 (https://phabricator.wikimedia.org/T430835) (owner: 10TrainBranchBot) [08:09:31] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1055.eqiad.wmnet with reason: host reimage [08:09:55] 10SRE-swift-storage, 06Commons, 06DBA, 10media-backups, and 5 others: old file revisions missing of File:A_Warm_Shade_of_Ivory_-_Henry_Mancini_album_cover.jpg - https://phabricator.wikimedia.org/T428406#12229811 (10KylieTastic) >>! In T428406#12228702, @KylieTastic wrote: > I reverted the full deletion of... [08:10:31] !log cmooney@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr[1-2]-codfw with reason: upgrade pfw1a-codfw and pfw1b-codfw pair [08:11:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [08:13:00] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2902.codfw.wmnet with OS trixie [08:13:01] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host db2902.codfw.wmnet [08:13:26] (03CR) 10Dpogorzelski: [C:03+1] Update to 1.29.4 [debs/istio] - 10https://gerrit.wikimedia.org/r/1321949 (https://phabricator.wikimedia.org/T434198) (owner: 10JMeybohm) [08:14:04] !log aklapper@deploy1003 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.16 refs T430835 [08:14:10] T430835: 1.47.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T430835 [08:14:28] (03CR) 10Fabfur: [C:03+1] P:trafficserver::backend map thumb to swift backend [puppet] - 10https://gerrit.wikimedia.org/r/1298820 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [08:14:39] !log fceratto@cumin1003 START - Cookbook sre.hosts.reimage for host db2902.codfw.wmnet with OS trixie [08:15:15] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1207.eqiad.wmnet with reason: host reimage [08:15:17] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1208.eqiad.wmnet with reason: host reimage [08:15:25] (03PS1) 10Dpogorzelski: kserve: update images to 0.20 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1327059 (https://phabricator.wikimedia.org/T433973) [08:15:40] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet [08:15:49] (03CR) 10Hnowlan: "lgtm mostly, one change and a few nits. I assume this has been tested in dry run already? :)" [puppet] - 10https://gerrit.wikimedia.org/r/1321669 (https://phabricator.wikimedia.org/T434690) (owner: 10Cwhite) [08:17:02] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1172.eqiad.wmnet with reason: host reimage [08:17:12] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1173.eqiad.wmnet with reason: host reimage [08:17:39] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1174.eqiad.wmnet with reason: host reimage [08:18:48] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1207.eqiad.wmnet with reason: host reimage [08:20:24] !log mvernon@cumin2003 START - Cookbook sre.swift.convert-disks for host ms-be1065 [08:20:45] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be1064.eqiad.wmnet with OS trixie [08:22:21] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1172.eqiad.wmnet with reason: host reimage [08:23:08] (03CR) 10Slyngshede: [C:03+2] P:trafficserver::backend map thumb to swift backend [puppet] - 10https://gerrit.wikimedia.org/r/1298820 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [08:23:47] !log reboot pfw1-codfw firewall pair to upgrade JunOS T434865 [08:23:50] (03PS1) 10Tiziano Fogli: sloth: promote severity to critical [alerts] - 10https://gerrit.wikimedia.org/r/1327060 [08:23:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:23:51] T434865: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865 [08:24:43] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet [08:25:04] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003" [08:25:08] mvernon@cumin2003 convert-disks (PID 3000756) is awaiting input [08:26:24] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1174.eqiad.wmnet with reason: host reimage [08:26:27] (03CR) 10Tiziano Fogli: [C:03+2] sloth: promote severity to critical [alerts] - 10https://gerrit.wikimedia.org/r/1327060 (owner: 10Tiziano Fogli) [08:27:40] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003" [08:27:40] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1055.eqiad.wmnet with OS trixie [08:28:04] !log fceratto@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db2902.codfw.wmnet with reason: host reimage [08:28:07] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Pool db1222.eqiad.wmnet in after cloning [08:28:11] FIRING: PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-eqiad and (null) (10.195.0.249) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=pfw1-eqiad:9804&var-bgp_group=VPN&var-bgp_neighbor=(null) - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [08:28:17] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1222.eqiad.wmnet onto db1276.eqiad.wmnet [08:28:37] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:28:51] FIRING: [4x] SwitchCoreInterfaceDown: Switch core interface down - fasw1-f5a-codfw:et-0/0/47 (Core: pfw1-codfw:et-0/1/0) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [08:29:34] !log filippo@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [08:31:44] !log filippo@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [08:34:08] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1208.eqiad.wmnet with reason: host reimage [08:34:55] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be1068.eqiad.wmnet with OS trixie [08:36:48] (03CR) 10Blake: "I'm not sure that I've understood this correctly - the change to helmfile.yaml should be, in the mw-script namespace, to change the value " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325884 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [08:37:58] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1064.eqiad.wmnet with reason: host reimage [08:38:11] RESOLVED: PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-eqiad and (null) (10.195.0.249) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=pfw1-eqiad:9804&var-bgp_group=VPN&var-bgp_neighbor=(null) - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [08:38:16] !log fceratto@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2902.codfw.wmnet with reason: host reimage [08:38:50] (03CR) 10Blake: [C:03+2] envoy: New upstream version 1.39.0. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1326818 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [08:38:51] RESOLVED: [4x] SwitchCoreInterfaceDown: Switch core interface down - fasw1-f5a-codfw:et-0/0/47 (Core: pfw1-codfw:et-0/1/0) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [08:38:59] (03CR) 10Blake: [V:03+2 C:03+2] envoy: New upstream version 1.39.0. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1326818 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [08:39:13] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1207.eqiad.wmnet with OS bookworm [08:39:31] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [08:40:46] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1057.eqiad.wmnet with OS trixie [08:41:48] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1064.eqiad.wmnet with reason: host reimage [08:45:34] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1173.eqiad.wmnet with reason: host reimage [08:45:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:46:02] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1172.eqiad.wmnet with OS bookworm [08:46:25] filippo@cumin1003 provision (PID 1230398) is awaiting input [08:50:31] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1174.eqiad.wmnet with OS bookworm [08:51:43] 06SRE, 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12229923 (10cmooney) >>! In T434865#12228113, @cmooney wrote: > As discussed on irc I'll kick off the upgrade on pfw1-codfw at 09:00am UTC tomrorow (... [08:52:12] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1068.eqiad.wmnet with reason: host reimage [08:53:10] !log fceratto@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2902.codfw.wmnet with OS trixie [08:54:13] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1208.eqiad.wmnet with OS bookworm [08:56:05] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1057.eqiad.wmnet with reason: host reimage [08:58:44] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1068.eqiad.wmnet with reason: host reimage [09:01:28] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [09:02:28] 06SRE, 10SRE-Access-Requests, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Requesting access to Analytics Data Lake for mkrolik/mkrolik-wmf - https://phabricator.wikimedia.org/T434877#12229945 (10Gehel) [09:02:50] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1057.eqiad.wmnet with reason: host reimage [09:03:04] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1056.eqiad.wmnet with OS trixie [09:03:52] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1064.eqiad.wmnet with OS trixie [09:03:53] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.convert-disks (exit_code=0) for host ms-be1064 [09:06:39] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host ldap-rw2001.wikimedia.org with OS trixie [09:06:50] 06SRE, 06Infrastructure-Foundations, 07LDAP: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12229964 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host ldap-rw2001.wikimedia.org with OS trixie [09:07:22] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1173.eqiad.wmnet with OS bookworm [09:07:24] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [09:07:50] !log fceratto@cumin1003 END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) [09:09:36] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org [09:09:46] !log installing Postgresql security updates [09:09:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:09:53] (03PS1) 10Slyngshede: Revert "P:trafficserver::backend map thumb to swift backend" [puppet] - 10https://gerrit.wikimedia.org/r/1327064 [09:10:07] mvernon@cumin2003 convert-disks (PID 3007881) is awaiting input [09:10:55] (03CR) 10Slyngshede: [C:03+2] Revert "P:trafficserver::backend map thumb to swift backend" [puppet] - 10https://gerrit.wikimedia.org/r/1327064 (owner: 10Slyngshede) [09:12:51] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [09:13:16] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be1065.eqiad.wmnet with OS trixie [09:13:24] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [09:13:25] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org [09:14:52] !log atsuko@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [09:15:31] (03PS9) 10Dreamy Jazz: Register the mediawiki.wikimedia_antiabuse.content_policy_score stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1321579 (https://phabricator.wikimedia.org/T432848) (owner: 10Mpostoronca) [09:15:48] !log atsuko@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [09:17:58] (03PS1) 10Slyngshede: Revert^2 "P:trafficserver::backend map thumb to swift backend" [puppet] - 10https://gerrit.wikimedia.org/r/1327065 [09:18:23] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003" [09:18:26] !log filippo@cumin1003 END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003" [09:18:27] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1057.eqiad.wmnet with OS trixie [09:18:36] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1056.eqiad.wmnet with reason: host reimage [09:19:24] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "cloudvirt1057 - filippo@cumin1003" [09:19:38] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "cloudvirt1057 - filippo@cumin1003" [09:20:03] !log filippo@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [09:20:40] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1068.eqiad.wmnet with OS trixie [09:20:41] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.convert-disks (exit_code=0) for host ms-be1068 [09:20:52] (03PS2) 10Slyngshede: P:trafficserver::backend map thumb to swift backend [puppet] - 10https://gerrit.wikimedia.org/r/1327065 (https://phabricator.wikimedia.org/T427465) [09:23:51] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet [09:24:00] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1056.eqiad.wmnet with reason: host reimage [09:24:33] (03PS1) 10Arnaudb: Revert "gitlab: point gitlab.wikimedia.org at the CDN" [dns] - 10https://gerrit.wikimedia.org/r/1327066 [09:25:10] (03PS1) 10Arnaudb: Revert "gitlab: discard firewall throttling on the primary" [puppet] - 10https://gerrit.wikimedia.org/r/1327067 [09:26:17] (03CR) 10Arnaudb: [C:03+2] Revert "gitlab: point gitlab.wikimedia.org at the CDN" [dns] - 10https://gerrit.wikimedia.org/r/1327066 (owner: 10Arnaudb) [09:26:25] !log arnaudb@dns1006 START - running authdns-update [09:26:52] (03CR) 10Arnaudb: [C:03+2] Revert "gitlab: discard firewall throttling on the primary" [puppet] - 10https://gerrit.wikimedia.org/r/1327067 (owner: 10Arnaudb) [09:27:38] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet [09:27:59] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-rw2001.wikimedia.org with reason: host reimage [09:28:34] !log arnaudb@dns1006 END - running authdns-update [09:29:40] RECOVERY - Gitlab SSH healthcheck git daemon on gitlab.wikimedia.org is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [09:30:50] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1065.eqiad.wmnet with reason: host reimage [09:31:11] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet [09:33:19] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-rw2001.wikimedia.org with reason: host reimage [09:34:51] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet [09:35:52] (03CR) 10Fabfur: [C:03+2] P:trafficserver::backend map thumb to swift backend [puppet] - 10https://gerrit.wikimedia.org/r/1327065 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [09:36:21] !log filippo@cumin1003 END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [09:36:43] !log make HE transport circuits from magru live [09:36:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:36:57] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1065.eqiad.wmnet with reason: host reimage [09:37:14] (03PS2) 10Ryan Kemper: kafka: converge topic config from hieradata [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) [09:39:34] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003" [09:40:33] (03CR) 10Ryan Kemper: "This patch works on pontoon. Before this is prod-worthy we need (1) dedicated pki cert and (2) kafka-configurator packaged in apt.wikimedi" [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [09:42:38] filippo@cumin1003 reimage (PID 1265709) is awaiting input [09:42:50] 06SRE, 10MathSearch: ?format=TEXT does no longer give base64 encoded files from gitiles - https://phabricator.wikimedia.org/T435289#12230092 (10Physikerwelt) [09:43:09] jouncebot: nowandnext [09:43:09] For the next 0 hour(s) and 16 minute(s): MediaWiki train - Utc-0+Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T0800) [09:43:10] In 0 hour(s) and 16 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1000) [09:44:03] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003" [09:44:03] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1056.eqiad.wmnet with OS trixie [09:46:10] (03CR) 10JMeybohm: "I would suggest to follow upstream here and move to the separate files. That should make future updates easier - if I'm not missing anythi" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320114 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [09:46:24] !log filippo@cumin1003 START - Cookbook sre.hosts.provision for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [09:50:33] filippo@cumin1003 provision (PID 1301173) is awaiting input [09:51:27] !log jmm@cumin2003 START - Cookbook sre.netbox.restart-reboot rolling reboot on A:netbox [09:51:30] !log jmm@cumin2003 START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors [09:51:34] !log jmm@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors [09:51:56] !log filippo@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [09:52:06] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-rw2001.wikimedia.org with OS trixie [09:52:19] 06SRE, 06Infrastructure-Foundations, 07LDAP: Migrate the r/w LDAP servers to Trixie and MDB storage - https://phabricator.wikimedia.org/T331699#12230104 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host ldap-rw2001.wikimedia.org with OS trixie completed: - ldap-rw2... [09:53:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mpostoronca@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1321579 (https://phabricator.wikimedia.org/T432848) (owner: 10Mpostoronca) [09:53:51] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved Telxius circuit) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [09:53:58] PROBLEM - LDAP -writable server- on ldap-rw2001 is CRITICAL: Could not search/find objectclasses in dc=wikimedia,dc=org https://wikitech.wikimedia.org/wiki/LDAP%23Troubleshooting [09:54:09] ^ ldap-rw is expected, WIP [09:54:18] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12230109 (10fgiunchedi) Ok I re-ran the provision cookbook and after a few retries it worked for cloudvirt1055 / cloudvirt1056 / cloudvirt1057 and I was able to re... [09:54:44] (03Merged) 10jenkins-bot: Register the mediawiki.wikimedia_antiabuse.content_policy_score stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1321579 (https://phabricator.wikimedia.org/T432848) (owner: 10Mpostoronca) [09:55:49] !log jmm@cumin2003 START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors [09:55:50] !log mpostoronca@deploy1003 Started scap sync-world: Backport for [[gerrit:1321579|Register the mediawiki.wikimedia_antiabuse.content_policy_score stream (T432848)]] [09:55:53] !log jmm@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors [09:55:57] T432848: EventLogging: Record confidence score results - https://phabricator.wikimedia.org/T432848 [09:56:44] (03CR) 10Muehlenhoff: [C:03+2] Remove cumin2002 as mariadb root client [puppet] - 10https://gerrit.wikimedia.org/r/1326843 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:57:35] (03PS6) 10JMeybohm: cert-manager: import CRDs for v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320114 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [09:57:35] (03PS3) 10JMeybohm: cert-manager: update chart for v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320120 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [09:58:06] !log mpostoronca@deploy1003 mpostoronca: Backport for [[gerrit:1321579|Register the mediawiki.wikimedia_antiabuse.content_policy_score stream (T432848)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:58:51] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved Telxius circuit) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [09:59:00] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1065.eqiad.wmnet with OS trixie [09:59:01] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.convert-disks (exit_code=0) for host ms-be1065 [09:59:16] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1169.eqiad.wmnet with OS bookworm [09:59:18] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1185.eqiad.wmnet with OS bookworm [09:59:22] (03CR) 10CI reject: [V:04-1] cert-manager: import CRDs for v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320114 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1000) [10:00:17] !log mpostoronca@deploy1003 mpostoronca: Continuing with deployment [10:01:36] (03PS4) 10JMeybohm: cert-manager: update chart for v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320120 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:01:55] (03CR) 10CI reject: [V:04-1] cert-manager: update chart for v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320120 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:02:41] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [10:03:18] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [10:03:21] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved Telxius circuit) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [10:03:36] 06SRE, 10MathSearch: MathSearch can no longer download individual files from gitiles (403) - https://phabricator.wikimedia.org/T435289#12230179 (10Physikerwelt) [10:03:52] !log atsuko@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [10:04:43] !log mpostoronca@deploy1003 Finished scap sync-world: Backport for [[gerrit:1321579|Register the mediawiki.wikimedia_antiabuse.content_policy_score stream (T432848)]] (duration: 08m 53s) [10:04:48] T432848: EventLogging: Record confidence score results - https://phabricator.wikimedia.org/T432848 [10:04:50] !log atsuko@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [10:05:51] btullis@cumin1003 reimage (PID 1313795) is awaiting input [10:06:43] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host stat1008.eqiad.wmnet with OS bookworm [10:08:27] jmm@cumin2003 restart-reboot (PID 3041071) is awaiting input [10:11:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.netbox.restart-reboot (exit_code=0) rolling reboot on A:netbox [10:12:48] (03PS3) 10JMeybohm: Update cert-manager to 1.19.6 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313866 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:13:00] (03PS1) 10MVernon: swift: restore ms-be106{4,5,8} to the rings [puppet] - 10https://gerrit.wikimedia.org/r/1327070 (https://phabricator.wikimedia.org/T308644) [10:13:04] (03CR) 10Mvolz: [C:03+2] citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326228 (owner: 10PipelineBot) [10:13:54] (03PS1) 10Gkyziridis: ml-services: Deploy latest editing-suggestions on experimental staging and prod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327072 (https://phabricator.wikimedia.org/T430030) [10:14:07] (03PS1) 10Federico Ceratto: site.pp, instances.yaml: Set role and dbctl for db2902 [puppet] - 10https://gerrit.wikimedia.org/r/1327071 (https://phabricator.wikimedia.org/T435059) [10:14:24] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1185.eqiad.wmnet with reason: host reimage [10:14:42] (03PS5) 10JMeybohm: cert-manager: update chart for v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320120 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:14:52] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1169.eqiad.wmnet with reason: host reimage [10:15:03] (03Abandoned) 10JMeybohm: cert-manager: import CRDs for v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320114 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:15:32] (03Merged) 10jenkins-bot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326228 (owner: 10PipelineBot) [10:16:49] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1327070 (https://phabricator.wikimedia.org/T308644) (owner: 10MVernon) [10:17:00] (03CR) 10JMeybohm: [C:03+2] Update cert-manager to 1.19.6 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313866 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:17:06] (03CR) 10JMeybohm: [V:03+2 C:03+2] Update cert-manager to 1.19.6 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313866 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:19:36] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1185.eqiad.wmnet with reason: host reimage [10:22:13] (03CR) 10Muehlenhoff: [C:03+2] Remove cluster management role from cumin2002 [puppet] - 10https://gerrit.wikimedia.org/r/1326829 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [10:22:24] (03CR) 10Muehlenhoff: [C:03+2] Move the insetup role report to cumin2003 [puppet] - 10https://gerrit.wikimedia.org/r/1326828 (owner: 10Muehlenhoff) [10:23:10] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1169.eqiad.wmnet with reason: host reimage [10:23:40] (03CR) 10Marostegui: "Can you leave the instances.yaml out so I can do it at my own pace." [puppet] - 10https://gerrit.wikimedia.org/r/1327071 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [10:23:46] (03PS2) 10Muehlenhoff: Remove cluster management role from cumin2002 [puppet] - 10https://gerrit.wikimedia.org/r/1326829 (https://phabricator.wikimedia.org/T427897) [10:24:14] (03PS6) 10JMeybohm: cert-manager: update chart for v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320120 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:26:37] (03CR) 10MVernon: [C:03+2] swift: restore ms-be106{4,5,8} to the rings [puppet] - 10https://gerrit.wikimedia.org/r/1327070 (https://phabricator.wikimedia.org/T308644) (owner: 10MVernon) [10:27:05] (03PS1) 10JMeybohm: wikikube-staging codfw: Update cert-manager to 1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327073 (https://phabricator.wikimedia.org/T427402) [10:27:08] (03PS1) 10JMeybohm: wikikube-staging eqiad: Update to cert-manager 1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327074 (https://phabricator.wikimedia.org/T427402) [10:28:32] (03CR) 10Ozge: [C:03+1] ml-services: Deploy latest editing-suggestions on experimental staging and prod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327072 (https://phabricator.wikimedia.org/T430030) (owner: 10Gkyziridis) [10:31:01] (03CR) 10Muehlenhoff: [C:03+2] Remove cluster management role from cumin2002 [puppet] - 10https://gerrit.wikimedia.org/r/1326829 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [10:31:09] 10SRE-swift-storage, 13Patch-For-Review: unstable device mapping of SSDs causing swift/puppet problems - example reimage - https://phabricator.wikimedia.org/T308644#12230330 (10MatthewVernon) 05Open→03Resolved a:03MatthewVernon With ms-be106{4,5,8} reimaged, that's now every swift (and thanos-swift)... [10:31:18] 10SRE-swift-storage, 13Patch-For-Review: unstable device mapping of SSDs causing installer problems - example reimage with destruction of swift filesystem - https://phabricator.wikimedia.org/T308677#12230333 (10MatthewVernon) 05Open→03Resolved a:03MatthewVernon We've now converted the final straggler... [10:33:53] (03CR) 10Muehlenhoff: [C:03+2] Remove cumin2002 from RAPI access rules [puppet] - 10https://gerrit.wikimedia.org/r/1326842 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [10:38:04] (03PS2) 10Federico Ceratto: site.pp: Set MariaDB role for db2902 [puppet] - 10https://gerrit.wikimedia.org/r/1327071 (https://phabricator.wikimedia.org/T435059) [10:38:15] (03CR) 10Federico Ceratto: "Updated" [puppet] - 10https://gerrit.wikimedia.org/r/1327071 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [10:39:06] (03CR) 10Marostegui: [C:03+1] site.pp: Set MariaDB role for db2902 [puppet] - 10https://gerrit.wikimedia.org/r/1327071 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [10:40:40] (03CR) 10Kevin Bazira: "Thank you for working on this." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327001 (owner: 10Gkyziridis) [10:42:08] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1185.eqiad.wmnet with OS bookworm [10:42:47] (03CR) 10JMeybohm: [C:03+1] "CI diff is a mess due to the new CRD structure, but overall LGTM" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320120 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:42:47] (03CR) 10Muehlenhoff: [C:03+2] Remove DB grant for cumin2002 [puppet] - 10https://gerrit.wikimedia.org/r/1326845 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [10:42:50] (03CR) 10Federico Ceratto: [C:03+2] site.pp: Set MariaDB role for db2902 [puppet] - 10https://gerrit.wikimedia.org/r/1327071 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [10:42:51] (03PS1) 10Hnowlan: team-sre: add serviceops tag [alerts] - 10https://gerrit.wikimedia.org/r/1327076 (https://phabricator.wikimedia.org/T432376) [10:42:53] (03PS1) 10Hnowlan: team-netops: replace use of "team: sre" with "team: netops" [alerts] - 10https://gerrit.wikimedia.org/r/1327077 (https://phabricator.wikimedia.org/T432376) [10:45:02] (03CR) 10Hnowlan: [C:04-1] "Routing for `netops` might need to be changed before this can be merged - currently only goes to the two netops engineers via email and wi" [alerts] - 10https://gerrit.wikimedia.org/r/1327077 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [10:45:16] 06SRE, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 13Patch-For-Review: Fix remaining scoped legacy fact usage - https://phabricator.wikimedia.org/T435225#12230411 (10LSobanski) 05Open→03In progress [10:45:28] 06SRE, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 13Patch-For-Review: Fix remaining scoped legacy fact usage - https://phabricator.wikimedia.org/T435225#12230414 (10LSobanski) p:05Triage→03Medium [10:45:42] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1169.eqiad.wmnet with OS bookworm [10:47:01] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: wait_for_optimal() should ignore acked alerts - https://phabricator.wikimedia.org/T319277#12230426 (10LSobanski) p:05Medium→03Triage [10:47:21] (03PS1) 10Majavah: Manage acct logrotation on Cloud VPS [puppet] - 10https://gerrit.wikimedia.org/r/1327079 (https://phabricator.wikimedia.org/T410410) [10:48:03] (03CR) 10CI reject: [V:04-1] Manage acct logrotation on Cloud VPS [puppet] - 10https://gerrit.wikimedia.org/r/1327079 (https://phabricator.wikimedia.org/T410410) (owner: 10Majavah) [10:48:24] (03CR) 10Gmodena: WIP: Enable posting to eventgate from the proxy. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326893 (https://phabricator.wikimedia.org/T433375) (owner: 10Lerickson) [10:49:01] (03PS2) 10Majavah: Manage acct logrotation on Cloud VPS [puppet] - 10https://gerrit.wikimedia.org/r/1327079 (https://phabricator.wikimedia.org/T410410) [10:50:15] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Clarify when spicerack/cumin is going to retry - https://phabricator.wikimedia.org/T433698#12230464 (10LSobanski) a:03Mahveotm @Mahveotm assigning this task to you for workboard clarity. [10:51:55] (03PS3) 10Majavah: Manage acct logrotation on Cloud VPS [puppet] - 10https://gerrit.wikimedia.org/r/1327079 (https://phabricator.wikimedia.org/T410410) [10:52:31] (03PS1) 10Muehlenhoff: Remove cumin2002 from Cumin master firewall rules [puppet] - 10https://gerrit.wikimedia.org/r/1327080 (https://phabricator.wikimedia.org/T427897) [10:55:23] (03PS1) 10Muehlenhoff: Remove cumin2002 from alertmanager config [puppet] - 10https://gerrit.wikimedia.org/r/1327083 (https://phabricator.wikimedia.org/T427897) [10:55:43] (03PS1) 10Ladsgroup: Revert "Migrate database access to virtual domains" [extensions/Echo] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327084 (https://phabricator.wikimedia.org/T435305) [10:56:06] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host cumin2002.codfw.wmnet [10:57:33] (03CR) 10Gkyziridis: [C:03+2] ml-services: Deploy latest editing-suggestions on experimental staging and prod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327072 (https://phabricator.wikimedia.org/T430030) (owner: 10Gkyziridis) [10:59:06] (03PS4) 10Majavah: Manage acct logrotation on Cloud VPS [puppet] - 10https://gerrit.wikimedia.org/r/1327079 (https://phabricator.wikimedia.org/T410410) [10:59:06] (03PS1) 10Majavah: P:base::labs: Ensure pacct file exists [puppet] - 10https://gerrit.wikimedia.org/r/1327087 [10:59:59] andre: Hi, if you want to, just merge this patch https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo/+/1327084 [11:00:05] mvolz: #bothumor I � Unicode. All rise for Services – Citoid / Zotero deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1100). [11:00:06] jouncebot: nowandnext [11:00:06] For the next 0 hour(s) and 59 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1100) [11:00:06] In 1 hour(s) and 59 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1300) [11:00:10] (03Merged) 10jenkins-bot: ml-services: Deploy latest editing-suggestions on experimental staging and prod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327072 (https://phabricator.wikimedia.org/T430030) (owner: 10Gkyziridis) [11:00:16] (03CR) 10Ladsgroup: [C:03+2] Revert "Migrate database access to virtual domains" [extensions/Echo] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327084 (https://phabricator.wikimedia.org/T435305) (owner: 10Ladsgroup) [11:01:46] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin2002.codfw.wmnet [11:01:52] !log gkyziridis@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [11:03:11] (03CR) 10David Caro: P:base::labs: Ensure pacct file exists (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1327087 (owner: 10Majavah) [11:03:25] FIRING: SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:04:31] (03PS1) 10Btullis: install_server: Install grub on all RAID members in reuse recipes [puppet] - 10https://gerrit.wikimedia.org/r/1327088 (https://phabricator.wikimedia.org/T434562) [11:05:19] Deploying MinT, staging first. [11:05:39] (03CR) 10KartikMistry: [C:03+2] Update Mint to 2026-06-04-131507-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326088 (https://phabricator.wikimedia.org/T321316) (owner: 10KartikMistry) [11:06:27] !log upgrading the new trixie URL downloaders to Squid 7.6 T427282 [11:06:28] FIRING: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [11:06:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:06:32] T427282: Move URL downloaders to trixie - https://phabricator.wikimedia.org/T427282 [11:06:40] (03CR) 10Btullis: [C:03+2] install_server: Install grub on all RAID members in reuse recipes [puppet] - 10https://gerrit.wikimedia.org/r/1327088 (https://phabricator.wikimedia.org/T434562) (owner: 10Btullis) [11:07:26] !log btullis@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host stat1008.eqiad.wmnet with OS bookworm [11:07:35] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host maps-test2001.codfw.wmnet [11:07:47] (03Merged) 10jenkins-bot: Update Mint to 2026-06-04-131507-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326088 (https://phabricator.wikimedia.org/T321316) (owner: 10KartikMistry) [11:07:57] (03PS1) 10Gkyziridis: ml-services: Fix typo in editing-suggestions experimental values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327090 [11:08:10] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-cron: apply [11:08:18] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [11:08:19] !log cgoubert@deploy1003 helmfile [codfw] START helmfile.d/services/mw-cron: apply [11:08:26] !log cgoubert@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-cron: apply [11:09:45] (03CR) 10Ozge: [C:03+1] ml-services: Fix typo in editing-suggestions experimental values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327090 (owner: 10Gkyziridis) [11:10:20] !log kartik@deploy1003 helmfile [staging] START helmfile.d/services/machinetranslation: apply [11:10:20] (03CR) 10Majavah: P:base::labs: Ensure pacct file exists (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1327087 (owner: 10Majavah) [11:11:28] RESOLVED: KeyholderUnarmed: 1 unarmed Keyholder key(s) on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [11:12:13] !log mvolz@deploy1003 helmfile [staging] START helmfile.d/services/citoid: apply [11:12:31] !log mvolz@deploy1003 helmfile [staging] DONE helmfile.d/services/citoid: apply [11:13:18] !log kartik@deploy1003 helmfile [staging] DONE helmfile.d/services/machinetranslation: apply [11:16:46] (03CR) 10Filippo Giunchedi: [C:03+1] P:base::labs: Ensure pacct file exists [puppet] - 10https://gerrit.wikimedia.org/r/1327087 (owner: 10Majavah) [11:17:03] !log kartik@deploy1003 helmfile [codfw] START helmfile.d/services/machinetranslation: apply [11:17:07] (03Merged) 10jenkins-bot: Revert "Migrate database access to virtual domains" [extensions/Echo] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327084 (https://phabricator.wikimedia.org/T435305) (owner: 10Ladsgroup) [11:17:21] (03PS1) 10Federico Ceratto: sre.mysql.clone: Remove same-DC check [cookbooks] - 10https://gerrit.wikimedia.org/r/1327092 (https://phabricator.wikimedia.org/T435059) [11:17:32] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1327079 (https://phabricator.wikimedia.org/T410410) (owner: 10Majavah) [11:17:57] (03PS2) 10Gkyziridis: ml-services: Fix typo in editing-suggestions experimental values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327090 [11:18:07] (03CR) 10Marostegui: [C:03+1] sre.mysql.clone: Remove same-DC check [cookbooks] - 10https://gerrit.wikimedia.org/r/1327092 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [11:18:17] ACKNOWLEDGEMENT - MD RAID on maps-test2001 is CRITICAL: CRITICAL: State: degraded, Active: 3, Working: 3, Failed: 0, Spare: 0 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T435311 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [11:18:25] RESOLVED: SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:18:25] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps-test2001.codfw.wmnet [11:18:25] 10ops-codfw, 06SRE, 06DC-Ops: Degraded RAID on maps-test2001 - https://phabricator.wikimedia.org/T435311 (10ops-monitoring-bot) 03NEW [11:18:35] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host stat1008.eqiad.wmnet with OS bookworm [11:18:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:19:14] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9274/co" [puppet] - 10https://gerrit.wikimedia.org/r/1327087 (owner: 10Majavah) [11:19:47] (03PS5) 10Majavah: Manage acct logrotation on Cloud VPS [puppet] - 10https://gerrit.wikimedia.org/r/1327079 (https://phabricator.wikimedia.org/T410410) [11:19:47] (03PS2) 10Majavah: P:base::labs: Ensure pacct file exists [puppet] - 10https://gerrit.wikimedia.org/r/1327087 [11:20:03] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1327084|Revert "Migrate database access to virtual domains" (T435305)]] [11:20:09] T435305: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'itwiki.echo_push_provider' doesn't exist - https://phabricator.wikimedia.org/T435305 [11:20:31] (03CR) 10Ozge: [C:03+1] ml-services: Fix typo in editing-suggestions experimental values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327090 (owner: 10Gkyziridis) [11:21:01] !log mvolz@deploy1003 helmfile [codfw] START helmfile.d/services/citoid: apply [11:21:02] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9275/co" [puppet] - 10https://gerrit.wikimedia.org/r/1327087 (owner: 10Majavah) [11:21:30] !log mvolz@deploy1003 helmfile [codfw] DONE helmfile.d/services/citoid: apply [11:21:34] !log kartik@deploy1003 helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply [11:22:09] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1327084|Revert "Migrate database access to virtual domains" (T435305)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:22:25] FIRING: SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:22:49] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (DIFF 1 CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1327079 (https://phabricator.wikimedia.org/T410410) (owner: 10Majavah) [11:23:22] !log kartik@deploy1003 helmfile [eqiad] START helmfile.d/services/machinetranslation: apply [11:23:23] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-d3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:23:26] !log mvolz@deploy1003 helmfile [codfw] START helmfile.d/services/citoid: apply [11:23:55] (03CR) 10Majavah: [V:03+1 C:03+2] "PS5 is a trivial fix (`s/present/installed/`) compared to +1'd PS4, merging" [puppet] - 10https://gerrit.wikimedia.org/r/1327079 (https://phabricator.wikimedia.org/T410410) (owner: 10Majavah) [11:23:56] !log mvolz@deploy1003 helmfile [codfw] DONE helmfile.d/services/citoid: apply [11:24:07] !log mvolz@deploy1003 helmfile [eqiad] START helmfile.d/services/citoid: apply [11:24:39] !log mvolz@deploy1003 helmfile [eqiad] DONE helmfile.d/services/citoid: apply [11:24:47] (03CR) 10Gkyziridis: [C:03+2] ml-services: Fix typo in editing-suggestions experimental values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327090 (owner: 10Gkyziridis) [11:25:48] (03PS3) 10Majavah: P:base::labs: Ensure pacct file exists [puppet] - 10https://gerrit.wikimedia.org/r/1327087 [11:26:48] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [11:27:06] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9277/co" [puppet] - 10https://gerrit.wikimedia.org/r/1327087 (owner: 10Majavah) [11:27:08] (03Merged) 10jenkins-bot: ml-services: Fix typo in editing-suggestions experimental values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327090 (owner: 10Gkyziridis) [11:28:03] Amir1: not happy to see that land, but what can i do :-/ [11:28:43] urbanecm: me neither but I swear, I looked a lot to find it. Also it's not merged on master, we have a week to find a solution :P [11:28:46] !log gkyziridis@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [11:28:52] (03CR) 10Majavah: [V:03+1 C:03+2] P:base::labs: Ensure pacct file exists [puppet] - 10https://gerrit.wikimedia.org/r/1327087 (owner: 10Majavah) [11:28:59] !log kartik@deploy1003 helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply [11:29:20] !log Updated MinT to 2026-06-04-131507-production (T321316) [11:29:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:29:26] T321316: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316 [11:29:48] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [11:29:59] !log gkyziridis@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [11:30:39] Amir1: yeah... i can take a look a bit too, but i don't trust my ability to find rdbms solutions you don't see, so... :D [11:31:05] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327084|Revert "Migrate database access to virtual domains" (T435305)]] (duration: 11m 02s) [11:31:10] T435305: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'itwiki.echo_push_provider' doesn't exist - https://phabricator.wikimedia.org/T435305 [11:31:34] I think NameTableStore needs to be adjusted [11:36:10] (03CR) 10Majavah: "Do we know how long it takes for nova to notice a host being down, compared to node-exporter scrapes noticing the same? (I wonder if we sh" [alerts] - 10https://gerrit.wikimedia.org/r/1326288 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [11:36:11] RESOLVED: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:36:13] (03CR) 10Jelto: [C:03+2] cert-manager: update chart for v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320120 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [11:39:07] (03CR) 10Majavah: [C:03+1] team-wmcs: gate NeutronAgentDown based on host maintenance status [alerts] - 10https://gerrit.wikimedia.org/r/1326294 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [11:39:27] (03CR) 10Majavah: [C:03+1] team-wmcs: escalate NeutronAgentDown warning -> critical [alerts] - 10https://gerrit.wikimedia.org/r/1326296 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [11:40:23] (03CR) 10Majavah: team-wmcs: add alert on cloudvirt in maintenance for multiple days (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1326297 (https://phabricator.wikimedia.org/T284747) (owner: 10Filippo Giunchedi) [11:43:52] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Migrate remaining container build/report steps from build2001 to build2004 - https://phabricator.wikimedia.org/T417389#12230702 (10MoritzMuehlenhoff) [11:44:00] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on stat1008.eqiad.wmnet with reason: host reimage [11:44:52] (03PS2) 10Filippo Giunchedi: icinga: remove wikitech-static monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1326279 (https://phabricator.wikimedia.org/T362397) [11:45:46] 06SRE, 06Infrastructure-Foundations, 06ServiceOps: Migrate Docker reporting from build2002 to build2004 - https://phabricator.wikimedia.org/T435314 (10MoritzMuehlenhoff) 03NEW [11:45:50] 06SRE, 06Infrastructure-Foundations, 06ServiceOps: Migrate Docker reporting from build2002 to build2004 - https://phabricator.wikimedia.org/T435314#12230718 (10MoritzMuehlenhoff) p:05Triage→03Medium [11:46:43] (03Merged) 10jenkins-bot: cert-manager: update chart for v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320120 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [11:47:00] (03CR) 10CI reject: [V:04-1] icinga: remove wikitech-static monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1326279 (https://phabricator.wikimedia.org/T362397) (owner: 10Filippo Giunchedi) [11:47:12] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Migrate remaining container build/report steps from build2001 to build2004 - https://phabricator.wikimedia.org/T417389#12230720 (10MoritzMuehlenhoff) [11:48:18] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on stat1008.eqiad.wmnet with reason: host reimage [11:48:45] (03PS1) 10Gergő Tisza: SpecialMWOAuthListConsumers: Handle newFromMWUser returning null in addNavigationSubtitle [extensions/OAuth] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327094 (https://phabricator.wikimedia.org/T435167) [11:49:39] !log installing kerberos security updates [11:49:39] (03CR) 10Klausman: [C:03+1] install_server: Install grub on all RAID members in reuse recipes [puppet] - 10https://gerrit.wikimedia.org/r/1327088 (https://phabricator.wikimedia.org/T434562) (owner: 10Btullis) [11:49:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:50:15] (03PS3) 10Filippo Giunchedi: icinga: remove wikitech-static monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1326279 (https://phabricator.wikimedia.org/T362397) [11:50:35] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 19 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/OAuth] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327094 (https://phabricator.wikimedia.org/T435167) (owner: 10Gergő Tisza) [11:51:21] jouncebot: nowandnext [11:51:21] For the next 0 hour(s) and 8 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1100) [11:51:22] In 1 hour(s) and 8 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1300) [11:51:38] btullis@cumin1003 reimage (PID 1371291) is awaiting input [11:52:37] (03PS1) 10Ladsgroup: Switch to redis lock manager on s4 and s8 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327096 (https://phabricator.wikimedia.org/T366938) [11:53:15] (03PS1) 10Jelto: wikikube-staging: Update cert-manager to v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327097 (https://phabricator.wikimedia.org/T427402) [11:58:33] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host krb2002.codfw.wmnet [12:02:08] (03PS1) 10Dreamy Jazz: UserRightsNotificationHandler: Use correct virtual domain for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) [12:02:12] (03CR) 10Federico Ceratto: [C:03+2] sre.mysql.clone: Remove same-DC check [cookbooks] - 10https://gerrit.wikimedia.org/r/1327092 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [12:04:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2002.codfw.wmnet [12:04:49] (03CR) 10EarlyWarningBot: "[Failed command](https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/102636/consoleFull): `composer run --timeout=0 phpuni" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [12:04:56] (03CR) 10CI reject: [V:04-1] UserRightsNotificationHandler: Use correct virtual domain for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [12:05:56] (03PS2) 10Dreamy Jazz: UserRightsNotificationHandler: Use correct virtual domain for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) [12:06:25] (03CR) 10EarlyWarningBot: "[Failed command](https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/102638/consoleFull): `composer --ansi test`" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [12:07:22] (03Abandoned) 10Jelto: wikikube-staging: Update cert-manager to v1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327097 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [12:07:35] (03CR) 10CI reject: [V:04-1] UserRightsNotificationHandler: Use correct virtual domain for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [12:07:52] (03CR) 10Jelto: [C:03+2] "lgtm, thank you!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327073 (https://phabricator.wikimedia.org/T427402) (owner: 10JMeybohm) [12:08:41] (03PS1) 10Slyngshede: P:swift::proxy add thumb.wikimedia.org to swift frontend [puppet] - 10https://gerrit.wikimedia.org/r/1327100 (https://phabricator.wikimedia.org/T427465) [12:09:06] (03PS3) 10Dreamy Jazz: UserRightsNotificationHandler: Use correct virtual domain for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) [12:09:48] (03PS3) 10JMeybohm: coredns: Rename coredns to coredns1.11 to support multiple versions [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321573 (https://phabricator.wikimedia.org/T433590) [12:09:48] (03PS3) 10JMeybohm: Add coredns 1.12 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321574 (https://phabricator.wikimedia.org/T433590) [12:09:49] (03PS1) 10Dreamy Jazz: UserRightsNotificationHandler: Use correct virtual domain for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1327101 (https://phabricator.wikimedia.org/T435263) [12:09:54] (03CR) 10Ladsgroup: [C:03+1] P:swift::proxy add thumb.wikimedia.org to swift frontend [puppet] - 10https://gerrit.wikimedia.org/r/1327100 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [12:10:31] btullis@cumin1003 reimage (PID 1371291) is awaiting input [12:10:59] (03PS4) 10Dreamy Jazz: UserRightsNotificationHandler: Use correct database for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) [12:11:05] (03PS2) 10Dreamy Jazz: UserRightsNotificationHandler: Use correct database for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1327101 (https://phabricator.wikimedia.org/T435263) [12:11:27] jouncebot: nowandnext [12:11:27] No deployments scheduled for the next 0 hour(s) and 48 minute(s) [12:11:27] In 0 hour(s) and 48 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1300) [12:13:18] (03CR) 10EarlyWarningBot: "[Failed command](https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/102642/consoleFull): `composer run --timeout=0 phpuni" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [12:13:25] (03CR) 10CI reject: [V:04-1] UserRightsNotificationHandler: Use correct database for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [12:13:35] (03CR) 10EarlyWarningBot: "[Failed command](https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/102643/consoleFull): `composer run --timeout=0 phpuni" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1327101 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [12:13:42] (03CR) 10CI reject: [V:04-1] UserRightsNotificationHandler: Use correct database for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1327101 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [12:14:20] (03PS5) 10Dreamy Jazz: UserRightsNotificationHandler: Use correct database for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) [12:14:38] (Not going to use scap, not got enough time) [12:15:29] (03CR) 10Majavah: [C:03+1] icinga: remove wikitech-static monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1326279 (https://phabricator.wikimedia.org/T362397) (owner: 10Filippo Giunchedi) [12:15:35] (03CR) 10Elukey: [C:03+1] Puppet 8: Replace legacy facts in module interface [puppet] - 10https://gerrit.wikimedia.org/r/1326909 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [12:16:04] (03CR) 10Elukey: [C:03+1] admin::hashuser: fix gid check [puppet] - 10https://gerrit.wikimedia.org/r/1326900 (owner: 10JHathaway) [12:17:37] (03CR) 10Jelto: "two questions in line" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321573 (https://phabricator.wikimedia.org/T433590) (owner: 10JMeybohm) [12:17:41] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1326900 (owner: 10JHathaway) [12:17:52] (03Merged) 10jenkins-bot: wikikube-staging codfw: Update cert-manager to 1.19.6 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327073 (https://phabricator.wikimedia.org/T427402) (owner: 10JMeybohm) [12:17:56] (03CR) 10MVernon: [C:03+1] "LGTM :)" [puppet] - 10https://gerrit.wikimedia.org/r/1327100 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [12:18:05] (03PS1) 10Muehlenhoff: docker::baseimages: Deploy files/directories independent of the systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/1327103 (https://phabricator.wikimedia.org/T417389) [12:18:31] (03PS3) 10Dreamy Jazz: UserRightsNotificationHandler: Use correct database for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1327101 (https://phabricator.wikimedia.org/T435263) [12:18:56] (03PS2) 10Muehlenhoff: docker::baseimages: Deploy files/directories independent of the systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/1327103 (https://phabricator.wikimedia.org/T417389) [12:20:25] 06SRE, 10SRE-swift-storage, 06cloud-services-team, 10Cloud-VPS, and 2 others: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#12230867 (10taavi) Tagging what I believe are the right projects for the two remaining profiles. [12:21:02] (03PS1) 10Ladsgroup: swift: Also accept thumb.wikimedia.org in ensure_max_mage [puppet] - 10https://gerrit.wikimedia.org/r/1327104 (https://phabricator.wikimedia.org/T427465) [12:22:18] (03CR) 10Ladsgroup: swift: Also accept thumb.wikimedia.org in ensure_max_mage (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1327104 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [12:23:10] 06SRE, 10fundraising-tech-ops, 06Infrastructure-Foundations, 10netops: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12230878 (10cmooney) As per conversation with @Jgreen on irc we will do the eqiad pair at 09:00UTC tomorrow morning (Thurs 20th). [12:23:14] (03CR) 10Ladsgroup: [C:03+2] P:swift::proxy add thumb.wikimedia.org to swift frontend [puppet] - 10https://gerrit.wikimedia.org/r/1327100 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [12:25:07] Dreamy_Jazz: are you currently using the window? i'd like to do a helmfile-only deployment during this window if possible :) [12:25:07] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1327103 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [12:25:52] Nope, not using scap [12:25:55] (03CR) 10Clément Goubert: [C:03+1] mediawiki: stop using lamp.deployment in job.yaml.tpl. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325884 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [12:26:03] gotcha, thanks! [12:26:26] (03CR) 10Blake: [C:03+2] mediawiki: stop using lamp.deployment in job.yaml.tpl. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325884 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [12:27:49] (03CR) 10Jforrester: [C:03+2] "…" [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1326947 (owner: 10TrainBranchBot) [12:28:16] (03PS1) 10Btullis: Install amd_rocm version 6.1 on stat1008 after reimage to bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1327105 (https://phabricator.wikimedia.org/T434562) [12:28:37] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:30:03] (03PS3) 10Muehlenhoff: Remove obsolete ms-be.cfg Partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/1324354 (https://phabricator.wikimedia.org/T156955) [12:31:02] (03Merged) 10jenkins-bot: mediawiki: stop using lamp.deployment in job.yaml.tpl. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325884 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [12:31:48] (03CR) 10Btullis: [C:03+2] Install amd_rocm version 6.1 on stat1008 after reimage to bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1327105 (https://phabricator.wikimedia.org/T434562) (owner: 10Btullis) [12:32:31] (03CR) 10Ladsgroup: Allow setting a separate thumbUrl in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319920 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [12:32:41] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1326947 (owner: 10TrainBranchBot) [12:35:07] !log blake@deploy1003 Started scap sync-world: non-build deploy for T417800 [12:35:12] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [12:35:29] (03PS6) 10Blake: mediawiki: optionally use initContainers for sidecars. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325883 (https://phabricator.wikimedia.org/T417800) [12:36:21] (03CR) 10Elukey: [C:03+2] docker_registry: route /v2/releng to its new s3 backend [puppet] - 10https://gerrit.wikimedia.org/r/1326323 (https://phabricator.wikimedia.org/T432829) (owner: 10Elukey) [12:36:32] jouncebot: nowandnext [12:36:32] No deployments scheduled for the next 0 hour(s) and 23 minute(s) [12:36:32] In 0 hour(s) and 23 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1300) [12:36:48] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [12:37:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327096 (https://phabricator.wikimedia.org/T366938) (owner: 10Ladsgroup) [12:37:42] marostegui: ^ [12:37:54] going for redis lock manager on s8 and s4 [12:38:22] !log blake@deploy1003 Finished scap sync-world: non-build deploy for T417800 (duration: 03m 52s) [12:38:24] (03Merged) 10jenkins-bot: Switch to redis lock manager on s4 and s8 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327096 (https://phabricator.wikimedia.org/T366938) (owner: 10Ladsgroup) [12:38:48] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1327096|Switch to redis lock manager on s4 and s8 (T366938)]] [12:38:53] T366938: Reduce relying on database locks - https://phabricator.wikimedia.org/T366938 [12:39:03] (03PS3) 10Ladsgroup: Allow setting a separate thumbUrl in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319920 (https://phabricator.wikimedia.org/T427465) [12:39:55] Amir1: thanks [12:40:53] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1327096|Switch to redis lock manager on s4 and s8 (T366938)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:41:00] (03CR) 10MVernon: [C:03+1] "Should be good to go now, thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1324354 (https://phabricator.wikimedia.org/T156955) (owner: 10Muehlenhoff) [12:41:18] !log update cert-manager to 1.19.6 on wikikube staging-codfw - T427402 [12:41:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:41:23] T427402: Update cert-manager to 1.19 - https://phabricator.wikimedia.org/T427402 [12:42:05] !log jelto@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'sync'. [12:42:14] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqord cct - cmooney@cumin1003" [12:42:32] (03PS1) 10Cathal Mooney: Remove include for disued IPv6 link net for ulsfo<->eqord cct [dns] - 10https://gerrit.wikimedia.org/r/1327109 [12:42:46] !log jelto@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [12:43:21] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqord:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [12:43:42] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [12:44:26] (03CR) 10Filippo Giunchedi: [C:03+2] icinga: remove wikitech-static monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1326279 (https://phabricator.wikimedia.org/T362397) (owner: 10Filippo Giunchedi) [12:45:17] cmooney@cumin1003 netbox (PID 1429793) is awaiting input [12:45:33] !log move the /v2/releng.* prefix on the Docker Registry to its new s3 backend - T432829 [12:45:35] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqord cct - cmooney@cumin1003" [12:45:35] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:45:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:45:39] T432829: Move Docker images under the /v2/releng prefix to S3 - https://phabricator.wikimedia.org/T432829 [12:45:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:45:50] (03CR) 10Cathal Mooney: [C:03+2] Remove include for disued IPv6 link net for ulsfo<->eqord cct [dns] - 10https://gerrit.wikimedia.org/r/1327109 (owner: 10Cathal Mooney) [12:46:06] !log cmooney@dns3003 START - running authdns-update [12:47:22] (03PS3) 10Blake: admin_ng: Use kube-state-metrics 7.3.0 in staging-codfw. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327093 (https://phabricator.wikimedia.org/T427405) [12:47:42] (03CR) 10Muehlenhoff: [C:03+2] Remove obsolete ms-be.cfg Partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/1324354 (https://phabricator.wikimedia.org/T156955) (owner: 10Muehlenhoff) [12:47:59] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327096|Switch to redis lock manager on s4 and s8 (T366938)]] (duration: 09m 11s) [12:48:04] T366938: Reduce relying on database locks - https://phabricator.wikimedia.org/T366938 [12:48:21] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqord:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [12:48:22] !log cmooney@dns3003 END - running authdns-update [12:50:05] 10ops-eqiad, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T433348#12230994 (10BTullis) [12:50:41] 10ops-eqiad, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T433348#12230996 (10BTullis) [12:52:00] (03PS1) 10Kosta Harlan: AbuseReview: Take the review verdict as a REST path parameter [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327110 (https://phabricator.wikimedia.org/T435020) [12:52:02] (03PS1) 10Kosta Harlan: Special:AbuseReview: Add "no further action needed" review action [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327111 (https://phabricator.wikimedia.org/T435020) [12:52:25] jouncebot: nowandnext [12:52:25] No deployments scheduled for the next 0 hour(s) and 7 minute(s) [12:52:25] In 0 hour(s) and 7 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1300) [12:53:10] !log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2902.codfw.wmnet with reason: Cloning [12:53:20] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 19 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327110 (https://phabricator.wikimedia.org/T435020) (owner: 10Kosta Harlan) [12:53:37] (03CR) 10Elukey: "Should we exclude the /usr/local/bin scripts to avoid people accidentally running them on the wrong host? Or do you think it is not needed" [puppet] - 10https://gerrit.wikimedia.org/r/1327103 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [12:53:37] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 19 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327111 (https://phabricator.wikimedia.org/T435020) (owner: 10Kosta Harlan) [12:55:15] (03CR) 10JMeybohm: coredns: Rename coredns to coredns1.11 to support multiple versions (032 comments) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321573 (https://phabricator.wikimedia.org/T433590) (owner: 10JMeybohm) [12:55:25] (03PS4) 10JMeybohm: coredns: Rename coredns to coredns1.11 to support multiple versions [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321573 (https://phabricator.wikimedia.org/T433590) [12:55:25] (03PS4) 10JMeybohm: Add coredns 1.12 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321574 (https://phabricator.wikimedia.org/T433590) [12:56:01] (03CR) 10Filippo Giunchedi: "Good point, I did a quick test bringing down nova-compute and the control plane noticed after ~60s which more or less matches prometheus." [alerts] - 10https://gerrit.wikimedia.org/r/1326288 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [12:56:10] (03CR) 10Filippo Giunchedi: [C:03+2] team-wmcs: add alert on cloudvirt in maintenance for multiple days [alerts] - 10https://gerrit.wikimedia.org/r/1326297 (https://phabricator.wikimedia.org/T284747) (owner: 10Filippo Giunchedi) [12:56:36] (03CR) 10Filippo Giunchedi: [C:03+2] team-wmcs: escalate NeutronAgentDown warning -> critical [alerts] - 10https://gerrit.wikimedia.org/r/1326296 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [12:56:48] (03CR) 10Filippo Giunchedi: [C:03+2] team-wmcs: gate NeutronAgentDown based on host maintenance status [alerts] - 10https://gerrit.wikimedia.org/r/1326294 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [12:56:52] (03PS2) 10Filippo Giunchedi: team-wmcs: gate NeutronAgentDown based on host maintenance status [alerts] - 10https://gerrit.wikimedia.org/r/1326294 (https://phabricator.wikimedia.org/T424802) [12:56:55] (03CR) 10Filippo Giunchedi: [V:03+2 C:03+2] team-wmcs: gate NeutronAgentDown based on host maintenance status [alerts] - 10https://gerrit.wikimedia.org/r/1326294 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [12:57:08] (03PS2) 10Filippo Giunchedi: team-wmcs: escalate NeutronAgentDown warning -> critical [alerts] - 10https://gerrit.wikimedia.org/r/1326296 (https://phabricator.wikimedia.org/T424802) [12:57:10] !log sukhe@cumin1003 START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum and A:durum [12:57:11] (03CR) 10Filippo Giunchedi: [V:03+2 C:03+2] team-wmcs: escalate NeutronAgentDown warning -> critical [alerts] - 10https://gerrit.wikimedia.org/r/1326296 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [12:58:17] (03CR) 10Majavah: [C:03+1] "Meh, a minute is fine I think." [alerts] - 10https://gerrit.wikimedia.org/r/1326288 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [12:59:09] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [12:59:16] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [12:59:28] (03PS2) 10Filippo Giunchedi: team-wmcs: add alert on cloudvirt in maintenance for multiple days [alerts] - 10https://gerrit.wikimedia.org/r/1326297 (https://phabricator.wikimedia.org/T284747) [12:59:33] (03CR) 10Filippo Giunchedi: [C:03+2] team-wmcs: page on nova-reported cloudvirt status [alerts] - 10https://gerrit.wikimedia.org/r/1326288 (https://phabricator.wikimedia.org/T424802) (owner: 10Filippo Giunchedi) [12:59:47] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [12:59:54] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [13:00:04] Lucas_WMDE, urbanecm, and TheresNoTime: #bothumor My software never has bugs. It just develops random features. Rise for UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1300). [13:00:04] dyepezg, tgr, and kostajh: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:35] (03CR) 10Gkyziridis: "There is no Phabricator task for this task I think." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327001 (owner: 10Gkyziridis) [13:00:38] (03CR) 10Scott French: [C:03+2] wmnet: Reduce _etcd._tcp.conftool (R/W) SRV TTL to 10s [dns] - 10https://gerrit.wikimedia.org/r/1326363 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [13:00:51] !log swfrench@dns1004 START - running authdns-update [13:00:53] (03CR) 10Filippo Giunchedi: team-wmcs: add alert on cloudvirt in maintenance for multiple days (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1326297 (https://phabricator.wikimedia.org/T284747) (owner: 10Filippo Giunchedi) [13:01:03] !log sukhe@cumin1003 START - Cookbook sre.hosts.reboot-single for host ncmonitor1001.eqiad.wmnet [13:01:06] (03CR) 10Filippo Giunchedi: [V:03+2 C:03+2] team-wmcs: add alert on cloudvirt in maintenance for multiple days [alerts] - 10https://gerrit.wikimedia.org/r/1326297 (https://phabricator.wikimedia.org/T284747) (owner: 10Filippo Giunchedi) [13:01:10] FIRING: [2x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:103:10:192:32:58 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:01:34] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [13:01:41] dyepezg: are you around? I can deploy your patch if so [13:01:41] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [13:01:58] tgr|away: my patch touches i18n files, so I can go at the end, if you're ready to deploy yours now [13:02:39] (03PS3) 10Filippo Giunchedi: team-wmcs: add alert on cloudvirt in maintenance for multiple days [alerts] - 10https://gerrit.wikimedia.org/r/1326297 (https://phabricator.wikimedia.org/T284747) [13:02:45] (03CR) 10Filippo Giunchedi: [V:03+2] team-wmcs: add alert on cloudvirt in maintenance for multiple days [alerts] - 10https://gerrit.wikimedia.org/r/1326297 (https://phabricator.wikimedia.org/T284747) (owner: 10Filippo Giunchedi) [13:03:03] !log swfrench@dns1004 END - running authdns-update [13:03:09] (03PS2) 10Scott French: wmnet: Switch _etcd._tcp.conftool (R/W) SRV hosts to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1326364 (https://phabricator.wikimedia.org/T435103) [13:03:09] (03PS2) 10Scott French: wmnet: Restore _etcd._tcp.conftool (R/W) SRV TTL to 5M [dns] - 10https://gerrit.wikimedia.org/r/1326365 (https://phabricator.wikimedia.org/T435103) [13:04:11] not hearing from others, I'll start with mine [13:04:26] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327111 (https://phabricator.wikimedia.org/T435020) (owner: 10Kosta Harlan) [13:04:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327110 (https://phabricator.wikimedia.org/T435020) (owner: 10Kosta Harlan) [13:04:55] btullis@cumin1003 reimage (PID 1371291) is awaiting input [13:05:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:05:40] !log sudo -i cookbook sre.cdn.roll-upgrade-ats --query 'A:cp-eqiad' --task-id T434478 --reason '9.2.15 upgrade' [13:05:43] (03Merged) 10jenkins-bot: AbuseReview: Take the review verdict as a REST path parameter [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327110 (https://phabricator.wikimedia.org/T435020) (owner: 10Kosta Harlan) [13:05:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:05:45] (03Merged) 10jenkins-bot: Special:AbuseReview: Add "no further action needed" review action [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327111 (https://phabricator.wikimedia.org/T435020) (owner: 10Kosta Harlan) [13:05:53] sudo -i cookbook sre.cdn.roll-upgrade-ats --query 'A:cp-codfw' --task-id T434478 --reason '9.2.15 upgrade' [13:05:57] !log cdobbins@cumin1003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.15 upgrade (T434478) [13:06:02] !log cdobbins@cumin1003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.15 upgrade (T434478) [13:06:07] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1327111|Special:AbuseReview: Add "no further action needed" review action (T435020)]], [[gerrit:1327110|AbuseReview: Take the review verdict as a REST path parameter (T435020)]] [13:06:10] FIRING: [4x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:06:40] T435020: AbuseReview: Add ability to indicate "no further action needed" for revisions - https://phabricator.wikimedia.org/T435020 [13:07:11] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations, 13Patch-For-Review: Move Docker images under the /v2/releng prefix to S3 - https://phabricator.wikimedia.org/T432829#12231164 (10elukey) The /v2/releng prefix has been moved, and I created https://wikitech.wikimedia.org/wiki/Docker-registry#Storage... [13:09:13] 06SRE, 10SRE-Access-Requests, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Requesting access to Analytics Data Lake for mkrolik/mkrolik-wmf - https://phabricator.wikimedia.org/T434877#12231179 (10HShaikh) Sorry this slipped through the cracks. Yes I approve this request for Marcin [13:10:23] !log cdobbins@cumin1003 END (ERROR) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=97) Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.15 upgrade (T434478) [13:10:28] !log cdobbins@cumin1003 END (ERROR) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=97) Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.15 upgrade (T434478) [13:12:34] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [13:12:41] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [13:13:08] 10SRE-SLO, 10Citoid, 06Editing-team, 07Sustainability (Incident Followup): Improve monitoring in citoid so that url-downloader failures are detected - https://phabricator.wikimedia.org/T381372#12231182 (10Mvolz) @CDanis and I discussed this last week and determined the SLO paradigm doesn't really work for... [13:13:16] kostajh: sorry I was AFK. Not in a hurry, anyway. [13:13:25] all good [13:13:27] kostajh, i'm active now [13:16:51] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [13:16:58] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [13:17:29] (03CR) 10Dpogorzelski: [C:03+2] kserve: update images to 0.20 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1327059 (https://phabricator.wikimedia.org/T433973) (owner: 10Dpogorzelski) [13:17:34] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] kserve: update images to 0.20 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1327059 (https://phabricator.wikimedia.org/T433973) (owner: 10Dpogorzelski) [13:18:15] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 19 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326342 (owner: 10Ssingh) [13:19:55] (03CR) 10Jelto: "looks mostly good but the test fails, see comment in-line" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321573 (https://phabricator.wikimedia.org/T433590) (owner: 10JMeybohm) [13:20:03] (03PS2) 10Ssingh: static: add new dir bimi/ for BIMI SVG and PEM file [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1314962 (https://phabricator.wikimedia.org/T311685) [13:20:14] dyepezg: ack, will let you know later in the hour [13:21:10] FIRING: [5x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:21:42] Ok [13:23:45] (03PS1) 10Dpogorzelski: knative: fix build warnings [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1327114 [13:23:56] !log kharlan@deploy1003 kharlan: Backport for [[gerrit:1327111|Special:AbuseReview: Add "no further action needed" review action (T435020)]], [[gerrit:1327110|AbuseReview: Take the review verdict as a REST path parameter (T435020)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:23:56] (03CR) 10Dpogorzelski: [C:03+2] knative: fix build warnings [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1327114 (owner: 10Dpogorzelski) [13:23:58] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] knative: fix build warnings [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1327114 (owner: 10Dpogorzelski) [13:24:01] T435020: AbuseReview: Add ability to indicate "no further action needed" for revisions - https://phabricator.wikimedia.org/T435020 [13:24:39] !log btullis@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host an-worker1147 [13:24:51] !log btullis@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1147 [13:24:53] !log kharlan@deploy1003 kharlan: Continuing with deployment [13:25:35] !log sukhe@cumin1003 END (ERROR) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=97) rolling reboot on A:durum and A:durum [13:26:10] FIRING: [6x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:27:33] (03PS1) 10JMeybohm: coredns: Update to 1.12.1, add kubepods support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327115 (https://phabricator.wikimedia.org/T428573) [13:28:02] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [13:28:09] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [13:28:10] (03CR) 10Scott French: [C:03+2] hieradata: etcd read-only in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1326355 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [13:30:33] (03PS1) 10JavierMonton: stream: pageview-trending-relative (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327116 (https://phabricator.wikimedia.org/T435291) [13:31:31] !log starting etcd-main codfw -> eqiad switchover - T435103 [13:31:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:31:36] T435103: etcd primary cluster switchover (codfw -> eqiad) - https://phabricator.wikimedia.org/T435103 [13:32:54] (03PS5) 10JMeybohm: coredns: Rename coredns to coredns1.11 to support multiple versions [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321573 (https://phabricator.wikimedia.org/T433590) [13:32:54] (03PS5) 10JMeybohm: Add coredns 1.12 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321574 (https://phabricator.wikimedia.org/T433590) [13:32:55] (03CR) 10Scott French: [C:03+2] hieradata: switch etcd replication from eqiad to codfw [puppet] - 10https://gerrit.wikimedia.org/r/1326356 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [13:33:02] (03CR) 10JMeybohm: coredns: Rename coredns to coredns1.11 to support multiple versions (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321573 (https://phabricator.wikimedia.org/T433590) (owner: 10JMeybohm) [13:33:47] (03PS2) 10JMeybohm: coredns: Update to 1.12.1, add kubepods support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327115 (https://phabricator.wikimedia.org/T428573) [13:35:25] (03PS2) 10JavierMonton: stream: pageview-trending-relative (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327116 (https://phabricator.wikimedia.org/T435291) [13:37:48] dyepezg: I'll sync your patch next [13:37:50] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327111|Special:AbuseReview: Add "no further action needed" review action (T435020)]], [[gerrit:1327110|AbuseReview: Take the review verdict as a REST path parameter (T435020)]] (duration: 31m 43s) [13:37:54] T435020: AbuseReview: Add ability to indicate "no further action needed" for revisions - https://phabricator.wikimedia.org/T435020 [13:38:01] tgr_: after that, I have a config patch for urldownloaders to sync, then i'll hand over to you [13:38:07] :) [13:38:17] (03CR) 10Jelto: [V:03+1 C:03+1] "lgtm now, verified locally" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321573 (https://phabricator.wikimedia.org/T433590) (owner: 10JMeybohm) [13:38:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325532 (https://phabricator.wikimedia.org/T434748) (owner: 10Danielyepezgarces) [13:38:59] (03CR) 10Scott French: [C:03+2] wmnet: Switch _etcd._tcp.conftool (R/W) SRV hosts to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1326364 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [13:39:07] !log swfrench@dns1004 START - running authdns-update [13:39:25] (03Merged) 10jenkins-bot: srwiki: Allow bureaucrats to add and remove event-organizer group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325532 (https://phabricator.wikimedia.org/T434748) (owner: 10Danielyepezgarces) [13:39:51] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1325532|srwiki: Allow bureaucrats to add and remove event-organizer group (T434748)]] [13:39:56] T434748: Implement Serbian Wikipedia CampaignEvents community consensus - https://phabricator.wikimedia.org/T434748 [13:40:42] (03CR) 10Scott French: [C:03+2] hieradata: etcd read-write in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1326357 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [13:41:15] !log swfrench@dns1004 END - running authdns-update [13:41:30] If anyone can help me verify the change, as my university's internet is very slow, and I'm currently taking mobile development classes :’) [13:42:36] (03PS3) 10JMeybohm: coredns: Update to 1.12.1, add kubepods support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327115 (https://phabricator.wikimedia.org/T428573) [13:43:14] (03CR) 10JMeybohm: [V:03+2 C:03+2] coredns: Rename coredns to coredns1.11 to support multiple versions [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1321573 (https://phabricator.wikimedia.org/T433590) (owner: 10JMeybohm) [13:44:08] !log swfrench@cumin2003 conftool action : set/pooled=no; selector: name=wikikube-worker2330.codfw.wmnet [13:44:12] !log kharlan@deploy1003 kharlan, danielyepezgarces: Backport for [[gerrit:1325532|srwiki: Allow bureaucrats to add and remove event-organizer group (T434748)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:44:14] !log finished etcd-main codfw -> eqiad switchover - T435103 [13:44:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:44:21] T435103: etcd primary cluster switchover (codfw -> eqiad) - https://phabricator.wikimedia.org/T435103 [13:45:06] !log swfrench@cumin2003 conftool action : set/pooled=yes; selector: name=wikikube-worker2330.codfw.wmnet [13:45:48] dyepezg: I can verify it [13:46:14] Amir1: Re Echo patch: Sorry, I was out. I think I'll wait for the specialists to come to a conclusion, it doesn't block the train :) [13:46:29] tks [13:48:00] !log kharlan@deploy1003 kharlan, danielyepezgarces: Continuing with deployment [13:48:53] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 19 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1314962 (https://phabricator.wikimedia.org/T311685) (owner: 10Ssingh) [13:51:30] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [13:51:37] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [13:51:41] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-08-11-201322 to 2026-08-19-122930 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327118 (https://phabricator.wikimedia.org/T396943) [13:51:49] (03CR) 10Scott French: [C:03+2] wmnet: Restore _etcd._tcp.conftool (R/W) SRV TTL to 5M [dns] - 10https://gerrit.wikimedia.org/r/1326365 (https://phabricator.wikimedia.org/T435103) (owner: 10Scott French) [13:51:56] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-08-11-210638 to 2026-08-19-122639 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327119 (https://phabricator.wikimedia.org/T327413) [13:52:14] !log swfrench@dns1004 START - running authdns-update [13:54:25] !log swfrench@dns1004 END - running authdns-update [13:54:48] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325532|srwiki: Allow bureaucrats to add and remove event-organizer group (T434748)]] (duration: 14m 56s) [13:54:53] T434748: Implement Serbian Wikipedia CampaignEvents community consensus - https://phabricator.wikimedia.org/T434748 [13:54:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326342 (owner: 10Ssingh) [13:56:00] (03Merged) 10jenkins-bot: Revert^2 "wmf-config/ProductionServices: set URL for urldownloader to service record" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326342 (owner: 10Ssingh) [13:56:03] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:56:21] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1326342|Revert^2 "wmf-config/ProductionServices: set URL for urldownloader to service record"]] [13:57:38] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host stat1008.eqiad.wmnet with OS bookworm [13:58:24] !log kharlan@deploy1003 kharlan, sukhe: Backport for [[gerrit:1326342|Revert^2 "wmf-config/ProductionServices: set URL for urldownloader to service record"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:59:44] !log kharlan@deploy1003 kharlan, sukhe: Continuing with deployment [14:00:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1400) [14:00:14] still going with the backports [14:00:25] sukhe: I'll ship your static patch next [14:00:40] kostajh: thanks! [14:01:14] (03CR) 10Genoveva Galarza: [C:03+2] wikifunctions: Upgrade evaluators from 2026-08-11-201322 to 2026-08-19-122930 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327118 (https://phabricator.wikimedia.org/T396943) (owner: 10Jforrester) [14:03:27] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-08-11-201322 to 2026-08-19-122930 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327118 (https://phabricator.wikimedia.org/T396943) (owner: 10Jforrester) [14:04:01] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1326342|Revert^2 "wmf-config/ProductionServices: set URL for urldownloader to service record"]] (duration: 07m 40s) [14:04:18] !log gengh@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:04:31] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1314962 (https://phabricator.wikimedia.org/T311685) (owner: 10Ssingh) [14:05:21] !log gengh@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:05:22] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet [14:05:36] (03Merged) 10jenkins-bot: static: add new dir bimi/ for BIMI SVG and PEM file [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1314962 (https://phabricator.wikimedia.org/T311685) (owner: 10Ssingh) [14:05:55] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1314962|static: add new dir bimi/ for BIMI SVG and PEM file (T311685)]] [14:05:59] !log gengh@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:06:01] T311685: Add support for Brand Indicators for Message Identification (BIMI) for wiki mail - https://phabricator.wikimedia.org/T311685 [14:06:36] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [14:06:43] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [14:07:12] andre: the revert is in wmf.16, so it should be good to go :D [14:08:00] !log kharlan@deploy1003 kharlan, sukhe: Backport for [[gerrit:1314962|static: add new dir bimi/ for BIMI SVG and PEM file (T311685)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:09:24] !log gengh@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:09:33] sukhe: do you need to verify the static change or should I just sync it? [14:09:35] !log gengh@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:10:14] kostajh: checking quickly [14:11:04] !log gengh@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:11:06] kostajh: please continue, thanks [14:11:41] ok [14:11:44] !log kharlan@deploy1003 kharlan, sukhe: Continuing with deployment [14:11:54] tgr_: over to you after this sync finishes [14:12:37] (03CR) 10Genoveva Galarza: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-08-11-210638 to 2026-08-19-122639 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327119 (https://phabricator.wikimedia.org/T327413) (owner: 10Jforrester) [14:14:58] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-08-11-210638 to 2026-08-19-122639 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327119 (https://phabricator.wikimedia.org/T327413) (owner: 10Jforrester) [14:15:36] !log gengh@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:15:50] (03CR) 10Elukey: [C:03+2] sre.hosts.reimage: pass dhcp_filename_ipxe to http_boot_once [cookbooks] - 10https://gerrit.wikimedia.org/r/1315820 (https://phabricator.wikimedia.org/T394357) (owner: 10Elukey) [14:15:56] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1314962|static: add new dir bimi/ for BIMI SVG and PEM file (T311685)]] (duration: 10m 00s) [14:15:56] !log gengh@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:16:01] T311685: Add support for Brand Indicators for Message Identification (BIMI) for wiki mail - https://phabricator.wikimedia.org/T311685 [14:16:16] (03CR) 10Muehlenhoff: "Well, I _want_ to run these explicity (to test them as they are being migrated away from build2001/bullseye) :-)" [puppet] - 10https://gerrit.wikimedia.org/r/1327103 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [14:16:36] !log upgrade spicerack on cumin1003 and cumin2003 [14:16:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:16:56] !log gengh@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:17:34] !log gengh@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:17:39] !log gengh@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:17:41] !log disable puppet on hosts running BIRD BGP to test merge of patch to systemd healtchcheck service [14:17:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:18:02] (03CR) 10Cathal Mooney: [C:03+2] Anycast-healthchecker: wait until network-online target is reached (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1325881 (https://phabricator.wikimedia.org/T434806) (owner: 10Cathal Mooney) [14:18:19] !log gengh@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:20:02] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tgr@deploy1003 using scap backport" [extensions/OAuth] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327094 (https://phabricator.wikimedia.org/T435167) (owner: 10Gergő Tisza) [14:20:23] (03CR) 10Elukey: [C:03+1] docker::baseimages: Deploy files/directories independent of the systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/1327103 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [14:21:43] jouncebot: nowandnext [14:21:43] For the next 0 hour(s) and 38 minute(s): Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1400) [14:21:43] In 0 hour(s) and 8 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1430) [14:22:36] if nobody minds, I'll start a no-op MW deployment to build new php8.5 images so the real deployments are only slow, not horribly slow [14:23:00] Raine: I'm deploying a fix for the traing blocker [14:23:05] shouldn't take too long [14:23:11] (03Merged) 10jenkins-bot: SpecialMWOAuthListConsumers: Handle newFromMWUser returning null in addNavigationSubtitle [extensions/OAuth] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327094 (https://phabricator.wikimedia.org/T435167) (owner: 10Gergő Tisza) [14:23:24] oh, okay [14:23:30] thanks tgr_ [14:23:32] !log tgr@deploy1003 Started scap sync-world: Backport for [[gerrit:1327094|SpecialMWOAuthListConsumers: Handle newFromMWUser returning null in addNavigationSubtitle (T435167)]] [14:23:37] T435167: TypeError: MediaWiki\Extension\OAuth\Frontend\SpecialPages\SpecialMWOAuthListConsumers::updateLink(): Argument #2 ($user) must be of type MediaWiki\Extension\OAuth\Entity\UserEntity, null given, called in SpecialMWOAuthListConsumers.php on line 341 - https://phabricator.wikimedia.org/T435167 [14:23:56] !log cmooney@cumin1003 START - Cookbook sre.hosts.reboot-single for host durum3005.esams.wmnet [14:25:27] (03PS1) 10Urbanecm: Revert^2 "Migrate database access to virtual domains" [extensions/Echo] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327123 (https://phabricator.wikimedia.org/T435305) [14:25:38] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie [14:26:02] PROBLEM - Host krb1002 is DOWN: PING CRITICAL - Packet loss = 100% [14:26:17] (03PS1) 10Urbanecm: Pass the mapped domain of virtual-echo-shared to the push NameTableStores [extensions/Echo] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327124 (https://phabricator.wikimedia.org/T435305) [14:28:07] !log cmooney@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host durum3005.esams.wmnet [14:28:40] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [14:28:48] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [14:30:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1400) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1430) [14:34:44] (03PS1) 10Arlolra: Parsoid DataAccess: convert Parsoid fragment markers to/from strip tags [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327128 (https://phabricator.wikimedia.org/T432547) [14:36:55] !log tgr@deploy1003 tgr: Backport for [[gerrit:1327094|SpecialMWOAuthListConsumers: Handle newFromMWUser returning null in addNavigationSubtitle (T435167)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:37:00] T435167: TypeError: MediaWiki\Extension\OAuth\Frontend\SpecialPages\SpecialMWOAuthListConsumers::updateLink(): Argument #2 ($user) must be of type MediaWiki\Extension\OAuth\Entity\UserEntity, null given, called in SpecialMWOAuthListConsumers.php on line 341 - https://phabricator.wikimedia.org/T435167 [14:38:40] !log tgr@deploy1003 tgr: Continuing with deployment [14:39:39] (03CR) 10Ottomata: stream: pageview-trending-relative (staging) (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327116 (https://phabricator.wikimedia.org/T435291) (owner: 10JavierMonton) [14:39:49] (03CR) 10Ottomata: [C:03+1] "One comment, but +1 otherwise. Merge at will :)" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327116 (https://phabricator.wikimedia.org/T435291) (owner: 10JavierMonton) [14:41:14] !log fceratto@cumin1003 START - Cookbook sre.dns.netbox [14:42:11] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage [14:42:19] !log cmooney@cumin1003 START - Cookbook sre.hosts.reboot-single for host durum1001.eqiad.wmnet [14:42:58] !log tgr@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327094|SpecialMWOAuthListConsumers: Handle newFromMWUser returning null in addNavigationSubtitle (T435167)]] (duration: 19m 25s) [14:43:13] T435167: TypeError: MediaWiki\Extension\OAuth\Frontend\SpecialPages\SpecialMWOAuthListConsumers::updateLink(): Argument #2 ($user) must be of type MediaWiki\Extension\OAuth\Entity\UserEntity, null given, called in SpecialMWOAuthListConsumers.php on line 341 - https://phabricator.wikimedia.org/T435167 [14:43:33] !log btullis@cumin1003 START - Cookbook sre.hosts.reimage for host an-worker1147.eqiad.wmnet with OS bookworm [14:43:44] (03PS3) 10JavierMonton: stream: pageview-trending-relative (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327116 (https://phabricator.wikimedia.org/T435291) [14:43:54] (03CR) 10JavierMonton: stream: pageview-trending-relative (staging) (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327116 (https://phabricator.wikimedia.org/T435291) (owner: 10JavierMonton) [14:43:58] Raine: I guess that did the rebuild, but if you still need to deploy, over to you [14:46:09] !log fceratto@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Delete db2902 ipv6 addr - fceratto@cumin1003" [14:46:10] FIRING: [6x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:46:13] !log fceratto@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Delete db2902 ipv6 addr - fceratto@cumin1003" [14:46:14] !log fceratto@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:46:22] (03CR) 10JavierMonton: [C:03+2] stream: pageview-trending-relative (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327116 (https://phabricator.wikimedia.org/T435291) (owner: 10JavierMonton) [14:47:02] tgr_: I don't if you caught the pain, apologies for that [14:47:34] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12231680 (10cmooney) >>! In T434806#12224818, @ayounsi wrote: > We should reach out to the BIRD team, even if there is a race condition, BIRD's BFD pr... [14:48:18] !log cmooney@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host durum1001.eqiad.wmnet [14:48:34] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12231685 (10cmooney) Small update. We merged the patch to only start the anycast-healthchecker service (and thus BIRD) after the network-online targe... [14:48:38] (03Merged) 10jenkins-bot: stream: pageview-trending-relative (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327116 (https://phabricator.wikimedia.org/T435291) (owner: 10JavierMonton) [14:48:57] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage [14:49:08] (03PS1) 10Muehlenhoff: Stop using nftables for the new URL downloader trixie nodes [puppet] - 10https://gerrit.wikimedia.org/r/1327131 (https://phabricator.wikimedia.org/T429175) [14:49:24] (03CR) 10CI reject: [V:04-1] Parsoid DataAccess: convert Parsoid fragment markers to/from strip tags [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327128 (https://phabricator.wikimedia.org/T432547) (owner: 10Arlolra) [14:52:08] (03CR) 10Ssingh: [C:03+1] "Thank you, sounds good to me!" [puppet] - 10https://gerrit.wikimedia.org/r/1327131 (https://phabricator.wikimedia.org/T429175) (owner: 10Muehlenhoff) [14:52:17] 10ops-eqiad, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T433348#12231708 (10BTullis) I have managed to get this working using the LOM3 interface. Thanks to a tip from @VRiley-WMF, I got the netmask of the man... [14:55:06] 10ops-eqiad, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T433348#12231733 (10BTullis) 05In progress→03Resolved Created a journal entry in netbox to explain the irregular configuration of using the LOM3... [14:55:15] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [14:55:52] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [14:56:10] FIRING: [5x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:58:13] !log btullis@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1147.eqiad.wmnet with reason: host reimage [14:59:35] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [14:59:42] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [14:59:54] 10ops-codfw, 06SRE, 06DC-Ops: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12231812 (10elukey) I merged the new version of the reimage cookbook, so from the I/F point of view we are good (and we'll hopefully have less Supermicro-related headaches in the... [15:03:13] jouncebot: nowandnext [15:03:13] No deployments scheduled for the next 1 hour(s) and 56 minute(s) [15:03:13] In 1 hour(s) and 56 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1700) [15:03:18] I want to deploy now [15:03:54] Dreamy_Jazz: i want to deploy since morning... [15:04:34] Did you want to go first? [15:04:37] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1147.eqiad.wmnet with reason: host reimage [15:05:26] 10SRE-SLO, 10Citoid, 06Editing-team, 07Sustainability (Incident Followup): Improve monitoring in citoid so that url-downloader failures are detected - https://phabricator.wikimedia.org/T381372#12231904 (10ssingh) urldownloaders [[ https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1326342 | are... [15:05:32] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [15:05:35] Going to assume not? [15:05:39] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [15:05:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [15:05:58] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1327101 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [15:06:00] in a meeting now :D [15:06:04] Ah I see :D [15:06:10] FIRING: [5x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:06:18] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2010.codfw.wmnet with OS trixie [15:06:19] I too am in a meeting, but keeping it in the background :D [15:09:25] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354 (10MoritzMuehlenhoff) 03NEW [15:09:30] (03Merged) 10jenkins-bot: UserRightsNotificationHandler: Use correct database for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327098 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [15:09:32] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12231979 (10MoritzMuehlenhoff) p:05Triage→03High [15:09:32] (03Merged) 10jenkins-bot: UserRightsNotificationHandler: Use correct database for Echo [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.15) - 10https://gerrit.wikimedia.org/r/1327101 (https://phabricator.wikimedia.org/T435263) (owner: 10Dreamy Jazz) [15:10:01] !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1327098|UserRightsNotificationHandler: Use correct database for Echo (T435263)]], [[gerrit:1327101|UserRightsNotificationHandler: Use correct database for Echo (T435263)]] [15:10:06] T435263: Error 1146: Table 'enwiki.echo_notification' doesn't exist in UserRightsNotificationHandler::onUserGroupsChanged - https://phabricator.wikimedia.org/T435263 [15:11:10] RESOLVED: [5x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:12:22] !log dreamyjazz@deploy1003 dreamyjazz: Backport for [[gerrit:1327098|UserRightsNotificationHandler: Use correct database for Echo (T435263)]], [[gerrit:1327101|UserRightsNotificationHandler: Use correct database for Echo (T435263)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:12:59] !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment [15:17:13] !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327098|UserRightsNotificationHandler: Use correct database for Echo (T435263)]], [[gerrit:1327101|UserRightsNotificationHandler: Use correct database for Echo (T435263)]] (duration: 07m 13s) [15:17:19] T435263: Error 1146: Table 'enwiki.echo_notification' doesn't exist in UserRightsNotificationHandler::onUserGroupsChanged - https://phabricator.wikimedia.org/T435263 [15:18:26] jouncebot: nowandnext [15:18:26] No deployments scheduled for the next 1 hour(s) and 41 minute(s) [15:18:27] In 1 hour(s) and 41 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1700) [15:19:03] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Host crashed [15:21:21] (03PS1) 10Ladsgroup: Enable redis lock manager everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327138 (https://phabricator.wikimedia.org/T366938) [15:22:25] FIRING: SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:23:02] (03PS2) 10Ladsgroup: Enable redis lock manager everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327138 (https://phabricator.wikimedia.org/T366938) [15:23:32] Amir1: can i squeeze in the echo stuff before? [15:23:48] urbanecm: I'll be quick [15:23:51] this is mw config patch [15:23:53] +2 it [15:23:56] ok [15:23:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327138 (https://phabricator.wikimedia.org/T366938) (owner: 10Ladsgroup) [15:24:06] i mean i have one config change too, but... [15:24:06] until I deploy, it'll merge [15:24:16] i can merge the backports [15:24:24] (03CR) 10Urbanecm: [C:03+2] Revert^2 "Migrate database access to virtual domains" [extensions/Echo] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327123 (https://phabricator.wikimedia.org/T435305) (owner: 10Urbanecm) [15:24:26] (03CR) 10Urbanecm: [C:03+2] Pass the mapped domain of virtual-echo-shared to the push NameTableStores [extensions/Echo] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327124 (https://phabricator.wikimedia.org/T435305) (owner: 10Urbanecm) [15:24:34] !log jmm@cumin2003 END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host krb1002.eqiad.wmnet [15:25:09] (03Merged) 10jenkins-bot: Enable redis lock manager everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327138 (https://phabricator.wikimedia.org/T366938) (owner: 10Ladsgroup) [15:25:33] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1327138|Enable redis lock manager everywhere (T366938)]] [15:25:37] T366938: Reduce relying on database locks - https://phabricator.wikimedia.org/T366938 [15:25:45] (03CR) 10AOkoth: [C:03+2] phabricator: fix php version in restart script [puppet] - 10https://gerrit.wikimedia.org/r/1326292 (https://phabricator.wikimedia.org/T433988) (owner: 10AOkoth) [15:28:00] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1327138|Enable redis lock manager everywhere (T366938)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:28:45] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1147.eqiad.wmnet with OS bookworm [15:29:53] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [15:30:08] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations, 13Patch-For-Review: Move Docker images under the /v2/releng prefix to S3 - https://phabricator.wikimedia.org/T432829#12232243 (10dancy) >>! In T432829#12231164, @elukey wrote: > While talking with @dancy I got that /v2/dev.* is another prefix that... [15:30:20] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [15:30:57] (03PS1) 10Cathal Mooney: Remove include for old subnet used on ulsfo<->eqsin link [dns] - 10https://gerrit.wikimedia.org/r/1327140 [15:31:41] (03CR) 10AOkoth: [C:03+2] mariadb: add grants for phab1005 [puppet] - 10https://gerrit.wikimedia.org/r/1321581 (https://phabricator.wikimedia.org/T377889) (owner: 10AOkoth) [15:31:51] (03CR) 10Ssingh: [C:03+1] Remove include for old subnet used on ulsfo<->eqsin link [dns] - 10https://gerrit.wikimedia.org/r/1327140 (owner: 10Cathal Mooney) [15:31:55] (03CR) 10CI reject: [V:04-1] Remove include for old subnet used on ulsfo<->eqsin link [dns] - 10https://gerrit.wikimedia.org/r/1327140 (owner: 10Cathal Mooney) [15:33:27] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqsin - cmooney@cumin1003" [15:33:33] (03PS2) 10Cathal Mooney: Remove include for old subnet used on ulsfo<->eqsin connection [dns] - 10https://gerrit.wikimedia.org/r/1327140 [15:34:09] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327138|Enable redis lock manager everywhere (T366938)]] (duration: 08m 36s) [15:34:12] urbanecm: I am done :D [15:34:16] T366938: Reduce relying on database locks - https://phabricator.wikimedia.org/T366938 [15:35:23] (03CR) 10Ahmon Dancy: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1323829 (https://phabricator.wikimedia.org/T428972) (owner: 10BryanDavis) [15:35:51] (03CR) 10Ahmon Dancy: [C:03+1] "Good idea." [puppet] - 10https://gerrit.wikimedia.org/r/1326939 (owner: 10BryanDavis) [15:36:00] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqsin - cmooney@cumin1003" [15:36:00] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:36:08] (03CR) 10Cathal Mooney: [C:03+2] Remove include for old subnet used on ulsfo<->eqsin connection [dns] - 10https://gerrit.wikimedia.org/r/1327140 (owner: 10Cathal Mooney) [15:36:26] !log cmooney@dns3003 START - running authdns-update [15:37:08] (03PS1) 10AOkoth: wmnet: phabricator -> phab1005.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/1327141 (https://phabricator.wikimedia.org/T435087) [15:38:29] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): eqiad row A&B host migration details request for Data Platform - https://phabricator.wikimedia.org/T432645#12232329 (10BTullis) a:05BTullis→03RobH Reassigning. [15:38:44] !log cmooney@dns3003 END - running authdns-update [15:39:00] (03Merged) 10jenkins-bot: Revert^2 "Migrate database access to virtual domains" [extensions/Echo] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327123 (https://phabricator.wikimedia.org/T435305) (owner: 10Urbanecm) [15:39:03] (03Merged) 10jenkins-bot: Pass the mapped domain of virtual-echo-shared to the push NameTableStores [extensions/Echo] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327124 (https://phabricator.wikimedia.org/T435305) (owner: 10Urbanecm) [15:40:09] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): eqiad row A&B host migration details request for Data Platform - https://phabricator.wikimedia.org/T432645#12232357 (10RobH) 05Open→03Stalled Thanks! We're set to start this migration in mid-sept so will stall this task until... [15:42:06] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [15:42:17] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [15:43:21] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [15:44:25] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [15:44:51] !log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [15:45:09] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003" [15:45:18] (03PS1) 10Cathal Mooney: remove include for IPv6 net previously used on cr1-codfw<->cr4-ulsfo [dns] - 10https://gerrit.wikimedia.org/r/1327143 [15:48:13] cmooney@cumin1003 netbox (PID 1580858) is awaiting input [15:48:24] (03PS1) 10AOkoth: hiera: make phab1005 phabricator_active_server [puppet] - 10https://gerrit.wikimedia.org/r/1327144 (https://phabricator.wikimedia.org/T435087) [15:48:24] (03CR) 10Eevans: [C:03+2] sessionstore: Upgrade to Cassandra 5.0.8 (canary) [puppet] - 10https://gerrit.wikimedia.org/r/1326901 (https://phabricator.wikimedia.org/T435154) (owner: 10Eevans) [15:48:54] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003" [15:48:54] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:48:59] (03CR) 10Cathal Mooney: [C:03+2] remove include for IPv6 net previously used on cr1-codfw<->cr4-ulsfo [dns] - 10https://gerrit.wikimedia.org/r/1327143 (owner: 10Cathal Mooney) [15:49:10] !log cmooney@dns3003 START - running authdns-update [15:51:33] !log cmooney@dns3003 END - running authdns-update [15:52:25] !log beginning sessionstore Cassandra/JVM upgrade — T432944 [15:52:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:31] T432944: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944 [15:53:12] !log beginning sessionstore Cassandra/JVM upgrade — T435154 [15:53:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:53:17] T435154: Upgrade sessionstore cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T435154 [15:54:14] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore2004.codfw.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 [15:56:23] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore2004.codfw.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 [15:58:30] (03CR) 10AOkoth: "https://puppet-compiler.wmflabs.org/output/1327144/9279/" [puppet] - 10https://gerrit.wikimedia.org/r/1327144 (https://phabricator.wikimedia.org/T435087) (owner: 10AOkoth) [15:59:35] (03PS1) 10Elukey: docker_registry: route /v2/dev/.* to the releng S3 bucket [puppet] - 10https://gerrit.wikimedia.org/r/1327146 (https://phabricator.wikimedia.org/T432829) [15:59:37] (03CR) 10Kevin Bazira: [C:03+1] "> There is no Phabricator task for this task I think." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327001 (owner: 10Gkyziridis) [16:01:19] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore1004.eqiad.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 [16:01:32] (03CR) 10Alex Paskulin: [C:03+1] "That's correct. There is no task, but let me know if you'd like me to open one" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327001 (owner: 10Gkyziridis) [16:01:49] T435154: Upgrade sessionstore cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T435154 [16:03:28] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore1004.eqiad.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 [16:05:59] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [16:06:01] (03CR) 10Eevans: [C:03+2] sessionstore: Upgrade to Cassandra 5.0.8 (complete) [puppet] - 10https://gerrit.wikimedia.org/r/1326904 (https://phabricator.wikimedia.org/T435154) (owner: 10Eevans) [16:06:41] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1327123|Revert^2 "Migrate database access to virtual domains" (T435305)]], [[gerrit:1327124|Pass the mapped domain of virtual-echo-shared to the push NameTableStores (T435305)]] [16:06:45] !log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication [16:06:47] T435305: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'itwiki.echo_push_provider' doesn't exist - https://phabricator.wikimedia.org/T435305 [16:06:57] !log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) [16:08:57] !log urbanecm@deploy1003 urbanecm: Backport for [[gerrit:1327123|Revert^2 "Migrate database access to virtual domains" (T435305)]], [[gerrit:1327124|Pass the mapped domain of virtual-echo-shared to the push NameTableStores (T435305)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:10:08] !log urbanecm@deploy1003 urbanecm: Continuing with deployment [16:11:18] (03PS4) 10Urbanecm: Echo: Start using virtual domains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326881 (https://phabricator.wikimedia.org/T380385) [16:11:22] cmooney@cumin1003 netbox (PID 1598255) is awaiting input [16:11:24] (03CR) 10Urbanecm: [C:03+2] Echo: Start using virtual domains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326881 (https://phabricator.wikimedia.org/T380385) (owner: 10Urbanecm) [16:11:41] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003" [16:12:19] (03Merged) 10jenkins-bot: Echo: Start using virtual domains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1326881 (https://phabricator.wikimedia.org/T380385) (owner: 10Urbanecm) [16:13:16] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore[2005-2006].codfw.wmnet,sessionstore[1005-1006].eqiad.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 [16:13:21] T435154: Upgrade sessionstore cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T435154 [16:14:23] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327123|Revert^2 "Migrate database access to virtual domains" (T435305)]], [[gerrit:1327124|Pass the mapped domain of virtual-echo-shared to the push NameTableStores (T435305)]] (duration: 07m 42s) [16:14:28] T435305: Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'itwiki.echo_push_provider' doesn't exist - https://phabricator.wikimedia.org/T435305 [16:14:45] cmooney@cumin1003 netbox (PID 1598255) is awaiting input [16:14:47] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1326881|Echo: Start using virtual domains (T380385)]] [16:14:52] T380385: Migrate Echo to virtual domains - https://phabricator.wikimedia.org/T380385 [16:15:01] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003" [16:15:01] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:15:08] (03PS1) 10Cathal Mooney: Delete include that was used on esams<->eqiad colt link [dns] - 10https://gerrit.wikimedia.org/r/1327148 [16:19:11] !log urbanecm@deploy1003 urbanecm: Backport for [[gerrit:1326881|Echo: Start using virtual domains (T380385)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:20:59] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore[2005-2006].codfw.wmnet,sessionstore[1005-1006].eqiad.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003 [16:21:03] T435154: Upgrade sessionstore cluster to Cassandra 5.0.8 & JDK 17 - https://phabricator.wikimedia.org/T435154 [16:21:47] !log Completed sessionstore Cassandra/JVM upgrade — T435154 [16:21:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:23:07] !log urbanecm@deploy1003 urbanecm: Continuing with deployment [16:27:59] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1326881|Echo: Start using virtual domains (T380385)]] (duration: 13m 12s) [16:28:04] T380385: Migrate Echo to virtual domains - https://phabricator.wikimedia.org/T380385 [16:28:37] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:29:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.17% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:33:09] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12232694 (10Jhancock.wm) [16:33:20] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12232695 (10Jhancock.wm) eqdfw confirmed [16:35:26] 06SRE, 06Privacy Engineering, 06Security-Team, 10Toolhub, and 3 others: prometheus and python metrics exposed to the world - https://phabricator.wikimedia.org/T309703#12232719 (10sbassett) [16:39:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:40:21] (03CR) 10RLazarus: "Sorry! Yeah, exactly that, except that I've generally had to say" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325884 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [16:40:52] !log sukhe@cumin1003 START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum and A:durum [16:41:05] !log sudo -i cookbook sre.cdn.roll-upgrade-ats --query 'A:cp-eqiad' --task-id T434478 --reason '9.2.15 upgrade' [16:41:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:41:14] !log sudo -i cookbook sre.cdn.roll-upgrade-ats --query 'A:cp-codfw' --task-id T434478 --reason '9.2.15 upgrade' [16:41:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:41:20] !log cdobbins@cumin1003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.15 upgrade (T434478) [16:41:26] !log cdobbins@cumin1003 START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.15 upgrade (T434478) [16:41:45] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.admin rebooting A:liberica-esams and A:liberica [16:43:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.17% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:45:10] FIRING: [2x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:103:10:192:32:58 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:46:58] (03CR) 10Scott French: [C:03+1] docker_registry: route /v2/dev/.* to the releng S3 bucket [puppet] - 10https://gerrit.wikimedia.org/r/1327146 (https://phabricator.wikimedia.org/T432829) (owner: 10Elukey) [16:48:12] (03PS1) 10CDobbins: hieradata: use new cfg flag for pdns v5 [puppet] - 10https://gerrit.wikimedia.org/r/1327151 (https://phabricator.wikimedia.org/T401832) [16:48:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.21% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:49:32] (03CR) 10Ssingh: "If we put in the config overrides at once, we will have to reimage them altogether. So the overrides have to be put in place one by one." [puppet] - 10https://gerrit.wikimedia.org/r/1327151 (https://phabricator.wikimedia.org/T401832) (owner: 10CDobbins) [16:50:10] FIRING: [4x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:51:04] (03CR) 10CDobbins: "I hadn't considered that. I'll amend it so only dns1004 has the new flag." [puppet] - 10https://gerrit.wikimedia.org/r/1327151 (https://phabricator.wikimedia.org/T401832) (owner: 10CDobbins) [16:53:01] (03PS2) 10GergesShamon: [arwiki] Enable restricted user page editing and grant edit permissions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325878 (https://phabricator.wikimedia.org/T434878) [16:53:08] (03PS3) 10Ssingh: wikimedia.org: add TXT record for BIMI [dns] - 10https://gerrit.wikimedia.org/r/1314966 (https://phabricator.wikimedia.org/T311685) [16:53:13] (03CR) 10GergesShamon: [arwiki] Enable restricted user page editing and grant edit permissions (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325878 (https://phabricator.wikimedia.org/T434878) (owner: 10GergesShamon) [16:53:18] (03CR) 10CI reject: [V:04-1] wikimedia.org: add TXT record for BIMI [dns] - 10https://gerrit.wikimedia.org/r/1314966 (https://phabricator.wikimedia.org/T311685) (owner: 10Ssingh) [16:53:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 19 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325878 (https://phabricator.wikimedia.org/T434878) (owner: 10GergesShamon) [16:53:36] !log sukhe@cumin1003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-esams and A:liberica [16:54:33] (03PS2) 10CDobbins: hieradata: use new cfg flag for pdns v5 [puppet] - 10https://gerrit.wikimedia.org/r/1327151 (https://phabricator.wikimedia.org/T401832) [16:55:40] (03PS4) 10Ssingh: wikimedia.org: add TXT record for BIMI [dns] - 10https://gerrit.wikimedia.org/r/1314966 (https://phabricator.wikimedia.org/T311685) [16:56:37] (03CR) 10CI reject: [V:04-1] wikimedia.org: add TXT record for BIMI [dns] - 10https://gerrit.wikimedia.org/r/1314966 (https://phabricator.wikimedia.org/T311685) (owner: 10Ssingh) [16:57:59] (03CR) 10Ssingh: [C:03+2] Delete include that was used on esams<->eqiad colt link [dns] - 10https://gerrit.wikimedia.org/r/1327148 (owner: 10Cathal Mooney) [16:58:11] !log sukhe@dns1004 START - running authdns-update [16:58:27] (03PS5) 10Ssingh: wikimedia.org: add TXT record for BIMI [dns] - 10https://gerrit.wikimedia.org/r/1314966 (https://phabricator.wikimedia.org/T311685) [16:59:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:00:04] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1700) [17:00:08] (03PS1) 10CDobbins: hieradata: add cfg flag for pdns v5 for dns1005 [puppet] - 10https://gerrit.wikimedia.org/r/1327152 (https://phabricator.wikimedia.org/T401832) [17:00:20] !log sukhe@dns1004 END - running authdns-update [17:01:10] (03PS3) 10CDobbins: hieradata: use cfg for pdns v5 on dns1004 [puppet] - 10https://gerrit.wikimedia.org/r/1327151 (https://phabricator.wikimedia.org/T401832) [17:02:55] Hey all - have a UBN sec issue we’d like to get patched sooner than later. Would that conflict with the SRE window right now? [17:04:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:05:07] (03Abandoned) 10CDobbins: hieradata: add cfg flag for pdns v5 for dns1005 [puppet] - 10https://gerrit.wikimedia.org/r/1327152 (https://phabricator.wikimedia.org/T401832) (owner: 10CDobbins) [17:05:10] FIRING: [5x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:09:49] sbassett: go for it - I don't think we have anything planned. [17:10:10] FIRING: [6x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:10:12] sbassett: if you could ping me when you're done, that would be swell (I might try to get something non-urgent out later in the window). [17:15:31] swfrench-wmf: thanks - should be quick-ish [17:15:32] (03PS1) 10CDobbins: hieradata: add pdns v5 flag for dns2005 [puppet] - 10https://gerrit.wikimedia.org/r/1327154 (https://phabricator.wikimedia.org/T401832) [17:17:02] (03PS2) 10CDobbins: hieradata: add pdns v5 flag for dns2004 [puppet] - 10https://gerrit.wikimedia.org/r/1327154 (https://phabricator.wikimedia.org/T401832) [17:18:12] (03PS1) 10CDobbins: hieradata: add pdns v5 flag for dns2005 [puppet] - 10https://gerrit.wikimedia.org/r/1327155 (https://phabricator.wikimedia.org/T401832) [17:19:37] (03PS1) 10CDobbins: hieradata: use pdns v5 cfg flag on dns5004 [puppet] - 10https://gerrit.wikimedia.org/r/1327156 (https://phabricator.wikimedia.org/T401832) [17:22:01] !log Deployed security fix for T435210 (wmf.15) [17:22:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:25:20] (03PS1) 10CDobbins: hieradata: use pdns v5 cfg flag on dns6002 [puppet] - 10https://gerrit.wikimedia.org/r/1327157 (https://phabricator.wikimedia.org/T401832) [17:26:45] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [17:29:43] !log Deployed security fix for T435210 (wmf.16) [17:29:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:29:53] swfrench-wmf: all done [17:30:05] sbassett: great, thanks! [17:32:02] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12233237 (10BTullis) [17:32:03] cmooney@cumin1003 netbox (PID 1657057) is awaiting input [17:32:21] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on GTT VPLS - cmooney@cumin1003" [17:32:28] (03PS1) 10Cathal Mooney: Remove include statements for [dns] - 10https://gerrit.wikimedia.org/r/1327158 [17:32:29] (03PS1) 10Cathal Mooney: Remove include statements for some GTT VPLS endpoints [dns] - 10https://gerrit.wikimedia.org/r/1327159 [17:33:27] (03CR) 10CI reject: [V:04-1] Remove include statements for [dns] - 10https://gerrit.wikimedia.org/r/1327158 (owner: 10Cathal Mooney) [17:33:51] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12233251 (10BTullis) Hi, I found this task because we got a sudden performance degradation of Hadoop, Spark, Presto etc. I wonder if we should swap the order in `kerberos_kdc_servers_to_clients` in `h... [17:34:37] (03Abandoned) 10Cathal Mooney: Remove include statements for [dns] - 10https://gerrit.wikimedia.org/r/1327158 (owner: 10Cathal Mooney) [17:34:59] (03CR) 10Scott French: [C:03+2] httpbb: Add test suite for testwiki [puppet] - 10https://gerrit.wikimedia.org/r/1323829 (https://phabricator.wikimedia.org/T428972) (owner: 10BryanDavis) [17:35:01] (03Abandoned) 10Cathal Mooney: Remove include statements for some GTT VPLS endpoints [dns] - 10https://gerrit.wikimedia.org/r/1327159 (owner: 10Cathal Mooney) [17:35:25] cmooney@cumin1003 netbox (PID 1657057) is awaiting input [17:36:44] (03PS3) 10BryanDavis: httpbb: sort managed directory list [puppet] - 10https://gerrit.wikimedia.org/r/1326939 [17:37:40] (03PS1) 10Muehlenhoff: Only pass krb2002 to Kerberos clients [puppet] - 10https://gerrit.wikimedia.org/r/1327160 (https://phabricator.wikimedia.org/T435354) [17:37:45] !log urbanecm@deploy1003 mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --verbose --scoreLessThan=0.7 --exceptDatasetChecksums=T434319-enwiki-models.txt # T434319 [17:37:50] (03PS1) 10Cathal Mooney: Remove include statements for some GTT VPLS endpoints #2 [dns] - 10https://gerrit.wikimedia.org/r/1327161 [17:37:51] T434319: Revalidate Add Link suggestions on enwiki - https://phabricator.wikimedia.org/T434319 [17:37:57] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on GTT VPLS - cmooney@cumin1003" [17:37:57] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:38:09] (03CR) 10Bking: [C:03+1] Only pass krb2002 to Kerberos clients [puppet] - 10https://gerrit.wikimedia.org/r/1327160 (https://phabricator.wikimedia.org/T435354) (owner: 10Muehlenhoff) [17:38:34] swfrench-wmf: I have a new release of scap to deploy when you're done. [17:39:16] (03CR) 10Muehlenhoff: [C:03+2] Only pass krb2002 to Kerberos clients [puppet] - 10https://gerrit.wikimedia.org/r/1327160 (https://phabricator.wikimedia.org/T435354) (owner: 10Muehlenhoff) [17:39:22] dancy: if you're ready to go, feel free to do so. I'm waiting for puppet-agent on the deployment host, which will take a little while :) [17:40:18] (03CR) 10Cathal Mooney: [C:03+2] Remove include statements for some GTT VPLS endpoints #2 [dns] - 10https://gerrit.wikimedia.org/r/1327161 (owner: 10Cathal Mooney) [17:40:30] !log dancy@deploy1003 Installing scap version "4.283.0" for 3 host(s) [17:40:35] !log cmooney@dns3003 START - running authdns-update [17:41:47] !log sukhe@cumin1003 END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum and A:durum [17:41:57] (03PS1) 10Krinkle: Article: Split subjectpageheader by model and disable for wikitext [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327162 [17:42:21] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 19 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327162 (owner: 10Krinkle) [17:42:25] !log dancy@deploy1003 Installation of scap version "4.283.0" completed for 3 hosts [17:42:34] swfrench-wmf: done [17:44:48] dancy: ack, thanks! so, it looks like the `bootstrap_scap_master` Exec resource may have somehow collided with your deployment. I'm going to be re-running puppet-agent again shortly, which should also presumably confirm whether that's just a transient issue. [17:45:05] OK sounds good. [17:45:07] (03CR) 10Scott French: [C:03+2] httpbb: sort managed directory list [puppet] - 10https://gerrit.wikimedia.org/r/1326939 (owner: 10BryanDavis) [17:46:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-eqdfw and cr2-esams (208.80.153.216) - group Confed_esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [17:47:48] (03CR) 10Majavah: [C:03+2] Puppet 8: Replace legacy facts beta module [puppet] - 10https://gerrit.wikimedia.org/r/1326898 (https://phabricator.wikimedia.org/T435225) (owner: 10JHathaway) [17:50:42] !log cmooney@dns3003 END - running authdns-update [17:55:10] FIRING: [6x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:56:39] RESOLVED: [3x] CoreBGPDown: Core BGP session down between cr2-eqdfw and cr2-esams (208.80.153.216) - group Confed_esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [17:56:51] !log cdobbins@cumin1003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.15 upgrade (T434478) [17:57:03] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [puppet] - 10https://gerrit.wikimedia.org/r/1327083 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [18:00:05] andre and brennen: #bothumor My software never has bugs. It just develops random features. Rise for MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T1800). [18:00:10] RESOLVED: [6x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:00:15] jouncebot: no thanks [18:01:30] :) [18:02:20] I have a deploy then :D [18:02:52] * andre passes the magic wand to turn servers into experience to Amir1 [18:03:32] !log cdobbins@cumin1003 END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.15 upgrade (T434478) [18:04:18] xD [18:04:28] 10ops-eqiad, 06SRE, 06DC-Ops, 13Patch-For-Review: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12233397 (10bking) Per linked ticket, the above change seems to have resolved the issue with slow access to HDFS. [18:06:16] (03PS4) 10Ladsgroup: Allow setting a separate thumbUrl in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319920 (https://phabricator.wikimedia.org/T427465) [18:07:49] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319920 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [18:08:56] (03Merged) 10jenkins-bot: Allow setting a separate thumbUrl in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319920 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [18:09:15] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1319920|Allow setting a separate thumbUrl in production (T427465)]] [18:09:29] T427465: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465 [18:23:49] (03CR) 10Ssingh: [C:03+2] wikimedia.org: add TXT record for BIMI [dns] - 10https://gerrit.wikimedia.org/r/1314966 (https://phabricator.wikimedia.org/T311685) (owner: 10Ssingh) [18:24:01] !log sukhe@dns1004 START - running authdns-update [18:25:13] (03PS1) 10Ladsgroup: Fix wmgThumbUrl config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327167 (https://phabricator.wikimedia.org/T427465) [18:26:10] !log sukhe@dns1004 END - running authdns-update [18:27:05] (03CR) 10Ladsgroup: [C:03+2] Fix wmgThumbUrl config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327167 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [18:27:53] (03PS1) 10Scott French: kubernetes: Enable testwiki httpbb test suite for pretrain [puppet] - 10https://gerrit.wikimedia.org/r/1327168 (https://phabricator.wikimedia.org/T428972) [18:28:02] (03Merged) 10jenkins-bot: Fix wmgThumbUrl config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327167 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [18:28:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327167 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [18:28:40] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1319920|Allow setting a separate thumbUrl in production (T427465)]], [[gerrit:1327167|Fix wmgThumbUrl config (T427465)]] [18:28:45] T427465: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465 [18:30:57] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1319920|Allow setting a separate thumbUrl in production (T427465)]], [[gerrit:1327167|Fix wmgThumbUrl config (T427465)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:31:34] (03PS1) 10Krinkle: Make uppercase greek letters normal (non-italic) font [extensions/Math] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327169 (https://phabricator.wikimedia.org/T434686) [18:33:22] (03PS1) 10Scott French: mw-pretrain: Relax x-wikimedia-debug match regex [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327164 (https://phabricator.wikimedia.org/T427668) [18:33:25] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12233524 (10VRiley-WMF) Looking into this now. [18:41:11] (03PS7) 10Scott French: trafficserver: Support testwiki pretrain routing in XWD [puppet] - 10https://gerrit.wikimedia.org/r/1304189 (https://phabricator.wikimedia.org/T427666) [18:41:12] (03PS3) 10Scott French: hieradata: Divert testwiki traffic to mw-pretrain at ATS [puppet] - 10https://gerrit.wikimedia.org/r/1326930 (https://phabricator.wikimedia.org/T427666) [18:43:19] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [18:47:33] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1319920|Allow setting a separate thumbUrl in production (T427465)]], [[gerrit:1327167|Fix wmgThumbUrl config (T427465)]] (duration: 18m 53s) [18:47:39] T427465: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465 [18:49:05] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T435240#12233604 (10VRiley-WMF) a:03VRiley-WMF [18:49:24] (03CR) 10Ladsgroup: Varnish: Reject non-thumb requests to thumbs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [18:49:43] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T435240#12233606 (10VRiley-WMF) 05Open→03Resolved This was a duplicate ticket [18:51:22] 10ops-eqiad, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Physical connection testing - https://phabricator.wikimedia.org/T435390 (10VRiley-WMF) 03NEW [18:51:51] 10ops-eqiad, 06DC-Ops, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Physical connection testing - https://phabricator.wikimedia.org/T435390#12233626 (10VRiley-WMF) 05Open→03Resolved This is a record to track the physical work that was completed on this machine. This has been resolved. [18:53:32] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12233630 (10VRiley-WMF) [18:55:04] (03CR) 10Ladsgroup: Varnish: Reject non-thumb requests to thumbs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1324264 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [18:56:48] (03CR) 10Arlolra: "recheck" [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327128 (https://phabricator.wikimedia.org/T432547) (owner: 10Arlolra) [19:00:02] (03PS1) 10Ladsgroup: varnish: Loosen the regex for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1327175 (https://phabricator.wikimedia.org/T427465) [19:03:51] (03CR) 10Ladsgroup: varnish: Loosen the regex for thumbnails (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1327175 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [19:05:44] (03CR) 10BCornwall: [C:03+1] varnish: Loosen the regex for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1327175 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [19:06:31] (03CR) 10BCornwall: [C:03+2] varnish: Loosen the regex for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1327175 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [19:13:24] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12233738 (10VRiley-WMF) Upon logging into the unit, there doesn't seem to be any hardware issues. However, I did gather logs and submitted a ticket to Supermicro. The ticket number is Case #CS-00195895... [19:16:52] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1152.eqiad.wmnet - https://phabricator.wikimedia.org/T434480#12233771 (10VRiley-WMF) [19:17:32] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1152.eqiad.wmnet - https://phabricator.wikimedia.org/T434480#12233781 (10VRiley-WMF) a:03VRiley-WMF [19:17:34] (03PS1) 10Krinkle: Skin: Avoid DB lookup for pagecategorieslink message [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327176 (https://phabricator.wikimedia.org/T347123) [19:17:49] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1152.eqiad.wmnet - https://phabricator.wikimedia.org/T434480#12233785 (10VRiley-WMF) 05Open→03Resolved completed [19:18:25] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1153.eqiad.wmnet - https://phabricator.wikimedia.org/T434638#12233788 (10VRiley-WMF) a:03VRiley-WMF [19:21:52] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1153.eqiad.wmnet - https://phabricator.wikimedia.org/T434638#12233800 (10VRiley-WMF) [19:22:07] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1153.eqiad.wmnet - https://phabricator.wikimedia.org/T434638#12233802 (10VRiley-WMF) 05Open→03Resolved This is completed [19:22:40] FIRING: SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:24:36] (03PS1) 10Catrope: AccountRecovery: Notify the email address of the on file of the request [extensions/EmailAuth] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327178 (https://phabricator.wikimedia.org/T425799) [19:24:57] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 19 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/EmailAuth] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327178 (https://phabricator.wikimedia.org/T425799) (owner: 10Catrope) [19:27:42] (03CR) 10Scott French: "Thanks in advance for the review!" [puppet] - 10https://gerrit.wikimedia.org/r/1304189 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [19:28:23] (03PS1) 10Scott French: hieradata: Tune mw-jobrunner mesh listener before adoption [puppet] - 10https://gerrit.wikimedia.org/r/1325935 (https://phabricator.wikimedia.org/T434925) [19:29:16] (03PS2) 10Scott French: hieradata: Tune mw-jobrunner mesh listener before adoption [puppet] - 10https://gerrit.wikimedia.org/r/1325935 (https://phabricator.wikimedia.org/T434925) [19:30:06] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1325935 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [19:30:33] (03PS1) 10Bking: wdqs: Absent unneeded monitors [puppet] - 10https://gerrit.wikimedia.org/r/1327181 (https://phabricator.wikimedia.org/T358029) [19:31:10] (03CR) 10CI reject: [V:04-1] wdqs: Absent unneeded monitors [puppet] - 10https://gerrit.wikimedia.org/r/1327181 (https://phabricator.wikimedia.org/T358029) (owner: 10Bking) [19:31:22] (03CR) 10Arlolra: "recheck" [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327128 (https://phabricator.wikimedia.org/T432547) (owner: 10Arlolra) [19:32:14] (03PS2) 10Bking: wdqs: Absent unneeded monitors [puppet] - 10https://gerrit.wikimedia.org/r/1327181 (https://phabricator.wikimedia.org/T358029) [19:32:48] (03CR) 10CI reject: [V:04-1] wdqs: Absent unneeded monitors [puppet] - 10https://gerrit.wikimedia.org/r/1327181 (https://phabricator.wikimedia.org/T358029) (owner: 10Bking) [19:35:24] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops, and 2 others: db2207 down - https://phabricator.wikimedia.org/T435271#12233861 (10Jhancock.wm) I have some updates and a quick ask. ask first, is it okay for me to do some firmware updates on this server right now? updates: 1) i pulled some info from the db servers... [19:38:24] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops, and 2 others: db2207 down - https://phabricator.wikimedia.org/T435271#12233874 (10Jhancock.wm) also, should the masters be moved off servers that have Intel Xeon Gold 5317? I have a list in a sheet that should be searchable by "DB Crash Report" [19:43:46] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance [19:44:49] (03PS3) 10Bking: wdqs: Absent unneeded monitors [puppet] - 10https://gerrit.wikimedia.org/r/1327181 (https://phabricator.wikimedia.org/T358029) [19:45:25] (03CR) 10CI reject: [V:04-1] wdqs: Absent unneeded monitors [puppet] - 10https://gerrit.wikimedia.org/r/1327181 (https://phabricator.wikimedia.org/T358029) (owner: 10Bking) [19:46:22] (03PS4) 10Bking: wdqs: Absent unneeded monitors [puppet] - 10https://gerrit.wikimedia.org/r/1327181 (https://phabricator.wikimedia.org/T358029) [19:48:50] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops, and 2 others: db2207 down - https://phabricator.wikimedia.org/T435271#12233896 (10CWilliams-WMF) @Jhancock.wm the server was already depooled, so I have extended the downtime by 1 day and stopped MariaDB for you with the normal steps that we do for a reboot: `sh sudo my... [19:48:52] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, August 19 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327128 (https://phabricator.wikimedia.org/T432547) (owner: 10Arlolra) [19:51:39] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1327181 (https://phabricator.wikimedia.org/T358029) (owner: 10Bking) [19:53:01] (03PS1) 10Bking: wdqs: remove unneeded monitors [puppet] - 10https://gerrit.wikimedia.org/r/1327182 (https://phabricator.wikimedia.org/T358029) [19:54:37] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1327182 (https://phabricator.wikimedia.org/T358029) (owner: 10Bking) [19:56:30] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host cassandra-dev2001.codfw.wmnet with OS bookworm [19:56:52] !log eevans@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cassandra-dev2001.codfw.wmnet with OS bookworm [19:59:12] 06SRE, 06Infrastructure-Foundations: Re-IP hosts running Cassandra to per-rack subnets in codfw row A and B. - https://phabricator.wikimedia.org/T354871#12233921 (10Eevans) While attempting to reimage cassandra-dev2001 with `--move-vlan` (I haven't investigated further): `counterexample Too many IPs configure... [19:59:44] !log eevans@cumin1003 START - Cookbook sre.hosts.reimage for host cassandra-dev2001.codfw.wmnet with OS bookworm [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: It is that lovely time of the day again! You are hereby commanded to deploy UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T2000). [20:00:05] _Gerges, Krinkle, RoanKattouw, and arlolra: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:14] <_Gerges> here [20:00:18] (03CR) 10Scott French: "Sending this your way in case you happen to recall anything of note from when you were originally looking at this for T360625. Thanks in a" [puppet] - 10https://gerrit.wikimedia.org/r/1325935 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [20:01:22] o/ [20:03:11] I'm here and I can do the deployment [20:05:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325878 (https://phabricator.wikimedia.org/T434878) (owner: 10GergesShamon) [20:05:48] (03CR) 10Catrope: [C:03+2] Parsoid DataAccess: convert Parsoid fragment markers to/from strip tags [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327128 (https://phabricator.wikimedia.org/T432547) (owner: 10Arlolra) [20:06:13] (03Merged) 10jenkins-bot: [arwiki] Enable restricted user page editing and grant edit permissions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325878 (https://phabricator.wikimedia.org/T434878) (owner: 10GergesShamon) [20:06:34] !log catrope@deploy1003 Started scap sync-world: Backport for [[gerrit:1325878|[arwiki] Enable restricted user page editing and grant edit permissions (T434878)]] [20:06:39] T434878: Enable wgRestrictUserPageEditing on arwiki - https://phabricator.wikimedia.org/T434878 [20:07:17] (03CR) 10Ryan Kemper: [C:03+2] cirrus: remove orphaned frozen writes command [puppet] - 10https://gerrit.wikimedia.org/r/1326393 (https://phabricator.wikimedia.org/T433306) (owner: 10Ryan Kemper) [20:07:44] I'll go a bit later. Don't wait for me. [20:08:56] !log catrope@deploy1003 catrope, gergesshamon: Backport for [[gerrit:1325878|[arwiki] Enable restricted user page editing and grant edit permissions (T434878)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:11:09] !log catrope@deploy1003 catrope, gergesshamon: Continuing with deployment [20:11:26] (03Merged) 10jenkins-bot: Parsoid DataAccess: convert Parsoid fragment markers to/from strip tags [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327128 (https://phabricator.wikimedia.org/T432547) (owner: 10Arlolra) [20:15:21] !log catrope@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325878|[arwiki] Enable restricted user page editing and grant edit permissions (T434878)]] (duration: 08m 46s) [20:15:25] T434878: Enable wgRestrictUserPageEditing on arwiki - https://phabricator.wikimedia.org/T434878 [20:17:16] !log eevans@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage [20:17:18] <_Gerges> I don't see any user group with `editalluserpages` permission in arwiki. Can someone please check?https://ar.wikipedia.org/wiki/%D8%AE%D8%A7%D8%B5:%D8%B3%D9%85%D8%A7%D8%AD%D8%A7%D8%AA_%D8%A7%D9%84%D9%85%D8%AC%D9%85%D9%88%D8%B9%D8%A9_%D8%A7%D9%84%D8%B9%D8%A7%D9%85%D8%A9 [20:19:57] (03CR) 10Ahmon Dancy: [C:03+1] "I tried out the two commands on the deployment server." [puppet] - 10https://gerrit.wikimedia.org/r/1327168 (https://phabricator.wikimedia.org/T428972) (owner: 10Scott French) [20:21:22] Very strange, it worked for me in testing, let me see.. [20:21:48] (03PS2) 10Cwhite: logstash: enable passing authentication headers to dashboards [puppet] - 10https://gerrit.wikimedia.org/r/1326956 (https://phabricator.wikimedia.org/T350516) [20:22:28] _Gerges: Oh that's the global groups page. The right place to look for this is https://ar.wikipedia.org/wiki/%D8%AE%D8%A7%D8%B5:%D8%B9%D8%B1%D8%B6_%D8%B5%D9%84%D8%A7%D8%AD%D9%8A%D8%A7%D8%AA_%D8%A7%D9%84%D9%85%D8%AC%D9%85%D9%88%D8%B9%D8%A7%D8%AA , which does have it [20:23:22] <_Gerges> Sorry, I didn't notice. Thank you. [20:23:23] (03PS1) 10Lerickson: Add egress to the service mesh. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327186 (https://phabricator.wikimedia.org/T433375) [20:23:37] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage [20:24:18] (03PS3) 10Lerickson: WIP: Enable posting to eventgate from the proxy. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326893 (https://phabricator.wikimedia.org/T433375) [20:24:24] !log catrope@deploy1003 Started scap sync-world: Backport for [[gerrit:1327128|Parsoid DataAccess: convert Parsoid fragment markers to/from strip tags (T432547)]] [20:24:30] T432547: "parsoidfragment" visible in rendered articles - https://phabricator.wikimedia.org/T432547 [20:24:58] RoanKattouw: nothing to test for my patch, we just need it to run rt testing later this week [20:25:24] 06SRE, 06Data-Platform-SRE, 06DC-Ops: Enable CPU performance governor on Relforge, Cloudelastic, and Elasticsearch hosts - https://phabricator.wikimedia.org/T386860#12234030 (10bking) 05In progress→03Declined We don't have enough evidence that the performance governor is helping, so we've decided to... [20:26:53] !log catrope@deploy1003 catrope, arlolra: Backport for [[gerrit:1327128|Parsoid DataAccess: convert Parsoid fragment markers to/from strip tags (T432547)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:27:44] !log catrope@deploy1003 catrope, arlolra: Continuing with deployment [20:28:37] FIRING: ProbeDown: Service ganeti3005:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:28:52] (03CR) 10Cwhite: [C:03+2] logstash: enable passing authentication headers to dashboards [puppet] - 10https://gerrit.wikimedia.org/r/1326956 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [20:31:55] !log catrope@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327128|Parsoid DataAccess: convert Parsoid fragment markers to/from strip tags (T432547)]] (duration: 07m 30s) [20:31:59] T432547: "parsoidfragment" visible in rendered articles - https://phabricator.wikimedia.org/T432547 [20:32:33] RoanKattouw: thank you [20:32:34] (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [extensions/EmailAuth] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327178 (https://phabricator.wikimedia.org/T425799) (owner: 10Catrope) [20:33:47] This one is going to take a while (because it includes an i18n message), and then I'll let Krinkle deploy his own patch [20:35:25] (03Merged) 10jenkins-bot: AccountRecovery: Notify the email address of the on file of the request [extensions/EmailAuth] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327178 (https://phabricator.wikimedia.org/T425799) (owner: 10Catrope) [20:35:47] !log catrope@deploy1003 Started scap sync-world: Backport for [[gerrit:1327178|AccountRecovery: Notify the email address of the on file of the request (T425799)]] [20:37:12] (03PS1) 10Bking: cirrussearch: revert to default scaling governor [puppet] - 10https://gerrit.wikimedia.org/r/1327187 (https://phabricator.wikimedia.org/T435400) [20:41:44] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1327187 (https://phabricator.wikimedia.org/T435400) (owner: 10Bking) [20:46:04] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops, and 2 others: db2207 down - https://phabricator.wikimedia.org/T435271#12234103 (10Jhancock.wm) I have added the smaller component firmware upgrades to this one. I'll update as soon as i can about any updates from Dell. [20:50:08] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops, and 2 others: db2207 down - https://phabricator.wikimedia.org/T435271#12234143 (10CWilliams-WMF) The server remains depooled and in downtime, but I have started replication and it should have caught up again shortly. [20:55:35] !log catrope@deploy1003 catrope: Backport for [[gerrit:1327178|AccountRecovery: Notify the email address of the on file of the request (T425799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:57:05] (03CR) 10Bking: "I missed some hosts, marking as WIP until I fix this." [puppet] - 10https://gerrit.wikimedia.org/r/1327187 (https://phabricator.wikimedia.org/T435400) (owner: 10Bking) [20:59:38] (03CR) 10Reedy: AccountRecovery: Adding additional Zendesk fields (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1233349 (https://phabricator.wikimedia.org/T414597) (owner: 10Foks) [21:00:04] Deploy window Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T2100) [21:05:50] (03PS1) 10Bking: cirrussearch: revert to default scaling governor [puppet] - 10https://gerrit.wikimedia.org/r/1327187 (https://phabricator.wikimedia.org/T435400) [21:05:50] (03CR) 10Bking: "I was wrong, I didn't miss any hosts. The problem is that removing the class isn't enough to change the governor. I'll put more details in" [puppet] - 10https://gerrit.wikimedia.org/r/1327187 (https://phabricator.wikimedia.org/T435400) (owner: 10Bking) [21:09:56] !log catrope@deploy1003 catrope: Continuing with deployment [21:10:58] 10ops-eqsin, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin switch migration - 2026-08-26 @ 08:00 UTC - https://phabricator.wikimedia.org/T435406 (10RobH) 03NEW [21:12:32] 10ops-eqsin, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin switch migration - 2026-08-26 @ 08:00 UTC - https://phabricator.wikimedia.org/T435406#12234210 (10RobH) p:05Triage→03High a:03ssingh @ssingh, I wanted to get your sign off on this for #traffic before I fully commit with Jin. We'd... [21:13:59] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2001.codfw.wmnet with OS bookworm [21:17:42] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12234228 (10jasmine_) >>! In T406596#12213935, @Jhancock.wm wrote: > so that first one failed because the firmware upgrade screwed u... [21:22:49] !log catrope@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327178|AccountRecovery: Notify the email address of the on file of the request (T425799)]] (duration: 47m 02s) [21:23:24] wow that took a while [21:23:35] I was half-convined the message got lost somewhere [21:23:50] That'll make you think twice about saying you can go last again :P [21:25:04] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12234250 (10jasmine_) a:05Jhancock.wm→03jasmine_ [21:26:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:26:32] Yeah sorry about that, part of it was because testing took me 20 mins, because a different part of the feature turned out to be broken for unrelated reasons [21:26:37] But we just got that fixed too [21:26:48] (by changing config in Zendesk, not an MW-side issue) [21:27:37] (03PS1) 10Ladsgroup: Enable thumb.wikimedia.org on mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327197 (https://phabricator.wikimedia.org/T427465) [21:28:41] !log eevans@cumin1003 START - Cookbook sre.hosts.reboot-single for host cassandra-dev2001.codfw.wmnet [21:29:17] RoanKattouw: oh that 47min includes the staging pause? [21:29:24] I was commenting on scap being slow, not you :) [21:29:31] scap was also slow yes [21:30:05] I started it at 13:35 PDT, it finished staging at 13:55, I told it to continue at 14:09, and the deploy finished at 14:22 [21:30:21] So yes that 47 minutes includes 19 minutes of me being confused during staging [21:30:42] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [21:30:42] PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [21:30:48] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327162 (owner: 10Krinkle) [21:30:49] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [extensions/Math] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327169 (https://phabricator.wikimedia.org/T434686) (owner: 10Krinkle) [21:30:50] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327176 (https://phabricator.wikimedia.org/T347123) (owner: 10Krinkle) [21:30:58] But the one before that was 8 minutes, of which 3 were staging [21:31:06] np :) [21:31:13] "soon" things will improve [21:32:02] Scap did warn me that I had an i18n change and that it would take over 20 minutes, and that wasn't far off, excluding testing time it was 28 minutes [21:32:03] T99740 should reduce built time (php faster than .cdb to write), as well as upload/download time (smaller footprint for docker registry) [21:32:03] T99740: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 [21:32:06] That's honestly better than it used to be [21:32:14] and no json/md5 indirection [21:32:57] A few years ago a full scap was I think down to 15min but then it went up again when we introduced php-fpm restarts (to avoid opcache corruption) and then further with docker/k8s builds. [21:33:15] but yeah, we've been around when scap was slower than that :) [21:33:30] at least we're not rsyncing from NFS anymore [21:34:02] wow, when was that a thing?? [21:34:11] 2011? [21:34:24] !log eevans@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cassandra-dev2001.codfw.wmnet [21:34:35] https://wikitech.wikimedia.org/wiki/Obsolete:Hwcp [21:35:21] https://wikitech.wikimedia.org/wiki/Test.wikipedia.org#Before_2013 [21:35:29] (03CR) 10Aghirelli: [C:04-1] "Remember to rename the param `mode` to `availability` after the merge of this patch: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/132" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324819 (https://phabricator.wikimedia.org/T434267) (owner: 10Milazg) [21:35:36] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 9235 bytes in 5.764 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [21:35:36] RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 55878 bytes in 5.811 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [21:35:53] https://wikitech.wikimedia.org/wiki/Scap#History [21:37:02] 2004: prod mounts NFS direct, 200X-2013: only the deployment server and testwiki mount NFS and we rsync to the ~300 other apache servers, 2013-2020: rsync from the deployment server to all apache servers (same but without NFS), 2020-: docker uploaded to registery and then pulled by k8s [21:37:13] wow. That rules [21:45:36] (03Merged) 10jenkins-bot: Article: Split subjectpageheader by model and disable for wikitext [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327162 (owner: 10Krinkle) [21:45:39] (03CR) 10CI reject: [V:04-1] Make uppercase greek letters normal (non-italic) font [extensions/Math] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327169 (https://phabricator.wikimedia.org/T434686) (owner: 10Krinkle) [21:49:03] (03Merged) 10jenkins-bot: Skin: Avoid DB lookup for pagecategorieslink message [core] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327176 (https://phabricator.wikimedia.org/T347123) (owner: 10Krinkle) [21:49:08] (03CR) 10Krinkle: [C:03+2] "Flaky T435272" [extensions/Math] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327169 (https://phabricator.wikimedia.org/T434686) (owner: 10Krinkle) [22:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260819T2200) [22:00:55] !log krinkle@deploy1003 Locking from deployment [ALL REPOSITORIES]: awaiting CI [22:00:56] !log krinkle@deploy1003 Unlocked for deployment [ALL REPOSITORIES]: awaiting CI (duration: 00m 01s) [22:01:01] !log krinkle@deploy1003 Locking from deployment [ALL REPOSITORIES]: awaiting CI [22:01:21] two of three pathes merged but undeployed.. [22:01:28] waiting for flaky CI to clear to finish it [22:01:38] hey Krinkle let me know when you are done so I can get a config patch deployed in the readers window (no rush! :)) [22:01:51] ack, I locked it in case you weren't here to avoid a crossing of the streams [22:02:04] scap isn't running but repo is dirty until it clears [22:02:09] should take 20min or so. [22:03:59] (03Merged) 10jenkins-bot: Make uppercase greek letters normal (non-italic) font [extensions/Math] (wmf/1.47.0-wmf.16) - 10https://gerrit.wikimedia.org/r/1327169 (https://phabricator.wikimedia.org/T434686) (owner: 10Krinkle) [22:04:08] !log krinkle@deploy1003 Unlocked for deployment [ALL REPOSITORIES]: awaiting CI (duration: 03m 06s) [22:04:39] scap backport started now [22:04:45] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1327162|Article: Split subjectpageheader by model and disable for wikitext]], [[gerrit:1327169|Make uppercase greek letters normal (non-italic) font (T434686 T434428)]], [[gerrit:1327176|Skin: Avoid DB lookup for pagecategorieslink message (T347123)]] [22:04:55] T434686: Fix forcemathmode attribute - https://phabricator.wikimedia.org/T434686 [22:04:55] T434428: \Omega rendered in italic rather than roman in Clientside SVG and MathML modes - https://phabricator.wikimedia.org/T434428 [22:04:55] T347123: Reduce database queries on parsercached logged-out page views (Sep 2023) - https://phabricator.wikimedia.org/T347123 [22:16:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx errors from lists.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqiad&var-cluster=text&var-origin=lists.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [22:17:18] !ack [22:17:18] All incidents are already acked. [22:21:10] (03PS2) 10Jdlrobson: Enable ReadingLists for all logged in users on test wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327201 (https://phabricator.wikimedia.org/T435258) [22:23:00] deployment 43% [22:24:57] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1327162|Article: Split subjectpageheader by model and disable for wikitext]], [[gerrit:1327169|Make uppercase greek letters normal (non-italic) font (T434686 T434428)]], [[gerrit:1327176|Skin: Avoid DB lookup for pagecategorieslink message (T347123)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:25:05] T434686: Fix forcemathmode attribute - https://phabricator.wikimedia.org/T434686 [22:25:05] T434428: \Omega rendered in italic rather than roman in Clientside SVG and MathML modes - https://phabricator.wikimedia.org/T434428 [22:25:06] T347123: Reduce database queries on parsercached logged-out page views (Sep 2023) - https://phabricator.wikimedia.org/T347123 [22:25:54] (03PS6) 10Bernard Wang: Remove reading list experiment instrumentation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1259251 (https://phabricator.wikimedia.org/T421939) (owner: 10LorenMora) [22:29:25] !log krinkle@deploy1003 krinkle: Continuing with deployment [22:36:50] (03PS3) 10DLynch: editcheck-headless: Add chart and deployment for the technical pilot [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326332 (https://phabricator.wikimedia.org/T434109) [22:38:37] (03CR) 10DLynch: "I updated the app:version to the now-available docker image." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1326332 (https://phabricator.wikimedia.org/T434109) (owner: 10DLynch) [22:42:37] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327162|Article: Split subjectpageheader by model and disable for wikitext]], [[gerrit:1327169|Make uppercase greek letters normal (non-italic) font (T434686 T434428)]], [[gerrit:1327176|Skin: Avoid DB lookup for pagecategorieslink message (T347123)]] (duration: 37m 52s) [22:42:46] Jdlrobson: all yours [22:42:46] T434686: Fix forcemathmode attribute - https://phabricator.wikimedia.org/T434686 [22:42:47] T434428: \Omega rendered in italic rather than roman in Clientside SVG and MathML modes - https://phabricator.wikimedia.org/T434428 [22:42:47] T347123: Reduce database queries on parsercached logged-out page views (Sep 2023) - https://phabricator.wikimedia.org/T347123 [22:43:26] (03Abandoned) 10Ladsgroup: Baseline user-facing metrics for thumbnails, with an experiment mechanism [puppet] - 10https://gerrit.wikimedia.org/r/1324778 (https://phabricator.wikimedia.org/T431597) (owner: 10Cparle) [22:45:04] thanks Krinkle [22:45:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327201 (https://phabricator.wikimedia.org/T435258) (owner: 10Jdlrobson) [22:46:17] (03Merged) 10jenkins-bot: Enable ReadingLists for all logged in users on test wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327201 (https://phabricator.wikimedia.org/T435258) (owner: 10Jdlrobson) [22:46:40] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1327201|Enable ReadingLists for all logged in users on test wiki (T435258)]] [22:46:45] T435258: Enable ReadingLists for all logged-in users [Aug 25] on phase 0 wikis (bnwiki, cswiki, viwiki and zhwiki) - https://phabricator.wikimedia.org/T435258 [22:49:13] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1327201|Enable ReadingLists for all logged in users on test wiki (T435258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:50:48] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [22:51:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx errors from lists.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqiad&var-cluster=text&var-origin=lists.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [22:58:00] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327201|Enable ReadingLists for all logged in users on test wiki (T435258)]] (duration: 11m 20s) [22:58:05] T435258: Enable ReadingLists for all logged-in users [Aug 25] on phase 0 wikis (bnwiki, cswiki, viwiki and zhwiki) - https://phabricator.wikimedia.org/T435258 [23:03:02] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:03:30] done [23:03:36] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:06:02] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:06:35] jouncebot: nowandnext [23:06:36] No deployments scheduled for the next 6 hour(s) and 53 minute(s) [23:06:36] In 6 hour(s) and 53 minute(s): MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260820T0600) [23:06:36] In 6 hour(s) and 53 minute(s): Primary database switchover (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260820T0600) [23:06:36] In 6 hour(s) and 53 minute(s): Phabricator maintenance window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260820T0600) [23:07:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327197 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [23:08:11] (03Merged) 10jenkins-bot: Enable thumb.wikimedia.org on mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327197 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [23:08:31] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1327197|Enable thumb.wikimedia.org on mediawiki.org (T427465)]] [23:08:36] T427465: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465 [23:09:01] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [23:09:02] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:09:06] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [23:09:25] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [23:10:44] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [23:10:56] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1327197|Enable thumb.wikimedia.org on mediawiki.org (T427465)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:12:34] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:15:11] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [23:15:34] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:16:09] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [23:18:43] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [23:19:21] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1327197|Enable thumb.wikimedia.org on mediawiki.org (T427465)]] (duration: 10m 50s) [23:19:25] T427465: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465 [23:21:02] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:22:40] FIRING: SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:24:02] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:25:27] 06SRE, 10DNS, 10Domains, 06Traffic-Icebox, 07HTTPS: Merge Wikipedia subdomains into one, to discourage censorship - https://phabricator.wikimedia.org/T215071#12234740 (10BCornwall) I also think this can be closed as there aren't any plans (that I know of) to remove the lang subdomains. mobile domains hav... [23:26:34] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:27:02] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:29:47] (03CR) 10BryanDavis: [V:03+1 C:03+1] "Both sets of checks ran from deployment server as expected:" [puppet] - 10https://gerrit.wikimedia.org/r/1327168 (https://phabricator.wikimedia.org/T428972) (owner: 10Scott French) [23:33:02] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:33:34] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:36:02] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:38:24] (03CR) 10Ladsgroup: [C:03+2] Allow ogg thumbs in png, jpg, and webp [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1326268 (https://phabricator.wikimedia.org/T430528) (owner: 10Samwilson) [23:38:41] (03PS7) 10Cwhite: centralserver: add utility to delete oldest file based on disk usage [puppet] - 10https://gerrit.wikimedia.org/r/1321669 (https://phabricator.wikimedia.org/T434690) [23:39:02] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:39:04] (03CR) 10Cwhite: centralserver: add utility to delete oldest file based on disk usage (036 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1321669 (https://phabricator.wikimedia.org/T434690) (owner: 10Cwhite) [23:41:34] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:42:15] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1327217 [23:42:15] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1327217 (owner: 10TrainBranchBot) [23:43:01] (03PS1) 10Krinkle: RunSingleJob: Define MW_ENTRY_POINT for flamegraph sample attribution [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1327219 (https://phabricator.wikimedia.org/T435422) [23:45:05] (03Merged) 10jenkins-bot: Allow ogg thumbs in png, jpg, and webp [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1326268 (https://phabricator.wikimedia.org/T430528) (owner: 10Samwilson) [23:47:02] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:49:34] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:49:46] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1327217 (owner: 10TrainBranchBot) [23:50:02] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:51:02] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:54:02] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:54:48] 10ops-eqiad, 06DC-Ops, 06ServiceOps: Q1:rack/setup/install conf101[0-2] - https://phabricator.wikimedia.org/T435426 (10RobH) 03NEW [23:55:12] 10ops-eqiad, 06DC-Ops, 06ServiceOps: Q1:rack/setup/install conf101[0-2] - https://phabricator.wikimedia.org/T435426#12234886 (10RobH) [23:56:22] 10ops-eqiad, 06DC-Ops, 06ServiceOps: Q1:rack/setup/install conf101[0-2] - https://phabricator.wikimedia.org/T435426#12234895 (10RobH) a:03Scott_French Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new serve... [23:56:34] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:57:02] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal