[00:01:23] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/9a618fb8c2f84d50999cace78c3555ab4a28e855 [00:01:23] 02statichelp/03main 07WikiTideBot 039a618fb Bot: Auto-update Tech namespace pages 2026-01-10 00:01:21 [00:12:53] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 73791MiB (16% inode=99%); [00:13:35] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 74067MiB (16% inode=99%); [00:15:23] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 72821MiB (16% inode=99%); [01:04:39] high load on db161 [01:05:31] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 24.60, 24.89, 12.49 [01:07:05] seems to go down again [01:09:05] hm it's not really recovering but also not crashing [01:19:21] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 0.99, 7.34, 11.88 [01:23:18] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.58, 4.11, 9.53 [09:05:17] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/129bd2646d6ccdfe5d0a8651d0cacfb323131e2e [09:05:17] 02ssl/03main 07WikiTideBot 03129bd26 Bot: Auto-update domain lists [11:14:54] !log re-run delbackups after fixing issue [11:14:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:18:41] [1/8] db disk space available pre-deletion: [11:18:41] [2/8] db201: 292G [11:18:42] [3/8] db161: 337G [11:18:42] [4/8] db171: 396G [11:18:42] [5/8] db151: 387G [11:18:43] [6/8] db182: 367G [11:18:43] [7/8] db181: 427G [11:18:43] [8/8] db192: 128G [12:59:46] swift disk space available: 1.1T [14:45:23] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49799MiB (10% inode=99%); [15:04:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.84, 17.68, 9.99 [15:10:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.91, 22.33, 14.71 [15:10:53] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49789MiB (10% inode=99%); [15:11:42] !log reinstall ns1 (as a new vm to do a different bios setup) [15:11:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:12:21] To do a trixie upgrade ^ [15:16:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.09, 22.62, 17.20 [15:17:35] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49868MiB (10% inode=99%); [15:18:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.79, 22.63, 17.84 [15:21:08] PROBLEM - db192 PowerDNS Recursor on db192 is CRITICAL: Domain 'db192.fsslc.wtnet' was not found by the server [15:21:09] PROBLEM - os202 PowerDNS Recursor on os202 is CRITICAL: No response from DNS 127.0.0.1 [15:21:38] PROBLEM - db201 PowerDNS Recursor on db201 is CRITICAL: Domain 'db201.fsslc.wtnet' was not found by the server [15:22:33] PROBLEM - db192 NTP time on db192 is UNKNOWN: check_ntp_time: Invalid hostname/address - ntp.fsslc.wtnetUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [15:22:51] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.71, 18.96, 17.49 [15:22:58] PROBLEM - os202 NTP time on os202 is UNKNOWN: check_ntp_time: Invalid hostname/address - ntp.fsslc.wtnetUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [15:24:19] huh [15:24:31] it should have fallen over [15:25:06] i guess must be the monitoring tool? [15:25:35] oh [15:25:55] PROBLEM - db201 NTP time on db201 is UNKNOWN: check_ntp_time: Invalid hostname/address - ntp.fsslc.wtnetUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [15:26:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.19, 21.38, 18.81 [15:26:54] RECOVERY - db192 NTP time on db192 is OK: NTP OK: Offset 6.252527237e-05 secs [15:27:06] RECOVERY - os202 NTP time on os202 is OK: NTP OK: Offset 6.809830666e-05 secs [15:27:09] RECOVERY - os202 PowerDNS Recursor on os202 is OK: DNS OK: 0.085 seconds response time. os202.fsslc.wtnet returns 10.0.20.167 [15:27:09] hosts with only a private ip address could only use ns1 and can't access ns2 [15:27:11] RECOVERY - db192 PowerDNS Recursor on db192 is OK: DNS OK: 0.032 seconds response time. db192.fsslc.wtnet returns 10.0.19.170 [15:27:22] cc @cosmicalpha [15:27:37] RECOVERY - db201 PowerDNS Recursor on db201 is OK: DNS OK: 0.031 seconds response time. db201.fsslc.wtnet returns 10.0.20.169 [15:27:51] RECOVERY - db201 NTP time on db201 is OK: NTP OK: Offset -2.628564835e-05 secs [15:30:51] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.59, 19.71, 18.89 [15:31:10] PROBLEM - os202 PowerDNS Recursor on os202 is CRITICAL: Domain 'os202.fsslc.wtnet' was not found by the server [15:31:13] PROBLEM - db192 PowerDNS Recursor on db192 is CRITICAL: No response from DNS 127.0.0.1 [15:31:40] PROBLEM - db201 PowerDNS Recursor on db201 is CRITICAL: No response from DNS 127.0.0.1 [15:33:40] [02mw-config] 07pskyechology opened pull request #6223: Set wgWmincExistingWikis to an empty array (07miraheze:03main...07pskyechology:03T14782) 13https://github.com/miraheze/mw-config/pull/6223 [15:34:31] miraheze/mw-config - pskyechology the build passed. [15:35:51] PROBLEM - os202 NTP time on os202 is UNKNOWN: check_ntp_time: Invalid hostname/address - ntp.fsslc.wtnetUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [15:35:54] PROBLEM - db192 NTP time on db192 is UNKNOWN: check_ntp_time: Invalid hostname/address - ntp.fsslc.wtnetUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [15:36:54] PROBLEM - db201 NTP time on db201 is UNKNOWN: check_ntp_time: Invalid hostname/address - ntp.fsslc.wtnetUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [15:50:50] RECOVERY - db192 NTP time on db192 is OK: NTP OK: Offset 0.0001966357231 secs [15:51:07] RECOVERY - db192 PowerDNS Recursor on db192 is OK: DNS OK: 0.030 seconds response time. db192.fsslc.wtnet returns 10.0.19.170 [15:51:38] RECOVERY - db201 PowerDNS Recursor on db201 is OK: DNS OK: 0.048 seconds response time. db201.fsslc.wtnet returns 10.0.20.169 [15:51:43] RECOVERY - db201 NTP time on db201 is OK: NTP OK: Offset 0.0001264512539 secs [15:52:22] RECOVERY - os202 NTP time on os202 is OK: NTP OK: Offset 0.000219732523 secs [15:55:05] PROBLEM - db192 PowerDNS Recursor on db192 is CRITICAL: Domain 'db192.fsslc.wtnet' was not found by the server [15:55:38] PROBLEM - db201 PowerDNS Recursor on db201 is CRITICAL: Domain 'db201.fsslc.wtnet' was not found by the server [15:59:36] PROBLEM - db192 NTP time on db192 is UNKNOWN: check_ntp_time: Invalid hostname/address - ntp.fsslc.wtnetUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [16:01:07] RECOVERY - db192 PowerDNS Recursor on db192 is OK: DNS OK: 0.039 seconds response time. db192.fsslc.wtnet returns 10.0.19.170 [16:01:07] RECOVERY - os202 PowerDNS Recursor on os202 is OK: DNS OK: 0.039 seconds response time. os202.fsslc.wtnet returns 10.0.20.167 [16:01:33] RECOVERY - db192 NTP time on db192 is OK: NTP OK: Offset 0.0002569258213 secs [16:01:40] RECOVERY - db201 PowerDNS Recursor on db201 is OK: DNS OK: 0.037 seconds response time. db201.fsslc.wtnet returns 10.0.20.169 [16:01:48] PROBLEM - os202 Puppet on os202 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Os_instance_conn_validator[graylog-internal-template],Opensearch_template[graylog-internal] [16:01:56] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw191 mw192 mw193 mw201 mw203 [16:01:59] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 56.70, 29.79, 12.68 [16:02:35] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 4 backends are down. mw152 mw171 mw182 mw203 [16:03:18] @paladox uh this is very not bueno [16:03:36] i saw but wikis are loading for me [16:03:38] well were [16:04:30] weird [16:05:06] !log restart pdns-recursor everywhere [16:05:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:05:25] yea stuff is loading for me too [16:05:56] either that or all the wikis I managed to check are not affected [16:05:56] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [16:06:10] yay [16:06:35] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [16:08:39] !log make ns1 have 2 cores [16:09:22] did that involve a quick restart [16:09:56] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 6 backends are down. mw151 mw161 mw163 mw173 mw201 mw203 [16:10:35] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 13 backends are down. mw151 mw152 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw183 mw191 mw192 mw193 [16:10:46] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 13 backends are down. mw152 mw161 mw162 mw181 mw182 mw153 mw163 mw183 mw191 mw192 mw193 mw201 mw203 [16:10:51] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 9 backends are down. mw152 mw181 mw182 mw153 mw163 mw191 mw192 mw193 mw201 [16:11:07] Erm [16:11:45] oh come on it was just up [16:11:56] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [16:12:35] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [16:12:42] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:12:46] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [16:12:51] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [16:13:26] PROBLEM - cp171 Puppet on cp171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:14:05] !log delete old ns1 vm replaced by newer one [16:14:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:15:26] RECOVERY - cp171 Puppet on cp171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:15:34] yay [16:25:59] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 1.93, 7.39, 11.63 [16:27:48] RECOVERY - os202 Puppet on os202 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:29:59] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.88, 3.84, 9.20 [16:42:51] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.43, 16.97, 9.97 [16:42:58] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:54:51] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 7.96, 16.19, 14.52 [17:02:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.92, 20.32, 16.48 [17:06:51] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.34, 18.53, 16.82 [17:15:51] !log started deletewikis.php [17:15:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:48:08] RECOVERY - ns2 Check unit status of prometheus_gdnsd_stats on ns2 is OK: OK: Status of the systemd unit prometheus_gdnsd_stats [18:27:10] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php rebuildtextindex --wiki=dappervolkwiki (START) [18:27:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:39] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php rebuildtextindex --wiki=dappervolkwiki (END - exit=0) [18:28:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:45:20] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/901bd5abc60b449941f7d9e495341d9fb49216ee [18:45:21] 02ssl/03main 07WikiTideBot 03901bd5a Bot: Auto-update domain lists [19:05:17] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c6f67d4198de4e6e5b4d9794baa637adc4b542e9 [19:05:18] 02ssl/03main 07WikiTideBot 03c6f67d4 Bot: Auto-update domain lists [19:06:14] [02puppet] 07Universal-Omega created 03matomo-php8.4 (+1 new commit) 13https://github.com/miraheze/puppet/commit/8082b3fa2864 [19:06:14] 02puppet/03matomo-php8.4 07CosmicAlpha 038082b3f matomo: use PHP 8.4 [19:06:23] [02puppet] 07Universal-Omega opened pull request #4656: matomo: use PHP 8.4 (03main...03matomo-php8.4) 13https://github.com/miraheze/puppet/pull/4656 [19:06:29] [02puppet] 07coderabbitai[bot] commented on pull request #4656:

[…] 13https://github.com/miraheze/puppet/pull/4656#issuecomment-3733396980 [19:11:20] PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures [19:13:29] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 234 packages available for upgrade (3 critical updates). [19:16:13] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-0 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-queuedtracking-0 [19:16:26] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-queuedtracking-1 [19:16:55] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.009 second response time [19:17:18] Huh [19:17:19] PROBLEM - matomo151 php-fpm on matomo151 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [19:17:21] Oh [19:17:26] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:17:41] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-queuedtracking-2 [19:17:47] @cosmicalpha learn what downtime is please [19:20:18] [02puppet] 07Universal-Omega merged pull request #4656: matomo: use PHP 8.4 (03main...03matomo-php8.4) 13https://github.com/miraheze/puppet/pull/4656 [19:20:18] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/d37e7b570d46789ac18a666cb5b83b86df2723e8 [19:20:19] 02puppet/03main 07CosmicAlpha 03d37e7b5 matomo: use PHP 8.4 [19:20:21] [02puppet] 07Universal-Omega 04deleted 03matomo-php8.4 at 038082b3f 13https://api.github.com/repos/miraheze/puppet/commit/8082b3f [19:20:27] PROBLEM - matomo151 PowerDNS Recursor on matomo151 is CRITICAL: No response from DNS ::1 [19:24:33] Sorry that was my bad. Will do next time. [19:28:07] [02puppet] 07Universal-Omega created 03python3.13 (+1 new commit) 13https://github.com/miraheze/puppet/commit/da4a820e08a8 [19:28:07] 02puppet/03python3.13 07CosmicAlpha 03da4a820 Use python 3.13 in workflow… [19:28:11] [02puppet] 07Universal-Omega opened pull request #4657: Use python 3.13 in workflow (03main...03python3.13) 13https://github.com/miraheze/puppet/pull/4657 [19:28:16] [02puppet] 07coderabbitai[bot] commented on pull request #4657:

[…] 13https://github.com/miraheze/puppet/pull/4657#issuecomment-3733419696 [19:29:41] miraheze/puppet - Universal-Omega the build has errored. [19:47:56] !log upgraded matomo151 to Debian Trixie [19:47:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:52] [02puppet] 07Universal-Omega pushed 1 new commit to 03python3.13 13https://github.com/miraheze/puppet/commit/73accd70af5550a45ba8b1058d00b0ffc1afb77e [19:54:52] 02puppet/03python3.13 07CosmicAlpha 0373accd7 Update packages [19:56:22] miraheze/puppet - Universal-Omega the build passed. [19:57:33] [02puppet] 07Universal-Omega merged pull request #4657: Use python 3.13 in workflow (03main...03python3.13) 13https://github.com/miraheze/puppet/pull/4657 [19:57:33] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/97c334bf53d17e472cb61fc8d94b53bfd4decb7d [19:57:34] 02puppet/03main 07CosmicAlpha 0397c334b Use python 3.13 in workflow… [19:57:36] [02puppet] 07Universal-Omega 04deleted 03python3.13 at 0373accd7 13https://api.github.com/repos/miraheze/puppet/commit/73accd7 [19:58:57] miraheze/puppet - Universal-Omega the build passed. [20:10:42] !log downtimed db182 in icinga for trixie upgrade [20:10:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:03] RECOVERY - matomo151 PowerDNS Recursor on matomo151 is OK: DNS OK: 0.026 seconds response time. matomo151.fsslc.wtnet returns 10.0.15.112 [20:23:07] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4498 bytes in 0.022 second response time [20:23:20] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [20:23:23] RECOVERY - matomo151 php-fpm on matomo151 is OK: PROCS OK: 17 processes with command name 'php-fpm8.4' [20:23:48] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [20:30:13] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-0 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-0 [20:34:13] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-0 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-queuedtracking-0 [20:38:55] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.225 second response time [20:39:07] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19375 bytes in 0.073 second response time [20:39:20] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17668 bytes in 0.034 second response time [20:39:41] !log upgraded db182 to Debian Trixie [20:39:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:13] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-0 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-0 [20:40:17] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[phd] [20:40:26] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-1 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-1 [20:40:37] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-2 [20:41:15] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:46:32] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:DeleteWikis --wiki=loginwiki --delete Reception123 (END - exit=2) [20:46:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:35] !log DELETED and DROPPED https://issue-tracker.miraheze.org/P575 [20:50:46] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 11 backends are down. mw162 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw201 [20:50:51] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw183 mw191 mw192 mw201 [20:51:43] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [20:51:56] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 2 backends are down. mw192 mw201 [20:52:36] is the delete going that hard or is it something else [20:52:46] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [20:52:51] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [20:53:43] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.193 second response time [20:53:56] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [21:08:45] !log stop ircecho on mon181 for trixie upgrade to prevent immediately spamming everything. [21:08:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:16] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e320f1686ad84ff0fd7a331e212548b906dd13cd [21:11:16] 02puppet/03main 07CosmicAlpha 03e320f16 mon181: upgrade to PHP 8.4 [21:12:50] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1aef82cd714288a06e582fa4c0e7e72fb119c4c3 [21:12:50] 02puppet/03main 07CosmicAlpha 031aef82c icingaweb2: use PHP 8.4 by default [21:41:20] [02RemovePII] 07SomeMWDev created 03remove-comments-2 from 03main (+0 new commit) 13https://github.com/miraheze/RemovePII/compare/remove-comments-2 [21:42:25] [02RemovePII] 07SomeMWDev pushed 1 new commit to 03remove-comments-2 13https://github.com/miraheze/RemovePII/commit/e9a930f41a98e19ef7fac1023dd4c2daa0facedb [21:42:25] 02RemovePII/03remove-comments-2 07SomeRandomDeveloper 03e9a930f Remove Comments table update from GeneratePII.php… [21:42:34] [02RemovePII] 07SomeMWDev opened pull request #116: Remove Comments table update from GeneratePII.php (03main...03remove-comments-2) 13https://github.com/miraheze/RemovePII/pull/116 [21:42:39] [02RemovePII] 07coderabbitai[bot] commented on pull request #116:

[…] 13https://github.com/miraheze/RemovePII/pull/116#issuecomment-3733584655 [21:47:22] miraheze/RemovePII - SomeMWDev the build passed. [21:51:55] [02puppet] 07Universal-Omega created 03fix-ircecho (+1 new commit) 13https://github.com/miraheze/puppet/commit/4dc7de364531 [21:51:55] 02puppet/03fix-ircecho 07CosmicAlpha 034dc7de3 Fix ircecho for Python 3.13 [21:52:01] [02puppet] 07Universal-Omega opened pull request #4658: Fix ircecho for Python 3.13 (03main...03fix-ircecho) 13https://github.com/miraheze/puppet/pull/4658 [21:52:06] [02puppet] 07coderabbitai[bot] commented on pull request #4658:

[…] 13https://github.com/miraheze/puppet/pull/4658#issuecomment-3733594217 [21:52:37] PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [21:53:27] miraheze/puppet - Universal-Omega the build passed. [21:54:39] [02puppet] 07Universal-Omega merged pull request #4658: Fix ircecho for Python 3.13 (03main...03fix-ircecho) 13https://github.com/miraheze/puppet/pull/4658 [21:54:39] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2916058b437e630404ba1009b9c56dfc52593822 [21:54:39] 02puppet/03main 07CosmicAlpha 032916058 Fix ircecho for Python 3.13 [21:54:41] [02puppet] 07Universal-Omega 04deleted 03fix-ircecho at 034dc7de3 13https://api.github.com/repos/miraheze/puppet/commit/4dc7de3 [21:56:08] miraheze/puppet - Universal-Omega the build passed. [21:56:28] !log upgraded mon181 to Debian Trixie [21:56:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:59:39] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:48:58] [02puppet] 07BlankEclair opened pull request #4659: Temporarily remove SSH keys for blankeclair (07miraheze:03main...07BlankEclair:03hell-entrance) 13https://github.com/miraheze/puppet/pull/4659 [22:49:03] [02puppet] 07coderabbitai[bot] commented on pull request #4659:

[…] 13https://github.com/miraheze/puppet/pull/4659#issuecomment-3733643343 [23:14:59] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 71819MiB (15% inode=99%); [23:15:13] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 72409MiB (15% inode=99%); [23:57:02] miraheze/puppet - Universal-Omega the build passed. [23:57:34] !log test [23:57:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:57:46] !log upgraded bots171 to Debian Trixie [23:57:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:59:55] miraheze/puppet - Universal-Omega the build passed.