[00:00:32] !log [universalomega@bots171] test [00:00:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:08] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27069MiB (5% inode=99%); [00:17:08] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 69664MiB (15% inode=99%); [00:22:16] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 0.128 second response time [00:36:56] PROBLEM - test151 Check unit status of mediawiki_job_mediamoderation-scan on test151 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_mediamoderation-scan [00:50:16] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.511 second response time [00:50:41] !log upgraded db172 to Debian Trixie [00:50:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:13:58] !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php OATHAuth:disableOATHAuthForUser --wiki=metawikibeta SomeRandomDeveloper (END - exit=65280) [01:14:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:14:17] [1/2] > Fatal error: Cannot declare class DisableOATHAuthForUser, because the name is already in use in /srv/mediawiki/1.45/extensions/OATHAuth/maintenance/disableOATHAuthForUser.php on line 15 [01:14:18] [2/2] gg [01:15:26] @cosmicalpha can you try using Special:DisableOATHForUser [01:15:30] idk why the maint script is not working [01:16:40] Done [01:16:56] I used it yesterday myself but I think I used full paths. [01:16:58] thx [01:34:56] RECOVERY - test151 Check unit status of mediawiki_job_mediamoderation-scan on test151 is OK: OK: Status of the systemd unit mediawiki_job_mediamoderation-scan [03:02:58] PROBLEM - puppet181 Disk Space on puppet181 is CRITICAL: DISK CRITICAL - free space: / 1822MiB (5% inode=79%); [03:06:58] RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4236MiB (13% inode=79%); [03:07:49] miraheze/mw-config - Universal-Omega the build passed. [03:09:19] miraheze/mw-config - Universal-Omega the build passed. [11:21:24] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/MirahezeMagic:CheckWikiDatabases.php --wiki=loginwiki (END - exit=256) [11:21:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:21:43] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/extensions/CheckWikiDatabases.php --wiki=loginwiki (END - exit=256) [11:21:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:49:08] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49771MiB (10% inode=99%); [12:35:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/28f907e3a9451d5710160912ebc74c31f9b379ef [12:35:19] 02ssl/03main 07WikiTideBot 0328f907e Bot: Auto-update domain lists [12:56:45] [02dns] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/dns/commit/3f1459f3f7cb [12:56:45] 02dns/03paladox-patch-1 07paladox 033f1459f Update ns1 ip [12:56:48] [02dns] 07paladox opened pull request #700: Update ns1 ip (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/700 [12:56:55] [02dns] 07paladox retitled PR #700: "Update ns1 ip" ➜ "Update ns1 private ip" 13https://github.com/miraheze/dns/pull/700 [12:57:40] miraheze/dns - paladox the build passed. [13:10:00] !log [skye@mwtask181] starting deploy of {'config': True} to all [13:10:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:10:30] !log [skye@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 29s [13:10:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:19:54] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/0b770c516d7b [13:19:55] 02puppet/03paladox-patch-1 07paladox 030b770c5 Update ns1 private ip [13:19:57] [02puppet] 07paladox opened pull request #4661: Update ns1 private ip (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4661 [13:20:02] [02puppet] 07coderabbitai[bot] commented on pull request #4661:

[…] 13https://github.com/miraheze/puppet/pull/4661#issuecomment-3734549787 [13:20:23] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/1e8be7bde07a7782e7dfd0e053c9e4d0f68c5d9c [13:20:23] 02puppet/03paladox-patch-1 07paladox 031e8be7b Update DNS nameserver from 10.0.17.136 to 10.0.17.171 [13:25:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/298bc6c947b76b9429b3d4055324736daf9d0aa1 [13:25:18] 02ssl/03main 07WikiTideBot 03298bc6c Bot: Auto-update domain lists [13:29:55] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [13:29:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:30:22] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 27s [13:30:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:55:44] [02mw-config] 07paladox created 03paladox-patch-4 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/ff8af758aca7 [13:55:44] 02mw-config/03paladox-patch-4 07paladox 03ff8af75 Increase wgQuickInstantCommonsPrefetchMaxLimit to 1000 [13:55:48] [02mw-config] 07paladox opened pull request #6224: Increase wgQuickInstantCommonsPrefetchMaxLimit to 1000 (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/6224 [13:55:55] [02mw-config] 07paladox merged pull request #6224: Increase wgQuickInstantCommonsPrefetchMaxLimit to 1000 (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/6224 [13:55:57] [02mw-config] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/df4b5331623e7ca3adcde706b703734afe61e989 [13:55:59] [02mw-config] 07paladox 04deleted 03paladox-patch-4 at 03ff8af75 13https://api.github.com/repos/miraheze/mw-config/commit/ff8af75 [13:56:01] 02mw-config/03main 07paladox 03df4b533 Increase wgQuickInstantCommonsPrefetchMaxLimit to 1000 (#6224) [13:56:18] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [13:56:19] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:56:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:41] miraheze/mw-config - paladox the build passed. [13:56:49] miraheze/mw-config - paladox the build passed. [13:56:59] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [13:57:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:57:29] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s [13:57:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:57:53] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [13:57:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:58:22] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 29s [13:58:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:40] [02mw-config] 07paladox created 03paladox-patch-4 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/70ed297e1c0d [14:36:40] 02mw-config/03paladox-patch-4 07paladox 0370ed297 Randomly shift between bast161 and bast181… [14:36:44] [02mw-config] 07paladox opened pull request #6225: Randomly shift between bast161 and bast181 (03main...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/6225 [14:37:23] [02mw-config] 07paladox pushed 1 new commit to 03paladox-patch-4 13https://github.com/miraheze/mw-config/commit/24a801eef775c17365987b0beec41890e2c6635c [14:37:24] 02mw-config/03paladox-patch-4 07paladox 0324a801e Update HTTP proxy configuration to use variable [14:37:35] miraheze/mw-config - paladox the build passed. [14:37:54] [02mw-config] 07paladox pushed 1 new commit to 03paladox-patch-4 13https://github.com/miraheze/mw-config/commit/bc79aada72c72be2d8d7f6b980403c548de5d053 [14:37:54] 02mw-config/03paladox-patch-4 07paladox 03bc79aad Update hCaptcha proxy URL to use variable [14:38:15] miraheze/mw-config - paladox the build passed. [14:38:48] miraheze/mw-config - paladox the build passed. [14:50:43] [02dns] 07paladox merged pull request #700: Update ns1 private ip (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/700 [14:50:43] [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/2ca6a6f06d7db325af3b4c5b203f79c394c748de [14:50:44] 02dns/03main 07paladox 032ca6a6f Update ns1 private ip (#700) [14:50:46] [02dns] 07paladox 04deleted 03paladox-patch-1 at 033f1459f 13https://api.github.com/repos/miraheze/dns/commit/3f1459f [14:51:20] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1a57fe16f809002e1d4bda7de4e532b8a649f212 [14:51:20] 02puppet/03main 07paladox 031a57fe1 Update ns1 private ip (#4661) [14:51:20] [02puppet] 07paladox merged pull request #4661: Update ns1 private ip (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4661 [14:51:21] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 031e8be7b 13https://api.github.com/repos/miraheze/puppet/commit/1e8be7b [14:51:40] miraheze/dns - paladox the build passed. [14:56:15] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:DeleteWikis --wiki=loginwiki --delete Reception123 (END - exit=2) [14:56:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:57:17] PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[openwebui_pip_install] [14:59:22] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: No response from DNS 10.0.17.136 [14:59:49] PROBLEM - ns1 SSH on ns1 is CRITICAL: connect to address 10.0.17.136 and port 22: No route to host [14:59:54] PROBLEM - ns1 NTP time on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host [15:00:03] PROBLEM - ping on ns1 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.136) [15:00:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host [15:00:14] Huh [15:00:23] PROBLEM - os202 Puppet on os202 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Os_instance_conn_validator[graylog-internal-template] [15:00:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host [15:00:29] @paladox that you? [15:00:44] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host [15:00:44] PROBLEM - ns1 Check unit status of prometheus_gdnsd_stats on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host [15:00:51] PROBLEM - ns1 conntrack_table_size on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host [15:00:51] PROBLEM - ns1 Current Load on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host [15:00:56] PROBLEM - Host ns1 is DOWN: CRITICAL - Host Unreachable (10.0.17.136) [15:01:05] yeh oh shit forgot to downtime because of the ip change [15:01:09] ns1 is up and working [15:01:10] Sigh [15:01:15] different ip that needs running on mon [15:01:23] i was running puppet everywhere [15:01:33] please use !log [15:01:35] Or say [15:01:56] sure [15:02:23] RECOVERY - os202 Puppet on os202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:36] RECOVERY - Host ns1 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [15:04:44] RECOVERY - ns1 Check unit status of prometheus_gdnsd_stats on ns1 is OK: OK: Status of the systemd unit prometheus_gdnsd_stats [15:04:52] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [15:04:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:04:54] RECOVERY - ns1 Current Load on ns1 is OK: LOAD OK - total load average: 0.27, 0.13, 0.07 [15:04:54] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.027 seconds response time. ns1.wikitide.net returns 2602:294:0:b23::111,38.46.223.204 [15:04:54] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -5.5372715e-05 secs [15:04:56] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [15:05:03] RECOVERY - ns1 conntrack_table_size on ns1 is OK: OK: nf_conntrack is 0 % full [15:05:10] RECOVERY - ping on ns1 is OK: PING OK - Packet loss = 0%, RTA = 0.29 ms [15:05:36] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49854MiB (10% inode=99%); [15:06:29] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 6328MiB (72.4% inode=86%); [15:11:21] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [15:21:46] !log [paladox@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/extensions/MatomoAnalytics/maintenance/DeleteCache.php [15:21:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:14] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49881MiB (10% inode=99%); [17:15:23] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/917dc43f2d58058317196f9789863eab54bb9d75 [17:15:23] 02ssl/03main 07WikiTideBot 03917dc43 Bot: Auto-update domain lists [17:21:02] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:CheckSwiftContainers --wiki=loginwiki (END - exit=0) [17:21:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:22:29] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768148510000&orgId=1&to=1768152149266 [17:25:21] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/bf9dcd81fca64f195dfea65b59ecc01a72f5fd8e [17:25:21] 02ssl/03main 07WikiTideBot 03bf9dcd8 Bot: Auto-update domain lists [17:27:29] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768148510000&orgId=1&to=1768152449266[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768148750000&orgId=1&to=1768152449266 [17:29:50] !log DELETED swift containers https://issue-tracker.miraheze.org/P576 [17:29:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:30:05] !log upgraded prometheus151 to Debian Trixie [17:30:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:29] [Grafana] RESOLVED: DatasourceError https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768148510000&orgId=1&to=1768152620000[Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768148930000&orgId=1&to=1768152680000 [17:33:34] PROBLEM - swiftproxy161 Current Load on swiftproxy161 is WARNING: LOAD WARNING - total load average: 7.08, 4.65, 2.88 [17:33:51] RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 6.76, 4.74, 2.95 [17:39:55] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f107b25721d9f71d8d3571c31c2f66630ab33d1c [17:39:55] 02puppet/03main 07CosmicAlpha 03f107b25 opensearch: upgrade to 2.19.4 [18:08:38] [02puppet] 07Universal-Omega created 03java-trixie (+1 new commit) 13https://github.com/miraheze/puppet/commit/72ffa80bdb5a [18:08:38] 02puppet/03java-trixie 07CosmicAlpha 0372ffa80 java: support debian trixie [18:08:43] [02puppet] 07Universal-Omega opened pull request #4662: java: support debian trixie (03main...03java-trixie) 13https://github.com/miraheze/puppet/pull/4662 [18:08:49] [02puppet] 07coderabbitai[bot] commented on pull request #4662:

[…] 13https://github.com/miraheze/puppet/pull/4662#issuecomment-3735228810 [18:14:37] [02puppet] 07Universal-Omega merged pull request #4662: java: support debian trixie (03main...03java-trixie) 13https://github.com/miraheze/puppet/pull/4662 [18:14:38] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/7151bee2540d469e1c2edbc4176f9559f287e50c [18:14:39] 02puppet/03main 07CosmicAlpha 037151bee java: support debian trixie [18:14:41] [02puppet] 07Universal-Omega 04deleted 03java-trixie at 0372ffa80 13https://api.github.com/repos/miraheze/puppet/commit/72ffa80 [18:16:10] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/60a0f99d9b499551be6d31ae319e33a81e57230f [18:16:10] 02puppet/03main 07CosmicAlpha 0360a0f99 mattermost: support debian trixie [18:50:16] !log upgraded os151 to Debian Trixie [18:50:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:46] PROBLEM - db161 Puppet on db161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [18:53:47] PROBLEM - eventgate181 Puppet on eventgate181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [18:53:47] PROBLEM - bast161 Puppet on bast161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [18:53:47] PROBLEM - cp191 Puppet on cp191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [18:53:50] PROBLEM - ldap171 Puppet on ldap171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [18:53:50] PROBLEM - mw163 Puppet on mw163 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [18:53:51] PROBLEM - db172 Puppet on db172 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [18:53:52] PROBLEM - db151 Puppet on db151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 23 minutes ago with 0 failures [18:53:56] PROBLEM - mw192 Puppet on mw192 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [18:53:57] PROBLEM - cloud18 Puppet on cloud18 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [18:54:03] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [18:54:04] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [18:54:05] PROBLEM - bast181 Puppet on bast181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [18:54:05] PROBLEM - mw201 Puppet on mw201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [18:54:06] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [18:54:06] PROBLEM - reports171 Puppet on reports171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 23 minutes ago with 0 failures [18:54:10] PROBLEM - mattermost1 Puppet on mattermost1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [18:54:11] PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [18:54:12] PROBLEM - kafka181 Puppet on kafka181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [18:54:12] PROBLEM - cloud17 Puppet on cloud17 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [18:54:14] PROBLEM - cp201 Puppet on cp201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [18:54:16] PROBLEM - cp171 Puppet on cp171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [18:54:17] PROBLEM - mem161 Puppet on mem161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [18:54:17] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 21 minutes ago with 0 failures [18:54:17] PROBLEM - mem151 Puppet on mem151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [18:54:23] PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 17 minutes ago with 0 failures [18:54:23] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [18:54:26] PROBLEM - db171 Puppet on db171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [18:54:26] PROBLEM - graylog161 Puppet on graylog161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [18:54:26] PROBLEM - db181 Puppet on db181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 26 minutes ago with 0 failures [18:54:28] PROBLEM - bots171 Puppet on bots171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [18:54:30] PROBLEM - mw193 Puppet on mw193 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [18:54:31] PROBLEM - cp161 Puppet on cp161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [18:54:32] PROBLEM - mw183 Puppet on mw183 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [18:54:32] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures [18:54:34] PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [18:54:49] PROBLEM - changeprop201 Puppet on changeprop201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [18:54:49] PROBLEM - mw153 Puppet on mw153 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [18:54:52] PROBLEM - swiftobject161 Puppet on swiftobject161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 23 minutes ago with 0 failures [18:54:53] PROBLEM - cloud20 Puppet on cloud20 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 22 minutes ago with 0 failures [18:54:58] PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 26 minutes ago with 0 failures [18:54:59] PROBLEM - rdb151 Puppet on rdb151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [18:54:59] PROBLEM - swiftobject191 Puppet on swiftobject191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [18:55:08] PROBLEM - swiftobject171 Puppet on swiftobject171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 29 minutes ago with 0 failures [18:55:09] PROBLEM - os202 Puppet on os202 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [18:55:12] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [18:55:15] PROBLEM - mw173 Puppet on mw173 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [18:55:15] PROBLEM - mw202 Puppet on mw202 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [18:55:18] PROBLEM - swiftobject201 Puppet on swiftobject201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [18:55:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/b8497066d14180cc6db8ed7f051e93d0b329e09d [18:55:18] 02ssl/03main 07WikiTideBot 03b849706 Bot: Auto-update domain lists [18:55:19] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [18:55:20] PROBLEM - swiftobject181 Puppet on swiftobject181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [18:55:21] PROBLEM - llm191 Puppet on llm191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [18:55:23] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 21 minutes ago with 0 failures [18:55:23] PROBLEM - cloud15 Puppet on cloud15 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [18:55:24] PROBLEM - swiftac171 Puppet on swiftac171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [18:55:25] PROBLEM - mem201 Puppet on mem201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures [18:55:25] PROBLEM - mw191 Puppet on mw191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [18:55:28] PROBLEM - db192 Puppet on db192 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 23 minutes ago with 0 failures [18:55:29] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [18:55:32] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [18:55:32] PROBLEM - mem191 Puppet on mem191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [18:55:33] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [18:55:33] PROBLEM - cloud19 Puppet on cloud19 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [18:55:35] PROBLEM - cloud16 Puppet on cloud16 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [18:55:37] PROBLEM - db201 Puppet on db201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [18:55:41] PROBLEM - mw203 Puppet on mw203 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [18:55:42] PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [18:55:45] PROBLEM - db182 Puppet on db182 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [18:55:45] PROBLEM - os162 Puppet on os162 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 29 minutes ago with 0 failures [18:55:46] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [18:55:46] PROBLEM - os191 Puppet on os191 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 25 minutes ago with 0 failures [18:55:48] PROBLEM - mwtask161 Puppet on mwtask161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [18:55:50] PROBLEM - puppet181 Puppet on puppet181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [18:55:50] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [18:55:50] PROBLEM - os161 Puppet on os161 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [18:55:51] PROBLEM - os201 Puppet on os201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [18:55:59] PROBLEM - swiftobject151 Puppet on swiftobject151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [18:59:38] ^ stopped ircecho [18:59:52] !log upgrade puppet181 to trixie [18:59:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:39] !log reboot puppet181 [19:11:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:37] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/1c603dc9e1d6 [19:18:37] 02puppet/03paladox-patch-1 07paladox 031c603dc puppetdb: Replace package 'policykit-1' with 'polkit'… [19:18:41] [02puppet] 07paladox opened pull request #4663: puppetdb: Replace package 'policykit-1' with 'polkit' (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4663 [19:18:46] [02puppet] 07coderabbitai[bot] commented on pull request #4663:

[…] 13https://github.com/miraheze/puppet/pull/4663#issuecomment-3735500286 [19:18:49] [02puppet] 07paladox merged pull request #4663: puppetdb: Replace package 'policykit-1' with 'polkit' (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4663 [19:18:49] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f738cfd8426d2b211fa517c357511c15bae2c048 [19:18:49] 02puppet/03main 07paladox 03f738cfd puppetdb: Replace package 'policykit-1' with 'polkit' (#4663)… [19:18:51] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 031c603dc 13https://api.github.com/repos/miraheze/puppet/commit/1c603dc [20:04:45] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/ac20d93fc39d9fabdd59b95212ec1a9f9ba87f5b [20:04:45] 02puppet/03main 07CosmicAlpha 03ac20d93 Update mattermost to 11.2.1 [20:15:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f318d0a9c0a6abe24765b71b8cdd413f5c3c0af5 [20:15:18] 02ssl/03main 07WikiTideBot 03f318d0a Bot: Auto-update domain lists [20:22:07] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/62c50b2a31ef [20:22:07] 02puppet/03paladox-patch-1 07paladox 0362c50b2 puppetdb: Change polkit package to polkitd and pkexec [20:22:11] [02puppet] 07paladox opened pull request #4664: puppetdb: Change polkit package to polkitd and pkexec (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4664 [20:22:16] [02puppet] 07paladox merged pull request #4664: puppetdb: Change polkit package to polkitd and pkexec (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4664 [20:22:18] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c9406fa0e3585e67dd8571b7ca386e3e1e414a41 [20:22:20] [02puppet] 07coderabbitai[bot] commented on pull request #4664:

[…] 13https://github.com/miraheze/puppet/pull/4664#issuecomment-3735691594 [20:22:22] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 0362c50b2 13https://api.github.com/repos/miraheze/puppet/commit/62c50b2 [20:22:23] 02puppet/03main 07paladox 03c9406fa puppetdb: Change polkit package to polkitd and pkexec (#4664) [20:29:33] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/f64565d0587d [20:29:33] 02puppet/03paladox-patch-1 07paladox 03f64565d mediawiki: Increase max files to 3000 in tidy for /tmp… [20:29:36] [02puppet] 07paladox opened pull request #4665: mediawiki: Increase max files to 3000 in tidy for /tmp (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4665 [20:29:41] [02puppet] 07coderabbitai[bot] commented on pull request #4665:

[…] 13https://github.com/miraheze/puppet/pull/4665#issuecomment-3735713294 [20:29:43] [02puppet] 07paladox merged pull request #4665: mediawiki: Increase max files to 3000 in tidy for /tmp (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4665 [20:29:45] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f43e577f06315d37a82bf435d3db6c4e6054ecee [20:29:47] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03f64565d 13https://api.github.com/repos/miraheze/puppet/commit/f64565d [20:29:49] 02puppet/03main 07paladox 03f43e577 mediawiki: Increase max files to 3000 in tidy for /tmp (#4665)… [20:35:45] RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [20:35:45] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:35:45] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:35:47] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:35:50] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:35:51] RECOVERY - swiftobject201 Puppet on swiftobject201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:35:56] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:35:57] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [20:36:02] RECOVERY - os202 Puppet on os202 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:36:03] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:36:04] RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:36:04] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:36:04] RECOVERY - os201 Puppet on os201 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:36:06] RECOVERY - os191 Puppet on os191 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:36:07] RECOVERY - swiftobject191 Puppet on swiftobject191 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:36:15] RECOVERY - mon181 IRCEcho on mon181 is OK: PROCS OK: 1 process with args '/usr/local/bin/ircecho' [20:37:43] [02MatomoAnalytics] 07SomeMWDev approved pull request #171 13https://github.com/miraheze/MatomoAnalytics/pull/171#pullrequestreview-3648476369 [20:37:46] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:40:11] PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:40:24] Huh [20:41:12] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:41:15] PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:41:51] PROBLEM - db172 Puppet on db172 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wikitide.net.key],File[/etc/ssl/private/miraheze-origin-cert.key],File[/etc/ssl/private/mirabeta-origin-cert.key],File[/etc/ferm/conf.d/02_main] [20:42:08] PROBLEM - eventgate181 Puppet on eventgate181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:42:23] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:42:53] @paladox ircecho came up a bit early? [20:43:11] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:43:14] yeh i ran puppet everywhere [20:43:18] which hit mon [20:43:45] it should recover [20:43:49] RECOVERY - eventgate181 Puppet on eventgate181 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:43:54] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [20:43:55] RECOVERY - db172 Puppet on db172 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:44:01] Cool [20:44:05] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:46:28] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:08:00] [02RemovePII] 07SomeMWDev merged pull request #116: Remove Comments table update from GeneratePII.php (03main...03remove-comments-2) 13https://github.com/miraheze/RemovePII/pull/116 [21:08:00] [02RemovePII] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/RemovePII/commit/9f4e9b17c7d40db3e969df7b2e6319f0d9540949 [21:08:00] 02RemovePII/03main 07SomeRandomDeveloper 039f4e9b1 Remove Comments table update from GeneratePII.php (#116)… [21:08:01] [02RemovePII] 07SomeMWDev 04deleted 03remove-comments-2 at 03e9a930f 13https://api.github.com/repos/miraheze/RemovePII/commit/e9a930f [21:08:13] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'RemovePII'} to test151 [21:08:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:08:15] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'RemovePII'} to test151 - SUCCESS in 1s [21:08:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:08:18] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'RemovePII'} to all [21:08:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:08:47] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'RemovePII'} to all - SUCCESS in 29s [21:08:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:16:02] !log DELETED and DROPPED https://issue-tracker.miraheze.org/P577 & https://issue-tracker.miraheze.org/P578 [21:16:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:19:31] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 11 backends are down. mw151 mw152 mw161 mw171 mw182 mw163 mw183 mw191 mw193 mw201 mw203 [21:19:33] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 10 backends are down. mw152 mw161 mw171 mw172 mw182 mw163 mw191 mw193 mw201 mw203 [21:19:36] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:19:38] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:19:45] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:20:02] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:20:07] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:20:19] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 9 backends are down. mw151 mw161 mw162 mw171 mw181 mw182 mw173 mw183 mw191 [21:20:24] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:20:27] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 11 backends are down. mw151 mw161 mw162 mw171 mw181 mw182 mw163 mw173 mw183 mw191 mw193 [21:20:31] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:20:56] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 49% [21:21:25] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:21:32] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.063 second response time [21:21:47] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 2.529 second response time [21:21:50] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:22:04] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 1.886 second response time [21:24:24] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [21:24:33] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [21:25:16] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [21:25:36] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.500 second response time [21:25:50] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:18] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.219 second response time [21:26:21] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 1.048 second response time [21:26:30] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:26:40] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 9.223 second response time [21:27:06] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4358 bytes in 4.717 second response time [21:27:44] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/59c5b12c9412ef9d2ed924319ac9a0afcb0ff9de [21:27:44] 02puppet/03main 07CosmicAlpha 0359c5b12 Remove for now [21:28:14] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 4.756 second response time [21:28:40] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:28:42] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [21:28:57] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 5.198 second response time [21:29:09] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:29:39] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:30:16] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [21:30:18] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [21:30:29] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [21:30:33] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:30:35] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.193 second response time [21:30:59] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:31:12] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:15]