[00:01:39] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/9cb278d1ec123b247a33b43cd74cef9d59567543
[00:01:39] <MacFanBot>	 02statichelp/03main 07WikiTideBot 039cb278d Bot: Auto-update Tech namespace pages 2026-08-16 00:01:37
[00:02:54] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.46 ms
[00:04:55] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[00:09:05] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[00:11:05] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.50 ms
[00:13:57] <icinga-miraheze>	 PROBLEM - mem161 Puppet on mem161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:21:45] <icinga-miraheze>	 PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.33 ms
[00:23:11] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.49 ms
[00:23:43] <icinga-miraheze>	 RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.35 ms
[00:24:55] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.46 ms
[00:26:13] <icinga-miraheze>	 PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats
[00:26:58] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[00:29:16] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.47 ms
[00:32:45] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[00:34:45] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.49 ms
[00:38:13] <icinga-miraheze>	 RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats
[00:39:36] <icinga-miraheze>	 PROBLEM - mem201 Puppet on mem201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:41:57] <icinga-miraheze>	 RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[00:42:59] <icinga-miraheze>	 PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:43:53] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.47 ms
[00:43:57] <icinga-miraheze>	 PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 13.92, 20.68, 12.08
[00:44:30] <MacFanBot>	 [02mw-config] 07SomeMWDev commented on pull request #6495: Based on an analysis I did a while ago for Weird Gloop, on 1.45, Parsoid is about 2.2x slower for content pages than the legacy parser. According to the CTT, this has improved in 1.46 and 1.47. I think we should restrict the setting and only enable […] 13https://github.com/miraheze/mw-config/pull/6495#issuecomment-5304993090
[00:45:52] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[00:45:57] <icinga-miraheze>	 RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 6.97, 15.67, 11.28
[00:46:01] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.48 ms
[00:48:01] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.47 ms
[00:50:20] <MacFanBot>	 [02mw-config] 07SomeMWDev requested changes on pull request #6495: Maybe we should disable https://github.com/wikimedia/mediawiki-extensions-ParserMigration/blob/a86122b/extension.json#L37-L44 as well? 13https://github.com/miraheze/mw-config/pull/6495#pullrequestreview-4945063899
[00:50:20] <MacFanBot>	 [02mw-config] 07SomeMWDev left a file comment in pull request #6495 03560fb79: Looks like this setting is being renamed in 1.46: https://github.com/wikimedia/mediawiki-extensions-ParserMigration/commit/b8ed877 […] 13https://github.com/miraheze/mw-config/pull/6495#discussion_r3790622503
[00:51:16] <MacFanBot>	 [02mw-config] 07SomeMWDev commented on pull request #6495: (Should've probably already noted this in the task, sorry about that) 13https://github.com/miraheze/mw-config/pull/6495#issuecomment-5305014811
[00:51:55] <MacFanBot>	 [02mw-config] 07pixDeVl commented on pull request #6495: If these are also new defaults compared to 1.45, then yes they should be. 13https://github.com/miraheze/mw-config/pull/6495#issuecomment-5305017002
[00:52:11] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.46 ms
[00:54:12] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.47 ms
[00:54:52] <MacFanBot>	 [02mw-config] 07pixDeVl left a file comment in pull request #6495 03560fb79: So, the following will all be explicitly false. […] 13https://github.com/miraheze/mw-config/pull/6495#discussion_r3790631946
[01:01:29] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.48 ms
[01:05:59] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[01:07:31] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.48 ms
[01:07:36] <icinga-miraheze>	 RECOVERY - mem201 Puppet on mem201 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[01:10:30] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.52 ms
[01:11:08] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[01:12:34] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.52 ms
[01:14:06] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.46 ms
[01:14:50] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1786839260000&orgId=1&to=1786842890033
[01:15:35] <miraheze-github>	 miraheze/mw-config - pixDeVl the build passed.
[01:15:59] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.45 ms
[01:17:17] <MacFanBot>	 [02mw-config] 07SomeMWDev approved pull request #6495 13https://github.com/miraheze/mw-config/pull/6495#pullrequestreview-4945108416
[01:17:27] <MacFanBot>	 [02mw-config] 07SomeMWDev merged 07pixDeVl's pull request #6495: T15830: Disable new ParserMigration defaults (07miraheze:03main...07pixDeVl:03patch-7) 13https://github.com/miraheze/mw-config/pull/6495
[01:17:29] <MacFanBot>	 [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/243ab8035c6155ea4434259c0d91ff82e96af5d7
[01:17:30] <MacFanBot>	 02mw-config/03main 07PixDeVl 03243ab80 T15830: Disable new ParserMigration defaults…
[01:17:34] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[01:17:36] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 2s
[01:17:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:17:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:17:58] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[01:18:12] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[01:18:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:18:39] <miraheze-github>	 miraheze/mw-config - SomeMWDev the build passed.
[01:18:54] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.49 ms
[01:18:56] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mwtask161
[01:19:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:19:18] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.46 ms
[01:19:49] <MirahezeRelay>	 <abaddriverlol> [1/15] ```
[01:19:50] <MirahezeRelay>	 <abaddriverlol> [2/15] Deploying /srv/mediawiki/config/ to mwtask161.
[01:19:50] <MirahezeRelay>	 <abaddriverlol> [3/15] Execute: sudo -u www-data rsync --update -r --delete -e "ssh -i /srv/mediawiki-staging/deploykey" /srv/mediawiki/config/ www-data@mwtask161.fsslc.wtnet:/srv/mediawiki/config/
[01:19:50] <MirahezeRelay>	 <abaddriverlol> [4/15] Completed (0) in 1s!
[01:19:51] <MirahezeRelay>	 <abaddriverlol> [5/15] Status: 503
[01:19:51] <MirahezeRelay>	 <abaddriverlol> [6/15] Text: False 
[01:19:51] <MirahezeRelay>	 <abaddriverlol> [7/15]  <html><body><h1>503 Service Unavailable/h1
[01:19:51] <MirahezeRelay>	 <abaddriverlol> [8/15] No server is available to handle this request.
[01:19:52] <MirahezeRelay>	 <abaddriverlol> [9/15] <script>(function(){function c(){var b=a.contentDocument||(a.contentWindow&&a.contentWindow.document);if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'a2bc9b98dc2b0d15',t:'MTc4Njg0MzEzNA=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getEle
[01:19:52] <MirahezeRelay>	 <abaddriverlol> [10/15] mentsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{
[01:19:52] <MirahezeRelay>	 <abaddriverlol> [11/15] var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();/script/body/html
[01:19:53] <MirahezeRelay>	 <abaddriverlol> [12/15] Debug: False
[01:19:53] <MirahezeRelay>	 <abaddriverlol> [13/15] Canary check failed for publictestwiki.com@mwtask161. Aborting... - use --force to proceed
[01:19:54] <MirahezeRelay>	 <abaddriverlol> [14/15] ```
[01:19:54] <MirahezeRelay>	 <abaddriverlol> [15/15] interesting
[01:20:06] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[01:20:18] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[01:20:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:20:57] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.50 ms
[01:21:01] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mwtask161
[01:21:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:21:18] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.84 ms
[01:22:06] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[01:22:14] <MirahezeRelay>	 <abaddriverlol> Looks like mwtask161 is down
[01:22:36] <MirahezeRelay>	 <abaddriverlol> I get a 503 too when I visit metawiki while proxying to mwtask161
[01:24:44] <MirahezeRelay>	 <abaddriverlol> filed https://issue-tracker.miraheze.org/T15844
[01:26:17] <MirahezeRelay>	 <abaddriverlol> gonna finish the config deployment for now so we don't have undeployed changes on certain servers
[01:26:31] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to mwtask161
[01:26:32] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to mwtask161 - SUCCESS in 1s
[01:26:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:26:45] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to mwtask171
[01:26:47] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to mwtask171 - SUCCESS in 1s
[01:27:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:27:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:27:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:33:06] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.46 ms
[01:34:50] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1786839260000&orgId=1&to=1786843820000
[01:35:58] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures
[01:38:09] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[01:45:57] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 0.50 ms
[01:48:31] <icinga-miraheze>	 PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 20.82, 18.86, 16.58
[01:50:42] <icinga-miraheze>	 PROBLEM - cp161 Current Load on cp161 is CRITICAL: LOAD CRITICAL - total load average: 12.31, 12.06, 8.54
[01:51:26] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.67, 22.37, 19.85
[01:51:59] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.50 ms
[01:52:36] <icinga-miraheze>	 PROBLEM - cp161 Current Load on cp161 is WARNING: LOAD WARNING - total load average: 8.41, 11.07, 8.62
[01:53:21] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.50 ms
[01:54:31] <icinga-miraheze>	 PROBLEM - cp161 Current Load on cp161 is CRITICAL: LOAD CRITICAL - total load average: 44.22, 20.15, 11.96
[01:55:26] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 25.83, 23.79, 20.94
[01:55:30] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.60, 20.37, 18.54
[01:55:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.46, 20.87, 18.77
[01:56:07] <icinga-miraheze>	 PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.83, 20.80, 18.47
[01:56:12] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.46 ms
[01:58:07] <icinga-miraheze>	 RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.52, 20.26, 18.57
[01:58:11] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.48 ms
[01:58:31] <icinga-miraheze>	 RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 20.03, 19.90, 18.35
[02:00:11] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.76 ms
[02:01:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1786842050000&orgId=1&to=1786845680032
[02:01:22] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.43, 19.81, 18.89
[02:01:26] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.89, 23.08, 21.59
[02:02:07] <icinga-miraheze>	 PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.01, 20.67, 19.12
[02:03:25] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 24.69, 23.48, 21.90
[02:04:18] <icinga-miraheze>	 PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 21.61, 20.54, 18.56
[02:04:31] <icinga-miraheze>	 PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.34, 20.62, 19.06
[02:05:13] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.81, 21.29, 19.69
[02:05:33] <icinga-miraheze>	 PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 20.11, 20.48, 19.12
[02:05:56] <icinga-miraheze>	 PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:06:14] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 1.20 ms
[02:06:23] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.48 ms
[02:06:29] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.50 ms
[02:06:31] <icinga-miraheze>	 RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 20.29, 20.09, 19.04
[02:06:35] <icinga-miraheze>	 PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:06:45] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.50 ms
[02:08:32] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[02:08:45] <icinga-miraheze>	 PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.04, 20.63, 19.25
[02:08:48] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.51 ms
[02:09:38] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.50, 20.03, 18.74
[02:09:58] <icinga-miraheze>	 PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:10:29] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.50, 20.57, 19.48
[02:10:31] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[02:10:32] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.38, 20.72, 18.69
[02:12:07] <icinga-miraheze>	 RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.06, 20.25, 19.82
[02:12:33] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.50 ms
[02:12:39] <icinga-miraheze>	 PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 22.49, 20.72, 18.82
[02:13:27] <icinga-miraheze>	 PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.13, 20.25, 18.89
[02:14:04] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:14:09] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.84, 20.49, 18.50
[02:14:31] <icinga-miraheze>	 PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 23.47, 21.04, 19.78
[02:14:47] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.46 ms
[02:15:37] <MirahezeRelay>	 <tsybangd> ok
[02:15:41] <icinga-miraheze>	 PROBLEM - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift-container-stats_mw-media
[02:15:50] <icinga-miraheze>	 PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:16:07] <icinga-miraheze>	 PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.29, 21.49, 20.43
[02:16:21] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[02:16:48] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.50 ms
[02:18:18] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.46 ms
[02:20:49] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.59, 20.02, 18.77
[02:21:41] <icinga-miraheze>	 RECOVERY - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is OK: OK: Status of the systemd unit swift-container-stats_mw-media
[02:23:00] <icinga-miraheze>	 RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 16.17, 19.78, 19.60
[02:24:31] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.49 ms
[02:26:49] <icinga-miraheze>	 PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 23.03, 21.25, 20.21
[02:27:56] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.03, 20.19, 19.84
[02:29:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.15, 23.41, 22.48
[02:30:49] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 20.25, 20.03, 19.47
[02:31:10] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.50 ms
[02:31:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.55, 23.49, 22.63
[02:32:30] <icinga-miraheze>	 RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 18.92, 20.10, 20.03
[02:33:03] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.38, 19.98, 20.31
[02:33:48] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 22.11, 21.19
[02:33:53] <icinga-miraheze>	 RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[02:34:35] <icinga-miraheze>	 RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[02:34:49] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.18, 20.94, 19.97
[02:35:11] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.49 ms
[02:35:44] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 23.16, 22.47, 21.44
[02:36:23] <MirahezeRelay>	 <posix_memalign> Our servers are still pretty overloaded.
[02:36:49] <icinga-miraheze>	 RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 19.22, 20.31, 20.32
[02:36:49] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.64, 20.26, 19.84
[02:38:37] <icinga-miraheze>	 PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.38 ms
[02:38:54] <icinga-miraheze>	 RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 19.49, 19.79, 20.38
[02:39:57] <icinga-miraheze>	 RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:40:35] <icinga-miraheze>	 RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.35 ms
[02:41:26] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 18.56, 22.49, 23.76
[02:42:03] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[02:43:14] <icinga-miraheze>	 RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 13.49, 18.49, 20.38
[02:43:22] <icinga-miraheze>	 PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:43:31] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 13.29, 18.17, 19.99
[02:43:36] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 13.09, 17.47, 19.54
[02:43:37] <icinga-miraheze>	 RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 13.88, 17.42, 19.71
[02:44:06] <icinga-miraheze>	 RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[02:44:31] <icinga-miraheze>	 RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 14.22, 17.80, 19.95
[02:44:39] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.50, 17.53, 19.98
[02:45:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.83, 18.44, 20.29
[02:47:25] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 25.48, 22.14, 22.92
[02:48:26] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.48 ms
[02:48:53] <icinga-miraheze>	 PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp161.wikitide.net port 443 after 0 ms: Could not connect to server
[02:49:25] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 13.18, 19.88, 22.09
[02:49:39] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki
[02:50:07] <icinga-miraheze>	 RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 7.38, 15.83, 19.14
[02:50:19] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mon181 on cp161 is CRITICAL: connect to address localhost and port 8201: Connection refused
[02:50:25] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.38 ms
[02:50:37] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw161 on cp161 is CRITICAL: connect to address localhost and port 8115: Connection refused
[02:50:38] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb
[02:50:39] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw153 on cp161 is CRITICAL: connect to address localhost and port 8121: Connection refused
[02:50:39] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw192 on cp161 is CRITICAL: connect to address localhost and port 8130: Connection refused
[02:50:42] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw171 on cp161 is CRITICAL: connect to address localhost and port 8117: Connection refused
[02:50:45] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw173 on cp161 is CRITICAL: connect to address localhost and port 8125: Connection refused
[02:50:59] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for phorge171 on cp161 is CRITICAL: connect to address localhost and port 8202: Connection refused
[02:51:03] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mwtask181 on cp161 is CRITICAL: connect to address localhost and port 8160: Connection refused
[02:51:09] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mwtask171 on cp161 is CRITICAL: connect to address localhost and port 8161: Connection refused
[02:51:14] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw152 on cp161 is CRITICAL: connect to address localhost and port 8114: Connection refused
[02:51:17] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw193 on cp161 is CRITICAL: connect to address localhost and port 8131: Connection refused
[02:51:18] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw202 on cp161 is CRITICAL: connect to address localhost and port 8133: Connection refused
[02:51:20] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1786844990000&orgId=1&to=1786848590000
[02:51:22] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw163 on cp161 is CRITICAL: connect to address localhost and port 8123: Connection refused
[02:51:25] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for test151 on cp161 is CRITICAL: connect to address localhost and port 8181: Connection refused
[02:51:25] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for swiftproxy171 on cp161 is CRITICAL: connect to address localhost and port 8207: Connection refused
[02:51:25] <icinga-miraheze>	 RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 5.83, 14.90, 19.99
[02:51:27] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw162 on cp161 is CRITICAL: connect to address localhost and port 8116: Connection refused
[02:51:37] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw182 on cp161 is CRITICAL: connect to address localhost and port 8120: Connection refused
[02:51:38] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw151 on cp161 is CRITICAL: connect to address localhost and port 8113: Connection refused
[02:51:40] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw172 on cp161 is CRITICAL: connect to address localhost and port 8118: Connection refused
[02:51:40] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for matomo151 on cp161 is CRITICAL: connect to address localhost and port 8203: Connection refused
[02:51:41] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw201 on cp161 is CRITICAL: connect to address localhost and port 8132: Connection refused
[02:51:42] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for puppet181 on cp161 is CRITICAL: connect to address localhost and port 8204: Connection refused
[02:51:50] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw203 on cp161 is CRITICAL: connect to address localhost and port 8134: Connection refused
[02:51:51] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mwtask161 on cp161 is CRITICAL: connect to address localhost and port 8163: Connection refused
[02:51:55] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 92%
[02:51:56] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb
[02:51:56] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mwtask151 on cp161 is CRITICAL: connect to address localhost and port 8162: Connection refused
[02:51:59] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw183 on cp161 is CRITICAL: connect to address localhost and port 8127: Connection refused
[02:52:03] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for swiftproxy161 on cp161 is CRITICAL: connect to address localhost and port 8206: Connection refused
[02:52:07] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw191 on cp161 is CRITICAL: connect to address localhost and port 8129: Connection refused
[02:52:08] <icinga-miraheze>	 PROBLEM - wikitide.org - LetsEncrypt on sslhost is CRITICAL: connect to address wikitide.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket
[02:52:18] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for mw181 on cp161 is CRITICAL: connect to address localhost and port 8119: Connection refused
[02:52:18] <icinga-miraheze>	 PROBLEM - cp161 Nginx Backend for reports171 on cp161 is CRITICAL: connect to address localhost and port 8205: Connection refused
[02:54:31] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[02:55:25] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for test151 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8181
[02:55:25] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for swiftproxy171 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8207
[02:55:27] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw162 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8116
[02:55:37] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw182 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8120
[02:55:38] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw151 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8113
[02:55:40] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw172 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8118
[02:55:40] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for matomo151 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8203
[02:55:42] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw201 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8132
[02:55:42] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for puppet181 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8204
[02:55:43] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.40 ms
[02:55:50] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw203 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8134
[02:55:51] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mwtask161 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8163
[02:55:56] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[02:55:56] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mwtask151 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8162
[02:55:59] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw183 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8127
[02:56:03] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for swiftproxy161 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8206
[02:56:08] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw191 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8129
[02:56:18] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw181 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8119
[02:56:18] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for reports171 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8205
[02:56:19] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mon181 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8201
[02:56:37] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw161 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8115
[02:56:38] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[02:56:39] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw153 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8121
[02:56:39] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw192 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8130
[02:56:39] <icinga-miraheze>	 RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1539 bytes in 0.792 second response time
[02:56:42] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw171 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8117
[02:56:45] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw173 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8125
[02:56:59] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for phorge171 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8202
[02:57:03] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mwtask181 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8160
[02:57:09] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mwtask171 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8161
[02:57:14] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw152 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8114
[02:57:17] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw193 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8131
[02:57:18] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw202 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8133
[02:57:22] <icinga-miraheze>	 RECOVERY - cp161 Nginx Backend for mw163 on cp161 is OK: TCP OK - 0.000 second response time on localhost port 8123
[02:57:26] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[02:57:43] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[02:57:48] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 13%
[03:00:07] <icinga-miraheze>	 PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.57, 16.32, 16.43
[03:00:29] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[03:00:31] <icinga-miraheze>	 PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 23.33, 16.51, 15.98
[03:00:56] <icinga-miraheze>	 PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 23.36, 16.44, 15.40
[03:01:09] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.10, 17.24, 16.21
[03:01:11] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 4 backends are down. mw172 mw153 mw163 mw191
[03:01:26] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.80, 20.28, 19.06
[03:01:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.22, 17.65, 16.77
[03:02:01] <icinga-miraheze>	 PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[03:02:07] <icinga-miraheze>	 RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 17.96, 16.29, 16.39
[03:02:31] <icinga-miraheze>	 RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 15.55, 15.88, 15.81
[03:02:35] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw191
[03:02:49] <icinga-miraheze>	 PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.39, 17.97, 16.12
[03:02:50] <icinga-miraheze>	 RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 20.23, 17.65, 15.98
[03:03:09] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 20.28, 18.18, 16.69
[03:03:11] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[03:03:36] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw193
[03:04:04] <icinga-miraheze>	 RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 6.921 second response time
[03:04:26] <icinga-miraheze>	 PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 23.15, 18.96, 17.10
[03:04:38] <icinga-miraheze>	 PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 23.50, 20.07, 17.27
[03:05:00] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 23.01, 19.32, 17.17
[03:05:02] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.51, 18.96, 15.63
[03:05:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.58, 19.33, 17.26
[03:05:14] <icinga-miraheze>	 PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 23.33, 20.55, 18.03
[03:05:25] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.17, 19.77, 16.97
[03:06:07] <icinga-miraheze>	 PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.56, 19.62, 17.81
[03:06:20] <icinga-miraheze>	 PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 20.60, 17.92
[03:06:31] <icinga-miraheze>	 PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 20.87, 19.12, 17.21
[03:06:49] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.00, 19.88, 17.01
[03:07:04] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw172
[03:07:06] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.57, 20.18, 16.62
[03:07:07] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.94, 18.24, 17.10
[03:07:09] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.49, 20.58, 18.03
[03:07:11] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw152
[03:07:25] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 25.68, 23.78, 21.03
[03:08:15] <icinga-miraheze>	 RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 18.17, 20.24, 18.14
[03:08:33] <icinga-miraheze>	 PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 23.35, 20.96, 17.92
[03:09:09] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 21.81, 18.79
[03:09:18] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.91, 20.34, 17.41
[03:09:36] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[03:10:21] <icinga-miraheze>	 PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 23.03, 20.75, 17.45
[03:10:49] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.56, 19.14, 16.85
[03:10:51] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 14.36, 18.87, 17.95
[03:10:54] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.18, 19.95, 17.44
[03:10:58] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.14, 20.65, 18.37
[03:11:09] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.41, 21.19, 18.92
[03:11:14] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.96, 19.84, 17.57
[03:11:21] <icinga-miraheze>	 RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:12:04] <icinga-miraheze>	 PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.72, 21.37, 19.04
[03:12:31] <icinga-miraheze>	 RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 18.24, 20.05, 18.40
[03:14:41] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.00, 21.35, 18.27
[03:14:42] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.89, 22.00, 18.81
[03:14:52] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[03:15:08] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.54, 21.28, 18.68
[03:15:11] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[03:16:31] <icinga-miraheze>	 PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 22.24, 19.67
[03:17:25] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.47, 23.98, 22.57
[03:17:27] <icinga-miraheze>	 PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds
[03:18:21] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 57%
[03:18:31] <icinga-miraheze>	 PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 23.94, 22.79, 20.19
[03:18:51] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 23.56, 21.71, 19.60
[03:19:22] <icinga-miraheze>	 PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[03:19:25] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 28.84, 25.51, 23.28
[03:19:29] <icinga-miraheze>	 RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1540 bytes in 6.305 second response time
[03:20:07] <icinga-miraheze>	 RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 16.81, 20.09, 19.62
[03:20:19] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 8%
[03:20:22] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[03:20:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.29, 22.81, 20.43
[03:20:51] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.67, 19.85, 19.19
[03:21:19] <icinga-miraheze>	 RECOVERY - wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Mon 14 Sep 2026 12:54:34 PM GMT +0000.
[03:21:23] <icinga-miraheze>	 RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 2.178 second response time
[03:21:25] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.12, 23.95, 23.00
[03:22:06] <icinga-miraheze>	 RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 15.96, 19.57, 19.19
[03:22:24] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.06, 19.38, 19.18
[03:22:30] <icinga-miraheze>	 RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 16.66, 19.81, 19.69
[03:22:31] <icinga-miraheze>	 RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 17.64, 20.24, 19.75
[03:22:32] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.48, 18.83, 18.71
[03:22:37] <icinga-miraheze>	 RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 14.20, 18.53, 19.24
[03:22:41] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.74, 21.21, 20.15
[03:22:49] <icinga-miraheze>	 RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 14.66, 20.31, 20.03
[03:22:49] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 14.58, 20.06, 19.95
[03:22:52] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.35, 18.83, 18.95
[03:23:03] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.13, 19.39, 19.67
[03:23:09] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.50, 19.80, 19.76
[03:23:36] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[03:23:37] <icinga-miraheze>	 RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 16.29, 19.80, 20.00
[03:24:39] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.32, 19.70, 19.76
[03:25:14] <icinga-miraheze>	 RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 15.71, 18.53, 19.55
[03:25:36] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[03:25:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.66, 18.43, 19.58
[03:26:51] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1786847180000&orgId=1&to=1786850810035
[03:28:29] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[03:30:25] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/d5e843020c7a6e5d762162fb9c69ca7b90756b57
[03:30:25] <MacFanBot>	 02ssl/03main 07WikiTideBot 03d5e8430 Bot: Auto-update domain lists
[03:30:28] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[03:39:25] <icinga-miraheze>	 RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 20.24, 19.29, 20.17
[03:45:25] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.99, 20.31, 20.24
[03:46:01] <icinga-miraheze>	 PROBLEM - cp161 Current Load on cp161 is WARNING: LOAD WARNING - total load average: 5.79, 6.34, 11.99
[03:47:25] <icinga-miraheze>	 RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 17.20, 19.36, 19.93
[03:49:14] <MacFanBot>	 [02dns] 07The-Voidwalker pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/a11661fd65ce1d61c3d7d245eb8f83a4c8dd739d
[03:49:14] <MacFanBot>	 02dns/03main 07The-Voidwalker 03a11661f point homevideo to cf-lb…
[03:50:09] <miraheze-github>	 miraheze/dns - The-Voidwalker the build passed.
[03:52:01] <icinga-miraheze>	 RECOVERY - cp161 Current Load on cp161 is OK: LOAD OK - total load average: 4.74, 5.33, 9.73
[03:56:50] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1786848830000&orgId=1&to=1786852430000
[04:10:28] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f6cdc89d1ce085dcf2a4fb5dc0e513594645b32c
[04:10:28] <MacFanBot>	 02ssl/03main 07WikiTideBot 03f6cdc89 Bot: Auto-update domain lists
[04:16:27] <MacFanBot>	 [02dns] 07The-Voidwalker pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/3d86aaa8ebbb875a485a12bad3210591da19d47d
[04:16:27] <MacFanBot>	 02dns/03main 07The-Voidwalker 033d86aaa add acme-challenges for homevideo to finish CF setup
[04:17:15] <miraheze-github>	 miraheze/dns - The-Voidwalker the build passed.
[04:28:10] <MacFanBot>	 [02dns] 07The-Voidwalker pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/c89d6c24397015d4aec590b86837e01daff1708e
[04:28:10] <MacFanBot>	 02dns/03main 07The-Voidwalker 03c89d6c2 coreection
[04:29:01] <miraheze-github>	 miraheze/dns - The-Voidwalker the build passed.
[04:41:36] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php updateSpecialPages --wiki=teflpediawiki (END - exit=0)
[04:41:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:45:42] <MirahezeLSBot>	 !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php Bucket:CreateInitialSchemaForBucketIssues --wiki=shminerwiki (END - exit=0)
[11:45:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:53:55] <MirahezeLSBot>	 !log [petramagna@test151] starting deploy of {'folders': '1.46/extensions/ConfirmEdit'} to test151
[12:53:56] <MirahezeLSBot>	 !log [petramagna@test151] finished deploy of {'folders': '1.46/extensions/ConfirmEdit'} to test151 - SUCCESS in 0s
[12:53:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:54:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:52:43] <MirahezeLSBot>	 !log [wwr@mwtask171] Starting import for danballwiki (XML: None; Images: ./images) (START)
[13:52:44] <MirahezeLSBot>	 !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=danballwiki --sleep=1 '--comment=Importing images from https://danball.fandom.com ([[phorge:T15843|T15843]])' -- ./images (START)
[13:52:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:52:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:41:34] <MirahezeLSBot>	 !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=danballwiki --sleep=1 '--comment=Importing images from https://danball.fandom.com ([[phorge:T15843|T15843]])' -- ./images (END - exit=0)
[15:41:35] <MirahezeLSBot>	 !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=danballwiki --update (START)
[15:41:36] <MirahezeLSBot>	 !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=danballwiki --update (END - exit=0)
[15:41:37] <MirahezeLSBot>	 !log [wwr@mwtask171] Finished import for danballwiki (XML: None; Images: ./images) (END - exit=0)
[15:41:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:41:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:41:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:41:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:00:19] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/3a0d7523b509bd452d74f34c7a3f8fff25f3144b
[16:00:20] <MacFanBot>	 02ssl/03main 07WikiTideBot 033a0d752 Bot: Auto-update domain lists
[17:26:23] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1786897550000&orgId=1&to=1786901183350
[17:28:09] <RhinosF1>	 That's a weird one
[17:30:24] <RhinosF1>	 It's refreshLinks
[17:31:45] <RhinosF1>	 I left a note on MM
[17:36:23] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1786897910000&orgId=1&to=1786901510000
[17:56:23] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1786899350000&orgId=1&to=1786902983351
[18:00:19] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/995b6dfc6a1d3db47ecf9438b246f6a60bd6a628
[18:00:19] <MacFanBot>	 02ssl/03main 07WikiTideBot 03995b6df Bot: Auto-update domain lists
[18:09:05] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.38 ms
[18:09:46] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[18:11:46] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[18:13:03] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.38 ms
[18:16:23] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1786900310000&orgId=1&to=1786903910000
[18:17:38] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.44 ms
[18:21:37] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[18:29:58] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.38 ms
[18:33:59] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.52 ms
[18:35:18] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.41 ms
[18:37:16] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.45 ms
[18:37:18] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[18:43:15] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[18:51:56] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 0.50 ms
[18:55:54] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.43 ms
[18:55:55] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[18:56:17] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.47 ms
[18:57:58] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.47 ms
[18:58:16] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[18:59:14] <icinga-miraheze>	 PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:59:16] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.43 ms
[19:01:16] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[19:02:25] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.42 ms
[19:04:23] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.45 ms
[19:04:24] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[19:05:32] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 0.43 ms
[19:07:32] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[19:08:23] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.58 ms
[19:09:57] <icinga-miraheze>	 PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:11:32] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[19:12:30] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.44 ms
[19:13:54] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[19:15:54] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.66 ms
[19:16:33] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.48 ms
[19:17:45] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[19:20:13] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.43 ms
[19:20:43] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.41 ms
[19:21:28] <icinga-miraheze>	 PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png]
[19:22:00] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.45 ms
[19:22:42] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[19:24:43] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[19:26:42] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[19:27:08] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.46 ms
[19:28:14] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[19:29:08] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[19:30:19] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[19:32:59] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.42 ms
[19:33:31] <MirahezeRelay>	 <kevin.lee770> https://cdn.discordapp.com/attachments/808001911868489748/1538631816473350205/image.jpg?ex=6a83620b&is=6a82108b&hm=f79d01156facb3f92a392d3139e1fdf56511e26148f0b59b0e967ed7e1cb77b2&
[19:34:34] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.47 ms
[19:34:59] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[19:36:29] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.42 ms
[19:37:16] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.42 ms
[19:37:57] <icinga-miraheze>	 RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[19:38:28] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.42 ms
[19:39:16] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[19:40:36] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[19:40:48] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.47 ms
[19:40:50] <icinga-miraheze>	 PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:42:37] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[19:43:25] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.48 ms
[19:44:08] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[19:45:25] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.48 ms
[19:46:46] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[19:47:27] <icinga-miraheze>	 RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[19:50:09] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[19:51:58] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.45 ms
[19:53:28] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.48 ms
[19:54:55] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[19:55:32] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[19:58:14] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[19:59:10] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 0.44 ms
[19:59:15] <icinga-miraheze>	 RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:59:48] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 0.44 ms
[20:02:00] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.40 ms
[20:03:11] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.42 ms
[20:03:56] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.51 ms
[20:05:49] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.42 ms
[20:05:55] <icinga-miraheze>	 PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png]
[20:06:17] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[20:10:15] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.43 ms
[20:10:51] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:11:57] <icinga-miraheze>	 PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:13:31] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[20:13:50] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.40 ms
[20:15:31] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.42 ms
[20:15:32] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.44 ms
[20:16:36] <icinga-miraheze>	 PROBLEM - swiftobject181 Puppet on swiftobject181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:17:31] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[20:18:14] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.41 ms
[20:20:13] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.40 ms
[20:21:55] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[20:22:42] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[20:23:39] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[20:23:55] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[20:23:56] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[20:27:51] <MirahezeLSBot>	 !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=dungeondefendersawakenedofficialwikiwiki --new=ddawikiwiki --user=TheWWRNerdGuy (END - exit=0)
[20:27:56] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[20:28:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:28:25] <icinga-miraheze>	 PROBLEM - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift-container-stats_mw-media
[20:30:23] <icinga-miraheze>	 RECOVERY - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is OK: OK: Status of the systemd unit swift-container-stats_mw-media
[20:31:43] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[20:33:26] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:33:53] <icinga-miraheze>	 RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[20:33:54] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[20:35:52] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.46 ms
[20:37:51] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[20:37:55] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[20:38:44] <icinga-miraheze>	 PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:39:57] <icinga-miraheze>	 RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[20:43:22] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[20:44:36] <icinga-miraheze>	 RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:45:21] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[20:45:38] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.41 ms
[20:51:39] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.42 ms
[20:51:45] <icinga-miraheze>	 PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.31 ms
[20:52:03] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.44 ms
[20:53:27] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.44 ms
[20:53:31] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[20:53:42] <icinga-miraheze>	 RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.29 ms
[20:54:07] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[20:55:31] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.42 ms
[20:55:44] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[20:57:47] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.60 ms
[20:58:16] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[21:00:09] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[21:01:26] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[21:01:31] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.40 ms
[21:02:09] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[21:02:17] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.42 ms
[21:02:51] <icinga-miraheze>	 PROBLEM - os191 Puppet on os191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:04:57] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.42 ms
[21:06:44] <icinga-miraheze>	 RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[21:08:56] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[21:10:49] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.47 ms
[21:12:53] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[21:12:57] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.47 ms
[21:14:57] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[21:15:26] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 1.14 ms
[21:17:19] <icinga-miraheze>	 PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_phorge-extensions]
[21:18:54] <icinga-miraheze>	 PROBLEM - mem151 Puppet on mem151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:21:28] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.40 ms
[21:26:39] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.42 ms
[21:30:51] <icinga-miraheze>	 RECOVERY - os191 Puppet on os191 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[21:32:41] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[21:33:05] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.49 ms
[21:35:09] <icinga-miraheze>	 PROBLEM - cloud20 Puppet on cloud20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:36:53] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[21:36:56] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[21:40:54] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[21:41:03] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[21:43:08] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.40 ms
[21:44:52] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[21:45:11] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[21:45:16] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:45:17] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[21:45:21] <icinga-miraheze>	 PROBLEM - mw173 Puppet on mw173 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:46:52] <icinga-miraheze>	 RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[21:47:08] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.40 ms
[21:47:09] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.37 ms
[21:48:52] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[21:55:16] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[21:56:17] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.41 ms
[22:00:00] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[22:02:16] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[22:02:57] <MirahezeLSBot>	 !log [wwr@mwtask171] swift delete miraheze-dungeondefendersawakenedofficialwikiwiki-local-deleted
[22:03:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:03:09] <icinga-miraheze>	 RECOVERY - cloud20 Puppet on cloud20 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[22:03:17] <MirahezeLSBot>	 !log [wwr@mwtask171] swift delete miraheze-dungeondefendersawakenedofficialwikiwiki-local-public (failed)
[22:03:22] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.47 ms
[22:03:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:07:22] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.47 ms
[22:08:02] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[22:09:34] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.42 ms
[22:11:20] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[22:11:59] <icinga-miraheze>	 PROBLEM - db172 Puppet on db172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[22:12:08] <icinga-miraheze>	 PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[22:12:10] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.45 ms
[22:14:03] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png]
[22:15:30] <icinga-miraheze>	 RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[22:17:27] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.43 ms
[22:17:38] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.70 ms
[22:18:04] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.47 ms
[22:18:10] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[22:19:25] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.42 ms
[22:27:27] <icinga-miraheze>	 PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[22:28:41] <icinga-miraheze>	 PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[22:30:20] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/835bcd2c2b4e894c7c58a39954ab98f64f7b4dcb
[22:30:20] <MacFanBot>	 02ssl/03main 07WikiTideBot 03835bcd2 Bot: Auto-update domain lists
[22:32:23] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.48 ms
[22:34:15] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.91 ms
[22:34:27] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[22:35:03] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.42 ms
[22:37:04] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[22:40:04] <icinga-miraheze>	 RECOVERY - db172 Puppet on db172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:40:08] <icinga-miraheze>	 RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[22:40:44] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.89 ms
[22:41:16] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.44 ms
[22:41:32] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[22:42:03] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[22:42:48] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.47 ms
[22:43:31] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[22:45:18] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[22:45:34] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[22:48:42] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.42 ms
[22:49:34] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[22:50:43] <MirahezeLSBot>	 !log [wwr@mwtask171] swift delete miraheze-dungeondefendersawakenedofficialwikiwiki-local-public
[22:50:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:50:52] <MirahezeLSBot>	 !log [wwr@mwtask171] swift delete miraheze-dungeondefendersawakenedofficialwikiwiki-local-thumb
[22:50:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:51:42] <MirahezeLSBot>	 !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CreateWiki:SetContainersAccess --wiki=ddawikiwiki (END - exit=0)
[22:51:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:52:46] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.47 ms
[22:55:27] <icinga-miraheze>	 RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[22:56:41] <icinga-miraheze>	 RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[22:57:22] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[22:58:43] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[22:58:50] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[23:00:51] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[23:02:50] <icinga-miraheze>	 PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:02:51] <icinga-miraheze>	 PROBLEM - os191 Puppet on os191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:02:59] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[23:05:41] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[23:06:12] <icinga-miraheze>	 PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.32 ms
[23:06:52] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.47 ms
[23:08:10] <icinga-miraheze>	 RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.31 ms
[23:08:56] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[23:09:43] <icinga-miraheze>	 PROBLEM - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift-container-stats_mw-media
[23:10:55] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[23:11:38] <icinga-miraheze>	 RECOVERY - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is OK: OK: Status of the systemd unit swift-container-stats_mw-media
[23:11:43] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[23:15:07] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[23:17:20] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.48 ms
[23:19:21] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[23:21:15] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[23:21:37] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[23:22:15] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.44 ms
[23:23:08] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.42 ms
[23:24:15] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[23:25:14] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.41 ms
[23:27:31] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.42 ms
[23:28:41] <icinga-miraheze>	 PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:29:08] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.40 ms
[23:29:32] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[23:30:50] <icinga-miraheze>	 RECOVERY - os191 Puppet on os191 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[23:30:54] <icinga-miraheze>	 RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[23:33:15] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.43 ms
[23:33:19] <icinga-miraheze>	 PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.45 ms
[23:35:14] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.43 ms
[23:35:47] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 0.45 ms
[23:39:20] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.46 ms
[23:39:42] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.44 ms
[23:39:44] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.47 ms
[23:41:19] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[23:41:50] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.42 ms
[23:43:52] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[23:45:45] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.52 ms
[23:47:25] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.46 ms
[23:47:25] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.43 ms
[23:47:46] <icinga-miraheze>	 PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.32 ms
[23:48:48] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.43 ms
[23:49:44] <icinga-miraheze>	 RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.35 ms
[23:51:26] <icinga-miraheze>	 RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 0.42 ms
[23:52:45] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.47 ms
[23:53:15] <icinga-miraheze>	 PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.48 ms
[23:55:31] <icinga-miraheze>	 PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.44 ms
[23:56:47] <icinga-miraheze>	 PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:56:53] <icinga-miraheze>	 PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 0.49 ms
[23:57:23] <icinga-miraheze>	 RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.45 ms
[23:58:41] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures