[00:11:05] PROBLEM - dccomicswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dccomicswiki.com' expires in 14 day(s) (Fri 28 Feb 2025 11:41:34 PM GMT +0000). [00:14:10] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 52526MiB (11% inode=99%); [00:48:40] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [00:54:44] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.40 ms [01:02:58] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [01:04:56] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.36 ms [01:50:10] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49838MiB (10% inode=99%); [03:44:04] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.32, 14.96, 7.33 [03:51:54] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 11.06, 17.38, 11.98 [04:47:36] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [04:53:18] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 62.50, 23.76, 9.02 [04:57:47] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.42 ms [04:59:18] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.70, 8.91, 7.30 [05:04:03] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [05:10:10] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.37 ms [05:47:56] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [05:51:24] !log swift delete miraheze-removededmsongswiki-local-transcoded [05:51:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:54:38] !log swift delete miraheze-animatedmusclewomenwiki-local-thumb (not deleted when the wiki was?) [05:54:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:08] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.33 ms [06:02:26] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [06:05:07] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/MirahezeMagic/maintenance/checkSwiftContainers.php --wiki=metawiki --delete (START) [06:05:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:06:34] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/MirahezeMagic/commit/e4ccf34f0009425a5644b5a864b9cbae7a06b1b3 [06:06:34] 02MirahezeMagic/03master 07CosmicAlpha 03e4ccf34 Skip dpl_clview when renaming tables… [06:14:33] !log [reception@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'MirahezeMagic'} to all [06:14:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:43] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.35 ms [06:14:54] !log [reception@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 21s [06:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:03] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/MirahezeMagic/maintenance/renameDatabase.php --wiki=loginwiki --old=wildcardsrchivewiki --new=wildcardarchivewiki (END - exit=0) [06:15:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:16:00] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/MirahezeMagic/maintenance/checkSwiftContainers.php --wiki=metawiki --delete (END - exit=0) [06:16:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:17:34] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/MirahezeMagic/maintenance/renameDatabase.php --wiki=loginwiki --old=tyrillianencyclopediawiki --new=tyrillymwiki (END - exit=0) [06:17:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:18:40] miraheze/MirahezeMagic - Universal-Omega the build passed. [06:19:01] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [06:20:59] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.35 ms [07:02:52] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [07:13:03] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.35 ms [07:19:19] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [07:24:05] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.44, 15.68, 8.10 [07:25:25] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.36 ms [07:28:05] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 9.86, 15.81, 10.15 [08:18:19] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [08:24:23] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.35 ms [08:53:05] !log [universalomega@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [08:53:06] !log [universalomega@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [08:53:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:53:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:54:40] !log [universalomega@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_pack': 'wikitide'} to test151 [08:54:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:54:59] !log [universalomega@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_pack': 'wikitide'} to test151 - SUCCESS in 18s [08:55:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:58:50] !log [universalomega@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_pack': 'universalomega'} to test151 [08:58:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:59:00] !log [universalomega@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_pack': 'universalomega'} to test151 - SUCCESS in 10s [08:59:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:26:01] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 58% [09:26:03] PROBLEM - mw174 HTTPS on mw174 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:26:08] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:26:15] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:26:15] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [09:26:16] PROBLEM - mw164 HTTPS on mw164 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:26:17] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.015 second response time [09:26:18] PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:26:20] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [09:26:21] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:26:22] PROBLEM - mw184 HTTPS on mw184 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:26:22] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:26:36] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:26:37] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:26:44] PROBLEM - mw154 HTTPS on mw154 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:26:45] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [09:26:47] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:26:47] PROBLEM - mw154 MediaWiki Rendering on mw154 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:26:52] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:26:54] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:26:58] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:27:02] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:27:03] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:27:03] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:27:09] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [09:27:11] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:27:11] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:27:27] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:27:32] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:38] PROBLEM - mw164 MediaWiki Rendering on mw164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:41] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:27:52] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [09:27:53] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:53] holy... [09:27:54] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:27:56] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:57] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 373.38, 165.44, 65.78 [09:28:02] PROBLEM - mw174 MediaWiki Rendering on mw174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:11] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [09:29:44] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:30:21] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:30:50] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.460 second response time [09:30:52] RECOVERY - mw154 HTTPS on mw154 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 2.951 second response time [09:30:55] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.198 second response time [09:30:57] RECOVERY - mw154 MediaWiki Rendering on mw154 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.150 second response time [09:30:58] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.053 second response time [09:30:59] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.201 second response time [09:31:06] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.053 second response time [09:31:08] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.172 second response time [09:31:10] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.060 second response time [09:31:10] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.058 second response time [09:31:19] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.060 second response time [09:31:21] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time [09:31:34] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.055 second response time [09:33:46] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.413 second response time [09:33:53] RECOVERY - mw164 MediaWiki Rendering on mw164 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.142 second response time [09:33:54] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.419 second response time [09:33:58] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.161 second response time [09:33:59] PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect [09:33:59] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.047 second response time [09:34:01] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 24% [09:34:01] RECOVERY - mw174 HTTPS on mw174 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.053 second response time [09:34:02] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.054 second response time [09:34:06] RECOVERY - mw174 MediaWiki Rendering on mw174 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.159 second response time [09:34:10] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.179 second response time [09:34:10] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.162 second response time [09:34:14] RECOVERY - mw164 HTTPS on mw164 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.048 second response time [09:34:15] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 0.053 second response time [09:34:20] RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.177 second response time [09:34:25] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.057 second response time [09:34:29] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.159 second response time [09:34:29] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.161 second response time [09:34:31] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.163 second response time [09:34:35] RECOVERY - mw184 HTTPS on mw184 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.063 second response time [09:34:35] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.054 second response time [09:34:38] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.056 second response time [09:34:45] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [09:34:57] PROBLEM - db161 MariaDB on db161 is CRITICAL: Can't connect to server on 'db161.wikitide.net' (115) [09:35:09] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4115 bytes in 0.055 second response time [09:36:11] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [09:56:15] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 59 minutes ago with 0 failures [09:57:42] RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 38.7%Current connections: 387 [09:58:21] RECOVERY - db161 MariaDB on db161 is OK: Uptime: 101 Threads: 387 Questions: 1494 Slow queries: 0 Opens: 163 Open tables: 157 Queries per second avg: 14.792 [09:59:21] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:59:44] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:59:47] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:59:48] PROBLEM - mw164 MediaWiki Rendering on mw164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:59:56] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:59:58] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:59:59] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:01] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 44% [10:00:02] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:00:02] PROBLEM - mw174 MediaWiki Rendering on mw174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:04] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:00:10] PROBLEM - mw174 HTTPS on mw174 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:00:11] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [10:00:12] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:15] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [10:00:21] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:22] PROBLEM - mw164 HTTPS on mw164 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:00:25] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:00:28] PROBLEM - mw184 MediaWiki Rendering on mw184 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:30] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:32] PROBLEM - mw184 HTTPS on mw184 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:00:32] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:00:46] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki [10:00:53] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:01:09] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:01:12] RECOVERY - db161 APT on db161 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [10:01:21] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.180 second response time [10:01:42] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.057 second response time [10:01:43] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.062 second response time [10:01:46] RECOVERY - mw164 MediaWiki Rendering on mw164 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.177 second response time [10:01:54] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.153 second response time [10:01:54] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.157 second response time [10:01:57] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.174 second response time [10:01:58] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.056 second response time [10:02:00] RECOVERY - mw174 MediaWiki Rendering on mw174 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.153 second response time [10:02:01] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 6% [10:02:03] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.058 second response time [10:02:04] RECOVERY - mw174 HTTPS on mw174 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.047 second response time [10:02:07] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.167 second response time [10:02:11] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [10:02:15] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 0.064 second response time [10:02:16] RECOVERY - mw164 HTTPS on mw164 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.049 second response time [10:02:19] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.149 second response time [10:02:22] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.055 second response time [10:02:22] RECOVERY - mw184 MediaWiki Rendering on mw184 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.164 second response time [10:02:27] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.160 second response time [10:02:30] RECOVERY - mw184 HTTPS on mw184 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.055 second response time [10:02:30] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.047 second response time [10:02:46] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [10:02:47] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3951 bytes in 0.053 second response time [10:03:09] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4113 bytes in 0.060 second response time [10:16:03] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.24, 15.46, 7.27 [10:20:03] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 9.79, 16.10, 9.69 [10:46:38] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [10:54:48] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.35 ms [10:59:02] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [11:01:55] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 0.45, 0.49, 11.38 [11:03:55] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 0.56, 0.57, 10.09 [11:13:23] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.49 ms [11:17:41] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [11:27:55] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.36 ms [12:03:55] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [12:07:11] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.25, 8.75, 4.16 [12:07:18] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 20.48, 16.80, 7.85 [12:09:06] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 6 backends are down. mw161 mw171 mw173 mw174 mw183 mw184 [12:09:18] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 3.09, 11.38, 6.95 [12:09:38] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 8 backends are down. mw151 mw152 mw171 mw172 mw154 mw163 mw164 mw174 [12:11:18] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.79, 7.76, 6.15 [12:14:06] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.42 ms [12:14:25] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:15:59] [02mw-config] 07BlankEclair opened pull request #5821: T13233: Add custom footer icons for fraudulentfronterawiki (07miraheze:03master...07BlankEclair:03T13233) 13https://github.com/miraheze/mw-config/pull/5821 [12:16:54] miraheze/mw-config - BlankEclair the build passed. [12:18:29] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4136 bytes in 0.066 second response time [12:19:11] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.39, 10.57, 8.82 [12:20:22] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [12:21:11] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.43, 11.34, 9.28 [12:23:11] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.22, 10.29, 9.13 [12:25:11] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 17.24, 13.37, 10.41 [12:26:27] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.49 ms [12:32:43] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [12:34:47] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy [12:35:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:35:03] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 29 backends are healthy [12:35:11] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.30, 11.37, 11.25 [12:36:32] !log [paladox@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'Cargo'} to all [12:36:35] !log [paladox@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'Cargo'} to all - SUCCESS in 2s [12:36:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:36:46] !log [paladox@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'Cargo'} to all [12:36:47] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.64, 20.43, 15.97 [12:36:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:34] !log [paladox@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'Cargo'} to all - SUCCESS in 48s [12:37:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:39:11] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.73, 6.97, 9.50 [12:40:47] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.07, 20.27, 17.03 [12:42:53] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.42 ms [12:47:11] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 19.07, 13.77, 11.29 [12:49:52] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 14.10, 6.44 [12:50:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:51:11] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.29, 11.95, 11.10 [12:53:06] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 31.37 ms [12:53:52] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.53, 18.93, 10.22 [12:55:11] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.33, 7.20, 9.50 [12:57:03] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.36 ms [12:57:52] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 6.01, 14.47, 10.63 [13:01:22] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:10:33] !log [paladox@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'Cargo'} to all [13:10:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:10:53] !log [paladox@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'Cargo'} to all - SUCCESS in 19s [13:10:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:11:36] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.33 ms [13:15:54] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:19:47] !log [paladox@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'Cargo'} to all [13:19:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:20:06] !log [paladox@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'Cargo'} to all - SUCCESS in 19s [13:20:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:26:08] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.36 ms [13:53:33] Now what [14:12:36] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 140.24 ms [14:14:39] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 140.24 ms [14:15:39] Oh good Claire went to sleep [14:15:42] 1am is ok [14:16:05] Morning @pixldev [14:20:10] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27156MiB (5% inode=99%); [14:34:52] Morning [14:42:10] What cable did you unplug [14:44:04] I didn’t do nothing I’ve spent the last 2 hours locking in on math [14:44:47] You tried the GCSE paper yet? [14:45:37] No I’ve been too busy with the American work :sobbing: [14:46:29] Aww [15:00:36] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.42 ms [15:02:34] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.40 ms [15:12:47] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [15:18:53] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.42 ms [15:29:06] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [15:35:10] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.39 ms [15:36:17] !log [paladox@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'Cargo'} to all [15:36:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:36:38] !log [paladox@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'Cargo'} to all - SUCCESS in 20s [15:36:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:36:49] !log [paladox@test151] starting deploy of {'force_upgrade': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': 'Cargo'} to test151 [15:36:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:36:51] !log [paladox@test151] finished deploy of {'force_upgrade': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': 'Cargo'} to test151 - SUCCESS in 1s [15:36:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:53:45] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.57, 15.57, 8.63 [15:55:45] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 4.88, 11.76, 8.12 [16:39:38] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:45:41] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.36 ms [16:51:57] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:00:06] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.41 ms [17:46:52] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:52:58] [02puppet] 07Universal-Omega created 03remove-graphite from 03master (+0 new commit) 13https://github.com/miraheze/puppet/compare/remove-graphite [17:59:08] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.40 ms [18:06:36] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [18:09:33] [02puppet] 07Universal-Omega pushed 1 new commit to 03remove-graphite 13https://github.com/miraheze/puppet/commit/0a3330119764cba58fb282ad07c9b4b3fcb84a40 [18:09:33] 02puppet/03remove-graphite 07CosmicAlpha 030a33301 Remove graphite [18:09:46] [02puppet] 07Universal-Omega opened pull request #4188: Remove graphite (03master...03remove-graphite) 13https://github.com/miraheze/puppet/pull/4188 [18:11:00] miraheze/puppet - Universal-Omega the build passed. [18:18:35] [02puppet] 07coderabbitai[bot] commented on pull request #4188: --- […] 13https://github.com/miraheze/puppet/pull/4188#issuecomment-2659986569 [18:18:52] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.30 ms [18:24:44] [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/36cd8241d00c0010431591ab79968d88f1a89659 [18:24:44] 02puppet/03master 07CosmicAlpha 0336cd824 Remove graphite (#4188) [18:24:44] [02puppet] 07Universal-Omega merged pull request #4188: Remove graphite (03master...03remove-graphite) 13https://github.com/miraheze/puppet/pull/4188 [18:24:46] [02puppet] 07Universal-Omega 04deleted 03remove-graphite at 030a33301 13https://github.com/miraheze/puppet/commit/0a33301 [18:26:00] miraheze/puppet - Universal-Omega the build passed. [18:27:32] !log remove graphite151 from puppet [18:27:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:32:14] PROBLEM - graphite151 Puppet on graphite151 is CRITICAL: connect to address 10.0.15.145 port 5666: No route to hostconnect to host 10.0.15.145 port 5666: No route to host [18:32:14] PROBLEM - graphite151 ferm_active on graphite151 is CRITICAL: connect to address 10.0.15.145 port 5666: No route to hostconnect to host 10.0.15.145 port 5666: No route to host [18:32:14] PROBLEM - graphite151 Disk Space on graphite151 is CRITICAL: connect to address 10.0.15.145 port 5666: No route to hostconnect to host 10.0.15.145 port 5666: No route to host [18:32:17] PROBLEM - graphite151 NTP time on graphite151 is CRITICAL: connect to address 10.0.15.145 port 5666: No route to hostconnect to host 10.0.15.145 port 5666: No route to host [18:32:25] PROBLEM - graphite151 SSH on graphite151 is CRITICAL: connect to address 10.0.15.145 and port 22: No route to host [18:32:36] PROBLEM - graphite151 statsd-proxy process on graphite151 is CRITICAL: connect to address 10.0.15.145 port 5666: No route to hostconnect to host 10.0.15.145 port 5666: No route to host [18:32:48] PROBLEM - graphite151 APT on graphite151 is CRITICAL: connect to address 10.0.15.145 port 5666: No route to hostconnect to host 10.0.15.145 port 5666: No route to host [18:32:51] PROBLEM - graphite151 PowerDNS Recursor on graphite151 is CRITICAL: connect to address 10.0.15.145 port 5666: No route to hostconnect to host 10.0.15.145 port 5666: No route to host [18:32:57] PROBLEM - ping on graphite151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.145) [18:32:57] PROBLEM - Host graphite151 is DOWN: CRITICAL - Host Unreachable (10.0.15.145) [18:36:00] !log destroy graphite151 VM [18:36:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:38:08] [02dns] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/3408354c4f48cfbd06b027ec7872181fa2eac455 [18:38:08] 02dns/03master 07CosmicAlpha 033408354 Remove graphite151 from DNS [18:39:54] miraheze/dns - Universal-Omega the build passed. [18:48:46] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:48:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:49:07] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 20s [18:49:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:49:36] !log [universalomega@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_pack': 'wikitide'} to all [18:49:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:24] !log [universalomega@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_pack': 'wikitide'} to all - SUCCESS in 227s [18:53:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:46] [02mw-config] 07Universal-Omega merged 07BlankEclair's pull request #5821: T13233: Add custom footer icons for fraudulentfronterawiki (07miraheze:03master...07BlankEclair:03T13233) 13https://github.com/miraheze/mw-config/pull/5821 [18:59:46] [02mw-config] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/d71600d56a12fd0308aca8f72688864a6838e392 [18:59:47] 02mw-config/03master 07Claire 03d71600d T13233: Add custom footer icons for fraudulentfronterawiki (#5821)… [19:00:42] miraheze/mw-config - Universal-Omega the build passed. [19:00:49] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:00:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:09] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 20s [19:01:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:03] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 17.79, 11.77, 4.95 [19:08:02] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 2.86, 8.02, 4.39 [19:27:40] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to matomo151.wikitide.net port 443 after 0 ms: Couldn't connect to server [19:32:36] [02CreateWiki] 07AgentIsai created 03logging-adjust (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/ebf075877f96 [19:32:36] 02CreateWiki/03logging-adjust 07Agent Isai 03ebf0758 Adjust logging for AI… [19:32:41] [02CreateWiki] 07AgentIsai opened pull request #658: Adjust logging for AI (03master...03logging-adjust) 13https://github.com/miraheze/CreateWiki/pull/658 [19:35:05] [02CreateWiki] 07github-actions[bot] pushed 1 new commit to 03logging-adjust 13https://github.com/miraheze/CreateWiki/commit/61257841f65b4626be73758d79cd6ee13bb99cdb [19:35:06] 02CreateWiki/03logging-adjust 07github-actions 036125784 CI: lint code to MediaWiki standards… [19:35:13] [02CreateWiki] 07coderabbitai[bot] commented on pull request #658: --- […] 13https://github.com/miraheze/CreateWiki/pull/658#issuecomment-2660119164 [19:36:35] miraheze/CreateWiki - AgentIsai the build has errored. [19:38:44] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.62, 20.03, 12.35 [19:40:44] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 6.87, 15.92, 11.85 [19:55:29] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 1.499 second response time [20:13:34] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.81, 18.20, 8.98 [20:19:34] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.83, 17.88, 12.56 [20:21:45] [02ImportDump] 07Universal-Omega created 03namespaced from 03master (+0 new commit) 13https://github.com/miraheze/ImportDump/compare/namespaced [20:22:33] [02ImportDump] 07Universal-Omega pushed 1 new commit to 03namespaced 13https://github.com/miraheze/ImportDump/commit/95585f6537cfb884d576e333b4b2ccea19606fce [20:22:33] 02ImportDump/03namespaced 07CosmicAlpha 0395585f6 Use namespaced classes [20:22:46] [02ImportDump] 07Universal-Omega opened pull request #130: Use namespaced classes (03master...03namespaced) 13https://github.com/miraheze/ImportDump/pull/130 [20:22:53] [02ImportDump] 07coderabbitai[bot] commented on pull request #130: --- […] 13https://github.com/miraheze/ImportDump/pull/130#issuecomment-2660195231 [20:24:36] [02ImportDump] 07github-actions[bot] pushed 1 new commit to 03namespaced 13https://github.com/miraheze/ImportDump/commit/a6e5abfd3fabad6ad20c5a8295838283405d123e [20:24:36] 02ImportDump/03namespaced 07github-actions 03a6e5abf CI: lint code to MediaWiki standards… [20:26:39] [02ImportDump] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/ImportDump/commit/12e65df4d9b319e76aaa178264113a992bbc223f [20:26:39] 02ImportDump/03master 07CosmicAlpha 0312e65df Use namespaced classes (#130) [20:26:39] [02ImportDump] 07Universal-Omega merged pull request #130: Use namespaced classes (03master...03namespaced) 13https://github.com/miraheze/ImportDump/pull/130 [20:26:40] [02ImportDump] 07Universal-Omega 04deleted 03namespaced at 03a6e5abf 13https://github.com/miraheze/ImportDump/commit/a6e5abf [20:34:15] miraheze/ImportDump - Universal-Omega the build passed. [20:38:15] miraheze/ImportDump - Universal-Omega the build passed. [21:43:14] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.36 ms [21:51:26] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.31 ms [23:09:59] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 31.37 ms [23:14:02] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.37 ms