[00:00:24] PROBLEM - template.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:22] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.57, 8.69, 5.40 [00:02:37] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.78, 11.17, 5.52 [00:03:09] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.93, 7.33, 4.04 [00:07:07] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.18, 5.25, 5.36 [00:07:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.29, 3.35, 3.14 [00:08:32] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.48, 3.79, 3.95 [00:09:04] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.20, 5.02, 5.25 [00:12:29] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.26, 1.91, 3.13 [00:12:59] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.08, 4.25, 4.91 [00:14:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.73, 2.97, 3.84 [00:18:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.27, 3.38, 3.78 [00:22:22] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.65, 3.39, 3.69 [00:26:17] PROBLEM - www.zetapedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.zetapedia.com' expires in 15 day(s) (Tue 01 Nov 2022 00:05:29 GMT +0000). [00:26:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.34, 3.60, 3.66 [00:27:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/523ec0b09b40...ace2063b00ef [00:27:27] [url] Comparing 523ec0b09b40...ace2063b00ef · miraheze/ssl · GitHub | github.com [00:27:30] [02miraheze/ssl] 07MirahezeSSLBot 03ace2063 - Bot: Update SSL cert for www.zetapedia.com [00:30:14] PROBLEM - template.miraheze.org - Sectigo on sslhost is WARNING: WARNING - Certificate '*.miraheze.org' expires in 24 day(s) (Wed 09 Nov 2022 23:59:59 GMT +0000). [00:30:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.39, 5.31, 3.99 [00:38:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.19, 3.29, 3.94 [00:40:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.03, 3.76, 3.79 [00:42:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.74, 2.32, 3.38 [00:44:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.52, 2.37, 3.20 [00:55:05] RECOVERY - www.zetapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zetapedia.com' will expire on Fri 13 Jan 2023 23:27:21 GMT +0000. [00:57:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.73, 2.75, 3.80 [00:59:43] !log [paladox@test131] starting deploy of {'world': True} to test131 [00:59:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:15] !log [paladox@test131] finished deploy of {'world': True} to test131 - SUCCESS in 211s [01:03:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.36, 4.91, 4.32 [01:04:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.92, 4.50, 3.65 [01:11:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.56, 3.52, 3.92 [01:15:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.90, 5.13, 4.46 [01:18:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.75, 3.50, 3.94 [01:19:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.75, 3.43, 3.91 [01:23:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.84, 4.07, 3.93 [01:25:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.07, 4.57, 4.17 [01:25:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.22, 3.94, 3.91 [01:31:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.89, 3.50, 3.64 [01:33:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.52, 3.31, 3.55 [01:35:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.36, 3.50, 3.57 [01:37:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.07, 3.19, 3.79 [01:39:26] PROBLEM - swiftproxy111 NTP time on swiftproxy111 is WARNING: NTP WARNING: Offset 0.2745734751 secs [01:40:37] PROBLEM - swiftobject112 NTP time on swiftobject112 is WARNING: NTP WARNING: Offset 0.2865757942 secs [01:41:14] PROBLEM - swiftobject113 NTP time on swiftobject113 is WARNING: NTP WARNING: Offset 0.2889423668 secs [01:41:25] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.03, 3.48, 3.66 [01:45:37] RECOVERY - swiftproxy111 NTP time on swiftproxy111 is OK: NTP OK: Offset 0.02958625555 secs [01:46:43] RECOVERY - swiftobject112 NTP time on swiftobject112 is OK: NTP OK: Offset 0.009092330933 secs [01:47:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.96, 2.71, 3.29 [01:47:21] RECOVERY - swiftobject113 NTP time on swiftobject113 is OK: NTP OK: Offset 0.01154142618 secs [01:49:09] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.75, 3.57, 2.57 [01:51:09] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.37, 2.44, 2.28 [01:51:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.63, 3.02, 3.39 [01:56:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 14.17, 8.12, 5.04 [02:11:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 4.83, 3.95 [02:12:31] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.65, 3.57, 2.97 [02:14:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.22, 3.69, 3.06 [02:16:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.91, 3.75, 3.16 [02:18:22] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.70, 3.90, 3.25 [02:22:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.77, 3.87, 3.38 [02:22:23] PROBLEM - swiftobject115 Current Load on swiftobject115 is CRITICAL: CRITICAL - load average: 6.39, 4.01, 2.30 [02:22:30] PROBLEM - swiftobject114 Current Load on swiftobject114 is CRITICAL: CRITICAL - load average: 6.05, 4.33, 2.49 [02:28:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.82, 2.85, 3.77 [02:32:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.35, 2.74, 3.17 [02:32:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.76, 2.31, 3.33 [02:33:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.61, 2.75, 3.82 [02:35:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.29, 3.89 [02:36:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.52, 4.33, 3.93 [02:37:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.99, 2.87, 3.67 [02:41:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.39, 2.27, 3.24 [02:46:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.71, 3.86, 3.94 [02:51:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/093cf404c769 [02:51:51] [02miraheze/puppet] 07paladox 03093cf40 - Update rewrite.py [02:51:54] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [02:51:54] [url] Page not found · GitHub · GitHub | github.com [02:51:56] [02puppet] 07paladox opened pull request 03#2935: Update rewrite.py - 13https://github.com/miraheze/puppet/pull/2935 [02:51:56] [url] Page not found · GitHub · GitHub | github.com [02:52:16] [02puppet] 07paladox synchronize pull request 03#2935: Update rewrite.py - 13https://github.com/miraheze/puppet/pull/2935 [02:52:16] [url] Page not found · GitHub · GitHub | github.com [02:52:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/093cf404c769...c92f5a56b3c5 [02:52:19] [url] Comparing 093cf404c769...c92f5a56b3c5 · miraheze/puppet · GitHub | github.com [02:52:20] [02miraheze/puppet] 07paladox 03c92f5a5 - Update rewrite.py [02:54:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.17, 3.74, 3.83 [02:56:22] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.98, 4.38, 3.00 [02:56:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.84, 3.42 [02:58:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.91, 3.16, 3.22 [02:58:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.29, 3.23, 3.62 [03:00:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.62, 3.35, 2.97 [03:00:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.74, 4.08, 3.88 [03:02:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.78, 4.33, 3.67 [03:04:47] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.24, 8.75, 5.50 [03:06:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.19, 3.34, 3.42 [03:09:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 3.70, 3.55 [03:10:10] PROBLEM - yokaiwatchwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'yokaiwatchwiki.com' expires in 15 day(s) (Tue 01 Nov 2022 02:48:17 GMT +0000). [03:11:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 3.13, 3.35 [03:14:32] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.64, 3.25, 3.92 [03:14:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.37, 2.95, 3.89 [03:15:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ace2063b00ef...72aa703db1ea [03:15:03] [url] Comparing ace2063b00ef...72aa703db1ea · miraheze/ssl · GitHub | github.com [03:15:03] [02miraheze/ssl] 07MirahezeSSLBot 0372aa703 - Bot: Update SSL cert for yokaiwatchwiki.com [03:16:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.79, 3.99, 4.14 [03:20:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.36, 2.29, 3.31 [03:24:22] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.53, 3.49, 3.54 [03:24:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.69, 3.21, 3.83 [03:30:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.14, 3.63, 3.64 [03:32:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.13, 3.29, 3.54 [03:34:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.47, 2.82, 3.32 [03:36:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.79, 3.52, 3.63 [03:38:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.89, 2.95, 3.40 [03:40:02] RECOVERY - yokaiwatchwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yokaiwatchwiki.com' will expire on Sat 14 Jan 2023 02:14:56 GMT +0000. [03:40:54] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:44:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.54, 3.67, 3.29 [03:46:21] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.72, 2.90, 3.05 [03:57:09] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 2.05, 5.11, 2.94 [03:59:06] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 0.51, 3.60, 2.64 [04:01:03] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.39, 2.53, 2.37 [04:01:04] PROBLEM - swiftobject115 Current Load on swiftobject115 is WARNING: WARNING - load average: 0.46, 1.54, 3.65 [04:02:39] PROBLEM - swiftobject114 Current Load on swiftobject114 is WARNING: WARNING - load average: 0.50, 1.33, 3.68 [04:03:01] RECOVERY - swiftobject115 Current Load on swiftobject115 is OK: OK - load average: 0.34, 1.13, 3.23 [04:04:37] RECOVERY - swiftobject114 Current Load on swiftobject114 is OK: OK - load average: 0.28, 0.98, 3.27 [04:10:28] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:35:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.53, 4.01, 3.22 [04:37:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.73, 3.80, 3.24 [04:38:46] PROBLEM - burnout.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'burnout.wiki' expires in 15 day(s) (Tue 01 Nov 2022 04:10:29 GMT +0000). [04:39:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.96, 6.25, 4.18 [04:43:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/72aa703db1ea...f31f9f585b86 [04:43:02] [url] Comparing 72aa703db1ea...f31f9f585b86 · miraheze/ssl · GitHub | github.com [04:43:03] [02miraheze/ssl] 07MirahezeSSLBot 03f31f9f5 - Bot: Update SSL cert for burnout.wiki [04:47:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.42, 3.14, 3.56 [04:49:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.35, 2.80, 3.37 [04:57:34] PROBLEM - entw.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'entw.ml' expires in 15 day(s) (Tue 01 Nov 2022 04:38:34 GMT +0000). [04:57:44] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1002005637 secs [05:04:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f31f9f585b86...4a89622b464c [05:04:42] [url] Comparing f31f9f585b86...4a89622b464c · miraheze/ssl · GitHub | github.com [05:04:44] [02miraheze/ssl] 07MirahezeSSLBot 034a89622 - Bot: Update SSL cert for entw.ml [05:07:03] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [05:07:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.20, 4.19, 3.55 [05:10:45] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=tenmeiyokoudel --r=Requested at [[phab:T9830 [05:10:46] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=tenmeiyokoudel --r=Requested at [[phab:T9830 [05:10:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:54] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=tenmeiyokoudelwiki --r=Requested at [[phab:T9830 [05:10:55] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=tenmeiyokoudelwiki --r=Requested at [[phab:T9830 [05:10:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:11:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:11:41] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=tenmeiyokouwiki --r=Requested at [[phab:T9830 [05:11:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:12:33] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=tenmeiyokouwiki --r=Requested at [[phab:T9830 [05:12:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:12:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.03, 3.44, 3.35 [05:13:10] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.0853973031 secs [05:14:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.57, 3.03, 3.20 [05:15:03] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is WARNING: HTTP WARNING: HTTP/1.1 401 Unauthorized - 473 bytes in 0.124 second response time [05:15:55] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is WARNING: HTTP WARNING: HTTP/1.1 401 Unauthorized - 473 bytes in 0.023 second response time [05:16:59] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=worldtriggerwiki --r=Requested at [[phab:T9829 [05:17:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:20:41] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=worldtriggerwiki --r=Requested at [[phab:T9829 [05:20:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:24:26] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=cdlccwikiwiki --summary=Imported from concours-de-la-chanson-classique.fandom.com --search-recursively /home/reception/cdlccwikiwiki (START) [05:24:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:26:57] PROBLEM - entw.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:27:29] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:29:09] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=cdlccwikiwiki --summary=Imported from concours-de-la-chanson-classique.fandom.com --search-recursively /home/reception/cdlccwikiwiki (END - exit=0) [05:29:09] PROBLEM - cp23 HTTPS on cp23 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 6.570 second response time [05:29:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 109.228.51.216/cpweb, 2a00:da00:1800:328::1/cpweb [05:29:23] PROBLEM - wc.miraheze.org on sslhost is WARNING: WARNING - Certificate '*.miraheze.org' expires in 24 day(s) (Wed 09 Nov 2022 23:59:59 GMT +0000). [05:30:50] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 17.91, 29.15, 14.29 [05:31:09] RECOVERY - cp23 HTTPS on cp23 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.076 second response time [05:31:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:36:15] RECOVERY - burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Sat 14 Jan 2023 03:42:54 GMT +0000. [05:46:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 4.38, 3.27 [05:51:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.91, 3.45 [05:52:04] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.56, 0.78, 3.87 [05:53:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.93, 3.50 [05:55:56] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.90, 0.90, 3.23 [05:56:09] RECOVERY - entw.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'entw.ml' will expire on Sat 14 Jan 2023 04:04:36 GMT +0000. [05:59:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.89, 3.70 [06:03:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 4.19, 3.85 [06:03:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.80, 3.70, 2.56 [06:04:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.72, 3.55, 2.67 [06:05:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.81, 3.60, 2.67 [06:06:26] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.83, 3.10, 2.62 [06:07:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.80, 2.88, 2.52 [06:10:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.92, 3.42, 2.87 [06:11:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.92, 3.94 [06:12:26] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.63, 3.13, 2.84 [06:13:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.36, 4.16, 4.01 [06:23:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.17, 2.82, 3.72 [06:27:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.29, 3.43, 2.94 [06:27:57] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.45, 5.34, 3.13 [06:27:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 2.51, 3.38 [06:28:00] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 9.07, 10.99, 5.86 [06:29:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.19, 2.60, 2.69 [06:31:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.22, 3.13, 2.71 [06:33:20] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [06:33:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:35:50] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.99, 2.91, 3.85 [06:39:50] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.87, 2.05, 3.30 [06:55:43] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2902233899 secs [07:00:50] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1217607558 secs [07:01:43] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.01367899776 secs [07:08:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.43, 3.75, 2.71 [07:18:45] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.66, 3.15, 3.35 [07:30:50] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08594939113 secs [07:38:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.40, 4.09, 3.06 [07:42:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.96, 3.98, 3.31 [07:46:09] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.23, 3.12, 3.12 [07:50:54] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1805420816 secs [07:56:50] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.68, 3.23, 2.23 [07:58:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.70, 2.93, 2.23 [08:01:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.66, 2.92 [08:03:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.38, 2.79, 2.69 [08:07:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.59, 3.04 [08:07:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.26, 3.51, 2.88 [08:09:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.83, 3.22, 2.97 [08:09:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.35, 2.77, 2.68 [08:13:43] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2730352879 secs [08:22:52] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02156224847 secs [08:35:43] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.02669069171 secs [08:42:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.96, 4.55, 3.13 [08:44:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.89, 3.26, 2.83 [09:33:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.37, 4.04, 2.33 [09:41:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.70, 3.06 [09:41:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.44, 3.63, 2.87 [09:43:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.24, 2.78, 2.65 [09:47:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.77, 3.30 [09:49:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 3.22, 3.15 [09:54:56] PROBLEM - puritwiki.p-e.kr - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406883955001831 seconds [09:56:19] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.84, 3.01, 2.12 [09:58:15] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.32, 2.78, 2.15 [10:47:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.08, 4.39, 3.06 [10:49:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.63, 2.94 [10:51:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 3.13, 2.83 [10:53:58] RECOVERY - puritwiki.p-e.kr - reverse DNS on sslhost is OK: SSL OK - puritwiki.p-e.kr reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:57:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.76, 4.27, 2.51 [11:01:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.56, 3.15, 2.47 [11:32:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 2.91, 2.52 [11:34:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.94, 2.48, 2.41 [11:45:54] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.17, 3.36, 2.34 [11:47:50] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.48, 2.67, 2.22 [12:01:02] PROBLEM - mwtask141 SSH on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.70, 3.43, 3.05 [12:02:28] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 10.22, 12.04, 5.84 [12:02:58] RECOVERY - mwtask141 SSH on mwtask141 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:03:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.58, 2.78, 2.85 [12:12:02] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.20, 2.86, 3.70 [12:15:51] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.45, 2.04, 3.16 [12:17:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.99, 4.84, 3.74 [12:23:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.77, 3.28, 3.43 [12:24:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 2.98, 3.29 [12:37:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 4.24, 3.62 [12:41:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.97, 3.71 [12:45:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.81, 2.83, 3.30 [13:09:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.32, 4.54, 3.58 [13:10:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.64, 3.87, 2.81 [13:12:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.30, 3.34, 2.75 [13:17:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.64, 3.65 [13:19:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.15, 2.77, 3.33 [13:31:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.87, 3.60 [13:35:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.76, 3.59 [13:37:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.45, 3.51 [13:39:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.73, 3.60 [13:41:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.48, 3.52 [13:43:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.53, 2.75, 3.25 [13:47:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.39, 3.42 [13:49:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.80, 3.19, 3.35 [13:59:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.78, 4.24, 3.63 [14:01:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.82, 3.55 [14:03:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.60, 4.58, 3.84 [14:07:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.84, 3.70 [14:09:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 4.06, 3.81 [14:11:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.75, 3.73 [14:13:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 4.00, 3.82 [14:15:06] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.04, 5.87, 4.66 [14:17:04] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.23, 4.96, 4.47 [14:17:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.75, 3.79 [14:19:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.75, 4.53, 4.07 [14:20:59] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.75, 5.89, 4.92 [14:22:58] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.44, 5.04, 4.73 [14:30:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.03, 3.17, 2.28 [14:30:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c91adfb351d1 [14:30:22] [02miraheze/puppet] 07paladox 03c91adfb - swift: set reclaim_age to 6 hours [14:30:24] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [14:30:24] [url] Page not found · GitHub · GitHub | github.com [14:30:27] [02puppet] 07paladox opened pull request 03#2936: swift: set reclaim_age to 6 hours - 13https://github.com/miraheze/puppet/pull/2936 [14:30:27] [url] Page not found · GitHub · GitHub | github.com [14:30:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c91adfb351d1...fba4a8ffb377 [14:30:57] [url] Comparing c91adfb351d1...fba4a8ffb377 · miraheze/puppet · GitHub | github.com [14:30:57] [02miraheze/puppet] 07paladox 03fba4a8f - Update object-server.conf.erb [14:30:59] [02puppet] 07paladox synchronize pull request 03#2936: swift: set reclaim_age to 6 hours - 13https://github.com/miraheze/puppet/pull/2936 [14:31:01] [url] Page not found · GitHub · GitHub | github.com [14:31:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fba4a8ffb377...b8c50a4cd1c3 [14:31:09] [url] Comparing fba4a8ffb377...b8c50a4cd1c3 · miraheze/puppet · GitHub | github.com [14:31:10] [02miraheze/puppet] 07paladox 03b8c50a4 - Update container-server.conf.erb [14:31:13] [02puppet] 07paladox synchronize pull request 03#2936: swift: set reclaim_age to 6 hours - 13https://github.com/miraheze/puppet/pull/2936 [14:31:13] [url] Page not found · GitHub · GitHub | github.com [14:32:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.75, 2.71, 2.23 [14:35:24] [02puppet] 07paladox synchronize pull request 03#2936: swift: set reclaim_age to 6 hours - 13https://github.com/miraheze/puppet/pull/2936 [14:35:24] [url] Page not found · GitHub · GitHub | github.com [14:35:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b8c50a4cd1c3...4f6dc1c4f3d2 [14:35:26] [url] Comparing b8c50a4cd1c3...4f6dc1c4f3d2 · miraheze/puppet · GitHub | github.com [14:35:27] [02miraheze/puppet] 07paladox 034f6dc1c - Update container-server.conf.erb [14:35:40] [02puppet] 07paladox synchronize pull request 03#2936: swift: set reclaim_age to 6 hours - 13https://github.com/miraheze/puppet/pull/2936 [14:35:41] ... [14:35:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4f6dc1c4f3d2...992154e32932 [14:35:43] [url] Comparing 4f6dc1c4f3d2...992154e32932 · miraheze/puppet · GitHub | github.com [14:35:44] [02miraheze/puppet] 07paladox 03992154e - Update object-server.conf.erb [14:35:57] [02puppet] 07paladox synchronize pull request 03#2936: swift: set reclaim_age to 6 hours - 13https://github.com/miraheze/puppet/pull/2936 [14:35:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.53, 3.30, 3.92 [14:35:58] [url] Page not found · GitHub · GitHub | github.com [14:35:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/992154e32932...8a70e7480ecd [14:36:00] [url] Comparing 992154e32932...8a70e7480ecd · miraheze/puppet · GitHub | github.com [14:36:01] [02miraheze/puppet] 07paladox 038a70e74 - Update account-server.conf.erb [14:39:14] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1168162823 secs [14:39:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.43, 3.80 [14:41:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.33, 3.06, 3.62 [14:42:46] PROBLEM - es141 Disk Space on es141 is WARNING: DISK WARNING - free space: / 20919 MB (10% inode=99%); [14:43:14] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08727318048 secs [14:44:46] RECOVERY - es141 Disk Space on es141 is OK: DISK OK - free space: / 25656 MB (12% inode=99%); [14:45:44] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.92, 3.16, 2.25 [14:45:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 4.27, 3.97 [14:47:44] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.20, 2.35, 2.06 [14:49:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.95, 3.36, 3.69 [14:50:44] [02puppet] 07paladox synchronize pull request 03#2936: swift: set reclaim_age to 6 hours - 13https://github.com/miraheze/puppet/pull/2936 [14:50:44] [url] Page not found · GitHub · GitHub | github.com [14:50:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8a70e7480ecd...37a68cf57f9d [14:50:47] [url] Comparing 8a70e7480ecd...37a68cf57f9d · miraheze/puppet · GitHub | github.com [14:50:49] [02miraheze/puppet] 07paladox 0337a68cf - Update object-server.conf.erb [14:51:56] PROBLEM - es131 Disk Space on es131 is WARNING: DISK WARNING - free space: / 21039 MB (10% inode=99%); [14:57:07] !log delete graylog indices 0 and 1 (1 contained over 300g and said seems to have inconsitencies?) [14:57:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:57:56] RECOVERY - es131 Disk Space on es131 is OK: DISK OK - free space: / 185529 MB (92% inode=99%); [14:57:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.87, 3.84, 3.65 [14:59:34] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.28, 5.42, 4.30 [14:59:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.62, 3.58 [15:00:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6182aec3e9c2...6b5f4902e499 [15:00:25] [url] Comparing 6182aec3e9c2...6b5f4902e499 · miraheze/puppet · GitHub | github.com [15:00:27] [02miraheze/puppet] 07paladox 036b5f490 - graylog: upgrade to 4.3.8 [15:00:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6b5f4902e499...736d69523217 [15:00:58] [url] Comparing 6b5f4902e499...736d69523217 · miraheze/puppet · GitHub | github.com [15:01:00] [02miraheze/puppet] 07paladox 03736d695 - graylog: upgrade mongodb to 4.4.17 [15:01:06] [02puppet] 07paladox closed pull request 03#2936: swift: set reclaim_age to 6 hours - 13https://github.com/miraheze/puppet/pull/2936 [15:01:06] [url] Page not found · GitHub · GitHub | github.com [15:01:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/736d69523217...e547db95b59c [15:01:09] [url] Comparing 736d69523217...e547db95b59c · miraheze/puppet · GitHub | github.com [15:01:10] [02miraheze/puppet] 07paladox 03e547db9 - swift: set reclaim_age to 6 hours (#2936) [15:01:11] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [15:01:14] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [15:01:14] [url] Page not found · GitHub · GitHub | github.com [15:01:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.78, 4.81, 4.02 [15:02:39] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [15:02:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:03:31] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.48, 5.93, 4.82 [15:05:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.83, 3.79 [15:07:28] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.39, 4.80, 4.63 [15:07:50] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.48, 3.18, 2.27 [15:07:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.66, 3.71 [15:08:51] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [15:08:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:09:13] !log bash ./delbackups.sh /home/reception/delwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php (running delbackups) [15:09:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:09:49] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.88, 3.08, 2.35 [15:15:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.61, 3.70 [15:17:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.82, 4.03, 3.85 [15:23:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.41, 3.69 [15:27:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.73, 2.62, 3.31 [15:30:31] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.46, 3.82, 2.74 [15:32:27] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.02, 2.74, 2.48 [15:33:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0b78e7748e73 [15:34:00] [02miraheze/puppet] 07paladox 030b78e77 - varnish: redirect (static|static-new).miraheze.org to commons.miraheze.org if it is just on "/" [15:34:01] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [15:34:02] [url] Page not found · GitHub · GitHub | github.com [15:34:04] [02puppet] 07paladox opened pull request 03#2937: varnish: redirect (static|static-new).miraheze.org to commons.miraheze.org if it is just on "/" - 13https://github.com/miraheze/puppet/pull/2937 [15:34:05] [url] Page not found · GitHub · GitHub | github.com [15:36:12] [02puppet] 07paladox closed pull request 03#2937: varnish: redirect (static|static-new).miraheze.org to commons.miraheze.org if it is just on "/" - 13https://github.com/miraheze/puppet/pull/2937 [15:36:13] [url] Page not found · GitHub · GitHub | github.com [15:36:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e547db95b59c...3c2c4f08de82 [15:36:16] [url] Comparing e547db95b59c...3c2c4f08de82 · miraheze/puppet · GitHub | github.com [15:36:17] [02miraheze/puppet] 07paladox 033c2c4f0 - varnish: redirect (static|static-new).miraheze.org to commons.miraheze.org if it is just on "/" (#2937) [15:36:20] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [15:36:23] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [15:36:23] ... [15:42:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.92, 3.85, 2.89 [15:44:26] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.28, 2.86, 2.64 [15:47:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.62, 3.55, 3.32 [15:49:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.66, 2.98, 3.14 [15:52:26] [02mw-config] 07paladox edited pull request 03#4932: Support container per directory per wiki - 13https://github.com/miraheze/mw-config/pull/4932 [15:52:27] [url] Page not found · GitHub · GitHub | github.com [15:53:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 2.65, 1.95 [15:53:37] [02mw-config] 07paladox synchronize pull request 03#4932: Support container per directory per wiki - 13https://github.com/miraheze/mw-config/pull/4932 [15:53:38] ... [15:53:39] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/233d547d159f...0f4f6ce289fb [15:53:39] [url] Comparing 233d547d159f...0f4f6ce289fb · miraheze/mw-config · GitHub | github.com [15:53:42] [02miraheze/mw-config] 07paladox 030f4f6ce - Update FileBackend.php [15:53:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 4.16, 3.62 [15:53:58] [02mw-config] 07paladox synchronize pull request 03#4932: Support container per directory per wiki - 13https://github.com/miraheze/mw-config/pull/4932 [15:53:58] [url] Page not found · GitHub · GitHub | github.com [15:54:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0f4f6ce289fb...779280ad7871 [15:54:01] [url] Comparing 0f4f6ce289fb...779280ad7871 · miraheze/mw-config · GitHub | github.com [15:54:02] [02miraheze/mw-config] 07paladox 03779280a - Update FileBackend.php [15:54:20] [02mw-config] 07paladox closed pull request 03#4932: Support container per directory per wiki - 13https://github.com/miraheze/mw-config/pull/4932 [15:54:20] [url] Page not found · GitHub · GitHub | github.com [15:54:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/1d6cab076a3b...04aba7d42b04 [15:54:23] [url] Comparing 1d6cab076a3b...04aba7d42b04 · miraheze/mw-config · GitHub | github.com [15:54:24] [02miraheze/mw-config] 07paladox 0304aba7d - Support container per directory per wiki (#4932) [15:54:27] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [15:54:30] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/mw-config [15:54:30] [url] Page not found · GitHub · GitHub | github.com [15:54:50] miraheze/mw-config - paladox the build passed. [15:55:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.35, 2.16, 1.86 [15:55:05] miraheze/mw-config - paladox the build passed. [15:55:15] miraheze/mw-config - paladox the build passed. [15:57:37] !log [@test131] starting deploy of {'config': True} to all [15:57:41] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 3s [15:57:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:54] PROBLEM - test131 Check Gluster Clients on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [16:01:54] RECOVERY - test131 Check Gluster Clients on test131 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [16:03:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.15, 3.44, 3.63 [16:05:01] [02mw-config] 07paladox closed pull request 03#4929: filebackend: set containerPaths - 13https://github.com/miraheze/mw-config/pull/4929 [16:05:02] [url] Page not found · GitHub · GitHub | github.com [16:05:03] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [16:05:05] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [16:05:05] [url] Page not found · GitHub · GitHub | github.com [16:05:45] [02puppet] 07paladox synchronize pull request 03#2935: Update rewrite.py - 13https://github.com/miraheze/puppet/pull/2935 [16:05:46] ... [16:05:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c92f5a56b3c5...0003127f208f [16:05:49] [url] Comparing c92f5a56b3c5...0003127f208f · miraheze/puppet · GitHub | github.com [16:05:49] [02miraheze/puppet] 07paladox 030003127 - Update rewrite.py [16:05:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.18, 3.69, 3.67 [16:07:19] PROBLEM - studyabroad.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'studyabroad.wiki' expires in 15 day(s) (Tue 01 Nov 2022 16:02:45 GMT +0000). [16:07:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.33, 3.24, 3.51 [16:10:33] !log [@mwtask141] starting deploy of {'config': True} to all [16:10:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:41] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [16:10:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:12:14] [02puppet] 07paladox edited pull request 03#2935: swift: update rewrites to correspond to layout - 13https://github.com/miraheze/puppet/pull/2935 [16:12:15] [url] Page not found · GitHub · GitHub | github.com [16:12:17] [02puppet] 07paladox closed pull request 03#2935: swift: update rewrites to correspond to layout - 13https://github.com/miraheze/puppet/pull/2935 [16:12:18] ... [16:12:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3c2c4f08de82...348757c50b81 [16:12:20] [url] Comparing 3c2c4f08de82...348757c50b81 · miraheze/puppet · GitHub | github.com [16:12:22] [02miraheze/puppet] 07paladox 03348757c - swift: update rewrites to correspond to layout (#2935) [16:12:25] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [16:12:27] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [16:12:28] [url] Page not found · GitHub · GitHub | github.com [16:13:01] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to test131 [16:13:02] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to test131 - SUCCESS in 0s [16:13:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:11] !log [paladox@test131] starting deploy of {'config': True} to test131 [16:13:12] !log [paladox@test131] finished deploy of {'config': True} to test131 - SUCCESS in 0s [16:13:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.70, 2.75, 3.24 [16:15:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4a89622b464c...e2f634a8d530 [16:15:07] [url] Comparing 4a89622b464c...e2f634a8d530 · miraheze/ssl · GitHub | github.com [16:15:09] [02miraheze/ssl] 07MirahezeSSLBot 03e2f634a - Bot: Update SSL cert for studyabroad.wiki [16:17:02] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Error - 273 bytes in 0.027 second response time [16:17:55] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Error - 273 bytes in 0.019 second response time [16:25:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/348757c50b81...371b82acd7f7 [16:25:03] [url] Comparing 348757c50b81...371b82acd7f7 · miraheze/puppet · GitHub | github.com [16:25:05] [02miraheze/puppet] 07paladox 03371b82a - Fix [16:27:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/371b82acd7f7...8c10b0f97152 [16:27:03] [url] Comparing 371b82acd7f7...8c10b0f97152 · miraheze/puppet · GitHub | github.com [16:27:04] [02miraheze/puppet] 07paladox 038c10b0f - Fix [16:27:55] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 367 bytes in 0.036 second response time [16:29:02] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 367 bytes in 0.048 second response time [16:43:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.62, 4.60, 3.50 [16:53:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.86, 3.72 [16:57:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.85, 3.74 [17:05:20] RECOVERY - studyabroad.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'studyabroad.wiki' will expire on Sat 14 Jan 2023 15:15:01 GMT +0000. [17:05:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-4932-paladox-patch-2 [+0/-0/±2] 13https://github.com/miraheze/mw-config/commit/d490156aaf47 [17:05:57] [02miraheze/mw-config] 07paladox 03d490156 - Revert "Support container per directory per wiki (#4932)" [17:06:00] [02mw-config] 07paladox created branch 03revert-4932-paladox-patch-2 - 13https://github.com/miraheze/mw-config [17:06:00] [url] Page not found · GitHub · GitHub | github.com [17:06:02] [02mw-config] 07paladox opened pull request 03#4933: Revert "Support container per directory per wiki" - 13https://github.com/miraheze/mw-config/pull/4933 [17:06:02] [url] Page not found · GitHub · GitHub | github.com [17:06:05] [02mw-config] 07paladox closed pull request 03#4933: Revert "Support container per directory per wiki" - 13https://github.com/miraheze/mw-config/pull/4933 [17:06:05] [url] Page not found · GitHub · GitHub | github.com [17:06:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/04aba7d42b04...801c17d9b54f [17:06:08] [url] Comparing 04aba7d42b04...801c17d9b54f · miraheze/mw-config · GitHub | github.com [17:06:11] [02miraheze/mw-config] 07paladox 03801c17d - Revert "Support container per directory per wiki" (#4933) [17:06:14] [02miraheze/mw-config] 07paladox deleted branch 03revert-4932-paladox-patch-2 [17:06:16] [02mw-config] 07paladox deleted branch 03revert-4932-paladox-patch-2 - 13https://github.com/miraheze/mw-config [17:06:16] [url] Page not found · GitHub · GitHub | github.com [17:06:29] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [17:06:38] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [17:06:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:45] !log [paladox@mwtask141] starting deploy of {'config': True} to all [17:06:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:54] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [17:06:57] miraheze/mw-config - paladox the build passed. [17:07:07] miraheze/mw-config - paladox the build passed. [17:07:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-4933-revert-4932-paladox-patch-2 [+0/-0/±2] 13https://github.com/miraheze/mw-config/commit/4ecffd89d1f2 [17:07:15] [02miraheze/mw-config] 07paladox 034ecffd8 - Revert "Revert "Support container per directory per wiki" (#4933)" [17:07:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:18] [02mw-config] 07paladox created branch 03revert-4933-revert-4932-paladox-patch-2 - 13https://github.com/miraheze/mw-config [17:07:18] [url] Page not found · GitHub · GitHub | github.com [17:07:20] [02mw-config] 07paladox opened pull request 03#4934: Support container per directory per wiki - 13https://github.com/miraheze/mw-config/pull/4934 [17:07:21] ... [17:08:24] miraheze/mw-config - paladox the build passed. [17:13:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.45, 3.00 [17:15:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 3.11, 2.93 [17:15:42] [02mw-config] 07paladox synchronize pull request 03#4934: Support container per directory per wiki - 13https://github.com/miraheze/mw-config/pull/4934 [17:15:43] [url] Page not found · GitHub · GitHub | github.com [17:15:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-4933-revert-4932-paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4ecffd89d1f2...6d1ee84dda73 [17:15:44] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.26, 3.56, 2.32 [17:15:44] [url] Comparing 4ecffd89d1f2...6d1ee84dda73 · miraheze/mw-config · GitHub | github.com [17:15:45] [02miraheze/mw-config] 07paladox 036d1ee84 - Update FileBackend.php [17:16:38] miraheze/mw-config - paladox the build passed. [17:17:44] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.31, 3.11, 2.31 [17:25:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.02, 4.63, 3.60 [17:26:33] !log [@test131] starting deploy of {'config': True} to all [17:26:34] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [17:26:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:27:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:45] [02mw-config] 07paladox closed pull request 03#4934: Support container per directory per wiki - 13https://github.com/miraheze/mw-config/pull/4934 [17:32:46] [url] Page not found · GitHub · GitHub | github.com [17:32:47] [02miraheze/mw-config] 07paladox deleted branch 03revert-4933-revert-4932-paladox-patch-2 [17:32:49] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/801c17d9b54f...2346d4b4a8b2 [17:32:50] [url] Comparing 801c17d9b54f...2346d4b4a8b2 · miraheze/mw-config · GitHub | github.com [17:32:50] [02miraheze/mw-config] 07paladox 032346d4b - Support container per directory per wiki (#4934) [17:32:52] [02mw-config] 07paladox deleted branch 03revert-4933-revert-4932-paladox-patch-2 - 13https://github.com/miraheze/mw-config [17:32:53] [url] Page not found · GitHub · GitHub | github.com [17:33:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.76, 3.70, 3.79 [17:33:05] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [17:33:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:14] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [17:33:17] !log [paladox@mwtask141] starting deploy of {'config': True} to all [17:33:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:27] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [17:33:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:49] miraheze/mw-config - paladox the build passed. [17:34:21] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to test131 [17:34:22] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to test131 - SUCCESS in 1s [17:34:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:28] !log [paladox@test131] starting deploy of {'config': True} to test131 [17:34:29] !log [paladox@test131] finished deploy of {'config': True} to test131 - SUCCESS in 0s [17:34:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.79, 2.69, 3.35 [17:39:09] !log [paladox@test131] starting deploy of {'pull': 'world', 'world': True} to test131 [17:39:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:41:12] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:41:14] !log shutdown all VMs on cloud11 [17:41:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:42:46] !log [paladox@test131] finished deploy of {'pull': 'world', 'world': True} to test131 - SUCCESS in 216s [17:42:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:15] PROBLEM - swiftproxy111 SSH on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:21] PROBLEM - swiftobject111 Swift Object Service on swiftobject111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:25] PROBLEM - swiftobject112 SSH on swiftobject112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:27] !log shutdown cloud11 to try recovery for disk [17:43:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:36] PROBLEM - ping6 on swiftac111 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:43:38] PROBLEM - swiftobject112 Swift Object Service on swiftobject112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:38] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:43:57] PROBLEM - swiftproxy111 memcached on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:22] PROBLEM - swiftobject114 SSH on swiftobject114 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:35] PROBLEM - ping6 on swiftproxy111 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:44:36] PROBLEM - ping6 on swiftobject113 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:44:36] PROBLEM - swiftproxy111 NTP time on swiftproxy111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:44:38] PROBLEM - swiftac111 Swift Container Service on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:45] PROBLEM - ping6 on swiftobject111 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:44:48] PROBLEM - swiftac111 Swift Account Service on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:06] PROBLEM - swiftobject111 SSH on swiftobject111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:13] PROBLEM - swiftobject113 SSH on swiftobject113 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:13] PROBLEM - swiftobject115 Current Load on swiftobject115 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:45:15] PROBLEM - swiftac111 SSH on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:21] PROBLEM - swiftobject115 Swift Object Service on swiftobject115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:24] PROBLEM - ping6 on swiftobject112 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:45:26] PROBLEM - Host swiftobject112 is DOWN: PING CRITICAL - Packet loss = 100% [17:45:30] PROBLEM - ping6 on swiftobject115 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:45:31] PROBLEM - Host swiftobject114 is DOWN: PING CRITICAL - Packet loss = 100% [17:45:36] PROBLEM - Host swiftobject115 is DOWN: PING CRITICAL - Packet loss = 100% [17:45:37] PROBLEM - swiftobject113 Swift Object Service on swiftobject113 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:42] PROBLEM - Host swiftobject111 is DOWN: PING CRITICAL - Packet loss = 100% [17:46:02] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:05] PROBLEM - swiftproxy111 Current Load on swiftproxy111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:46:06] PROBLEM - Host swiftproxy111 is DOWN: PING CRITICAL - Packet loss = 100% [17:46:08] PROBLEM - Host swiftobject113 is DOWN: PING CRITICAL - Packet loss = 100% [17:46:19] PROBLEM - swiftac111 conntrack_table_size on swiftac111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:46:31] PROBLEM - Host swiftac111 is DOWN: PING CRITICAL - Packet loss = 100% [17:46:40] PROBLEM - Host cloud11 is DOWN: PING CRITICAL - Packet loss = 100% [17:55:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.32, 3.26 [17:57:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 2.90, 3.11 [18:14:34] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.82, 3.78, 2.30 [18:16:30] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.55, 2.91, 2.16 [18:20:27] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 1.00 ms [18:20:58] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,2] - Reallocated_Sector_Ct is non-zero (3) --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean| [18:21:16] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:25:36] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:54] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:26:28] PROBLEM - ping6 on cloud11 is CRITICAL: PING CRITICAL - Packet loss = 100% [18:27:51] PROBLEM - Host cloud11 is DOWN: PING CRITICAL - Packet loss = 100% [18:39:23] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.62, 3.42, 2.75 [18:41:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.85, 3.36 [18:41:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.83, 2.76, 2.59 [18:45:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.36, 3.23, 3.23 [18:47:47] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.50, 5.94, 4.93 [18:51:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.81, 3.45 [18:53:47] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.79, 6.12, 5.23 [18:55:47] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.88, 5.19, 5.00 [18:57:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.98, 3.71 [18:57:47] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 4.14, 4.73, 4.84 [19:03:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.51, 3.10, 3.39 [19:07:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.56, 3.53 [19:09:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 3.09, 3.35 [19:11:47] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.00, 5.99, 5.23 [19:15:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.54, 3.51 [19:15:47] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.04, 5.26, 5.17 [19:17:47] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.54, 4.53, 4.91 [19:19:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.49, 3.47 [19:21:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 3.14, 3.34 [19:41:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.44, 3.14 [19:42:48] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms [19:43:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.17, 3.08 [19:43:07] PROBLEM - cloud11 Puppet on cloud11 is WARNING: WARNING: Puppet last ran 1 hour ago [19:43:52] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,2] - Reallocated_Sector_Ct is non-zero (3) --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean| [19:43:59] RECOVERY - ping6 on cloud11 is OK: PING OK - Packet loss = 0%, RTA = 0.59 ms [19:44:06] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:45:04] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:02:23] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.35, 3.42, 2.87 [20:04:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.00, 2.78, 2.69 [20:07:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.62, 3.26, 3.95 [20:08:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.43, 3.29 [20:10:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.59, 3.09, 3.18 [20:13:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.79, 3.94 [20:15:35] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.79, 3.38, 2.52 [20:17:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.82, 3.38, 2.64 [20:28:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.60, 3.21 [20:30:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.29, 3.15 [20:36:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 3.94, 3.43 [20:38:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.80, 3.84, 3.13 [20:39:14] !log DELETED AND DROPPED https://phabricator.miraheze.org/P471 (PART I) [20:39:15] [url] ✎ P471 Deleted wikis 16/10/2022 (PART I) | phabricator.miraheze.org [20:39:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.23, 2.87, 2.86 [20:42:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.62, 3.50 [20:42:29] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.84, 3.04, 3.30 [20:44:29] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29406 bytes in 1.448 second response time [20:47:30] !log shutdown cloud11 again, am going to rebuild the array configuration (and also have to re-install OS once again) [20:47:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.84, 4.66, 3.88 [20:51:07] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:52:15] PROBLEM - ping6 on cloud11 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:53:36] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:54:13] PROBLEM - Host cloud11 is DOWN: PING CRITICAL - Packet loss = 100% [20:54:26] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.11, 2.93, 2.49 [20:56:26] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.55, 2.46, 2.38 [20:59:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.78, 3.43, 3.86 [21:03:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.65, 2.32, 3.29 [21:09:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.02, 3.91 [21:15:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.59, 2.20, 3.25 [21:23:26] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/ef4a9acf6cb1...c5b5c03a7fe6 [21:23:27] [url] Comparing ef4a9acf6cb1...c5b5c03a7fe6 · miraheze/mediawiki · GitHub | github.com [21:23:29] [02miraheze/mediawiki] 07Universal-Omega 03c5b5c03 - Update Mirage [21:23:58] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a6daecc48c03...49b312959db3 [21:23:59] [url] Comparing a6daecc48c03...49b312959db3 · miraheze/mediawiki · GitHub | github.com [21:24:00] [02miraheze/mediawiki] 07Universal-Omega 0349b3129 - Update Mirage [21:32:55] !log [universalomega@mwtask141] starting deploy of {'world': True} to all [21:33:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:11] !log [universalomega@test131] starting deploy of {'world': True} to all [21:33:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:47] !log [universalomega@test131] finished deploy of {'world': True} to all - SUCCESS in 216s [21:36:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:37:58] !log [universalomega@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 303s [21:38:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.48, 2.96, 2.43 [21:40:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.72, 2.92, 2.49 [22:01:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.34, 2.80 [22:03:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.06, 2.49, 2.55 [22:07:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.58, 4.11, 3.21 [22:09:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.82, 3.80, 3.19 [22:11:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.67, 3.36, 3.10 [22:37:14] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1617808342 secs [22:41:14] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08940061927 secs [23:05:00] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.14, 4.93, 3.23 [23:13:14] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset -0.1301796436 secs [23:14:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.76, 2.93, 3.43 [23:15:14] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.0931147635 secs [23:16:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.56, 2.45, 3.19 [23:42:21] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.06, 2.86, 1.94 [23:44:17] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.88, 3.15, 2.16 [23:45:44] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.44, 3.03, 1.89 [23:46:13] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.05, 2.88, 2.19 [23:46:27] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.71, 2.84, 2.13 [23:47:44] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.78, 2.09, 1.68 [23:48:26] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.77, 2.71, 2.19 [23:59:50] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.35, 3.50, 2.53