[00:00:30] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [00:02:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.24, 6.94, 6.85 [00:03:03] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.62, 2.94, 2.00 [00:04:07] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 17.65, 9.43, 4.25 [00:04:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.29, 5.99, 6.51 [00:06:27] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 8 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [00:07:03] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.62, 2.94, 2.23 [00:07:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.51, 7.01, 6.55 [00:09:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-13 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f62a8320a753 [00:09:23] [02miraheze/puppet] 07paladox 03f62a832 - Jobrunner-hi: increase parseroidcache runners to 8 [00:09:24] [02puppet] 07paladox created branch 03paladox-patch-13 - 13https://github.com/miraheze/puppet [00:09:30] [02puppet] 07paladox opened pull request 03#3521: Jobrunner-hi: increase parseroidcache runners to 8 - 13https://github.com/miraheze/puppet/pull/3521 [00:09:39] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.82, 6.53, 6.43 [00:09:46] [02puppet] 07paladox closed pull request 03#3521: Jobrunner-hi: increase parseroidcache runners to 8 - 13https://github.com/miraheze/puppet/pull/3521 [00:09:47] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-13 [00:09:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/44562468a039...d20155840f25 [00:09:50] [02miraheze/puppet] 07paladox 03d201558 - Jobrunner-hi: increase parseroidcache runners to 8 (#3521) [00:09:53] [02puppet] 07paladox deleted branch 03paladox-patch-13 - 13https://github.com/miraheze/puppet [00:18:07] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.90, 6.62, 6.86 [00:18:47] PROBLEM - matomo121 APT on matomo121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:20:51] PROBLEM - matomo121 APT on matomo121 is CRITICAL: APT CRITICAL: 73 packages available for upgrade (30 critical updates). [00:22:07] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.55, 5.52, 6.38 [00:32:30] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:40:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.66, 7.09, 6.67 [00:42:16] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.71, 6.65, 6.57 [00:55:41] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [01:04:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.93, 9.86, 8.19 [01:06:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.44, 9.52, 8.26 [01:07:38] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.67, 3.18, 2.93 [01:09:35] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.15, 3.12, 2.93 [01:27:18] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:30:33] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [02:02:30] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:03:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.90, 6.57, 6.17 [02:05:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.47, 7.25, 6.47 [02:06:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.52, 6.84, 6.40 [02:08:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.48, 6.41, 6.30 [02:08:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.24, 10.28, 9.31 [02:10:36] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:10:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.85, 10.85, 9.65 [02:14:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 7.54, 9.74, 9.53 [02:24:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.57, 9.60, 9.11 [02:26:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.65, 9.46, 9.11 [02:43:54] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.51, 6.94, 6.39 [02:45:51] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.10, 6.69, 6.36 [02:49:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.28, 7.20, 6.66 [02:51:39] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.49, 6.63, 6.51 [03:01:27] RECOVERY - db101 Backups SQL on db101 is OK: FILE_AGE OK: /var/log/sql-backup.log is 86 seconds old and 0 bytes [03:06:28] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 11.13, 9.86, 6.26 [03:08:28] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.67, 7.87, 5.97 [03:10:37] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.41, 10.15, 9.71 [03:12:33] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 10.04, 10.11, 9.75 [03:21:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.99, 10.39, 9.95 [03:21:49] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw143 [03:21:52] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 14.74, 9.39, 6.22 [03:21:59] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw141 mw143 [03:22:01] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw141 mw143 [03:22:28] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 13.25, 11.40, 7.65 [03:22:32] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 19.52, 12.47, 7.21 [03:22:38] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw143 [03:23:44] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [03:23:52] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.86, 8.53, 6.32 [03:23:55] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [03:24:01] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [03:24:28] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.45, 8.67, 7.10 [03:24:28] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 4.74, 9.15, 6.61 [03:24:38] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [03:26:13] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.28, 7.84, 6.77 [03:26:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.09, 6.69, 6.22 [03:28:10] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.70, 7.52, 6.79 [03:28:37] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.89, 5.99, 6.01 [03:30:36] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:30:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 8.27, 10.06, 10.15 [03:32:05] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.40, 6.62, 6.61 [03:40:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.66, 10.04, 9.61 [03:56:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 7.57, 9.60, 9.96 [04:00:30] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [04:02:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.61, 10.69, 10.25 [04:03:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.81, 6.93, 6.45 [04:05:39] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.00, 6.56, 6.38 [04:13:53] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:14:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.12, 10.83, 10.50 [04:18:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.39, 11.31, 10.79 [04:30:48] PROBLEM - www.christipedia.nl - LetsEncrypt on sslhost is CRITICAL: connect to address www.christipedia.nl and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:32:30] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:40:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.49, 11.25, 10.87 [04:42:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.93, 11.16, 10.89 [04:46:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.07, 6.85, 6.51 [04:48:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.83, 6.22, 6.32 [04:52:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 5.67, 8.45, 9.86 [04:59:33] RECOVERY - www.christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Tue 19 Dec 2023 02:27:03 GMT +0000. [05:01:31] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [05:06:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.91, 10.01, 9.72 [05:10:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.34, 9.87, 9.75 [05:13:29] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [05:24:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.11, 10.40, 10.00 [05:24:58] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [05:32:28] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.93, 8.83, 5.60 [05:32:30] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:34:28] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.28, 7.37, 5.46 [05:40:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.88, 10.13, 10.18 [05:48:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.98, 10.52, 10.28 [05:56:35] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:00:52] RECOVERY - swiftac111 Backups Swift Account Container on swiftac111 is OK: FILE_AGE OK: /var/log/swift-account-container-backup.log is 51 seconds old and 0 bytes [06:00:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 5.93, 8.94, 9.79 [06:10:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.92, 9.55, 9.27 [06:16:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.19, 9.89, 9.56 [06:25:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.75, 10.90, 10.16 [06:58:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.76, 11.27, 10.74 [07:04:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.92, 11.99, 11.28 [07:14:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 5.79, 8.44, 10.08 [07:20:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.63, 9.31, 9.82 [07:24:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 10.00, 9.95, 10.00 [07:28:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.49, 10.63, 10.25 [07:46:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.73, 9.88, 10.17 [08:28:21] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.71, 5.57, 3.11 [08:30:21] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.00, 4.06, 2.85 [08:53:20] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.84, 9.74, 9.06 [08:54:50] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [08:55:17] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 7.86, 9.13, 8.93 [09:15:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.28, 9.69, 8.93 [09:17:40] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.24, 9.47, 8.94 [09:26:29] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:56:11] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [10:02:00] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 2.47, 1.19 [10:05:59] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.31, 3.34, 1.88 [10:20:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.63, 9.68, 9.02 [10:22:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 10.16, 9.87, 9.17 [10:27:48] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:30:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.14, 10.32, 9.65 [10:32:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 5.89, 8.70, 9.13 [10:34:21] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.41, 4.86, 3.05 [10:36:21] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.62, 5.99, 3.71 [10:38:21] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.68, 4.93, 3.62 [10:42:29] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 14.48, 8.76, 5.80 [10:44:12] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 13.28, 12.11, 7.35 [10:44:28] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 8.14, 9.12, 6.36 [10:46:10] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.52, 9.18, 6.86 [10:54:00] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [10:57:54] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [11:00:30] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [11:27:13] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [11:32:30] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:58:50] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:01:31] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [12:17:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.97, 10.60, 9.77 [12:27:40] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 7.38, 10.12, 10.08 [12:32:30] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:40:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.35, 9.99, 9.64 [12:44:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.48, 9.93, 9.71 [12:51:47] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 9.92, 10.25, 9.97 [12:53:44] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 10.02, 10.13, 9.96 [12:59:30] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.51, 10.09, 9.96 [13:01:27] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 10.03, 10.10, 9.98 [13:10:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:13:04] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.37, 10.15, 9.95 [13:18:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.64, 10.05, 10.00 [13:40:10] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [13:42:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.75, 9.79, 8.86 [13:44:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.72, 9.72, 8.95 [13:48:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.27, 10.13, 9.30 [13:52:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.27, 9.80, 9.37 [13:56:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.10, 10.44, 9.72 [13:58:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 6.92, 8.86, 9.21 [14:00:32] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [14:15:47] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.54, 10.35, 9.72 [14:19:40] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 7.70, 9.44, 9.53 [14:27:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.98, 10.18, 9.75 [14:31:17] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 6.43, 8.74, 9.32 [14:32:30] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:36:28] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fediverse.kr and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:37:03] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.35, 9.81, 9.59 [14:39:00] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 10.11, 9.93, 9.67 [14:55:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.88, 10.37, 10.02 [14:57:04] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [15:01:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 16.34, 9.17, 6.30 [15:02:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 13.59, 7.97, 5.51 [15:04:06] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.14, 6.50, 4.60 [15:05:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 8.95, 9.97, 10.08 [15:05:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.25, 7.85, 6.49 [15:06:06] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.91, 5.88, 4.61 [15:06:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.13, 7.34, 5.91 [15:06:27] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 8 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [15:07:32] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=hardcorewiki --no-updates /home/macfan/fanhardcore_pages_full.xml (START) [15:07:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:07:39] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 3.53, 6.41, 6.13 [15:08:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.81, 6.12, 5.64 [15:13:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.64, 10.51, 10.17 [15:28:42] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:38:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 5.36, 8.26, 9.71 [15:44:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.29, 10.08, 10.00 [15:52:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.88, 10.19, 10.16 [16:04:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.36, 10.53, 10.25 [16:16:20] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 6.62, 8.69, 9.79 [16:20:43] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.71, 3.51, 2.77 [16:22:40] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.18, 3.40, 2.81 [16:24:03] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.07, 9.71, 9.64 [16:47:24] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.04, 11.17, 10.85 [16:49:20] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.80, 10.91, 10.79 [16:57:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 7.23, 9.24, 10.18 [17:04:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.62, 9.98, 10.03 [17:05:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-13 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f07e481250e6 [17:05:33] [02miraheze/puppet] 07paladox 03f07e481 - mediawiki::nutcracker: listen on socket [17:05:36] [02puppet] 07paladox created branch 03paladox-patch-13 - 13https://github.com/miraheze/puppet [17:05:39] [02puppet] 07paladox opened pull request 03#3522: mediawiki::nutcracker: listen on socket - 13https://github.com/miraheze/puppet/pull/3522 [17:07:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1d12058e0c2d [17:07:36] [02miraheze/mw-config] 07paladox 031d12058 - memcache: use sockets for nutcracker [17:07:38] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [17:07:41] [02mw-config] 07paladox opened pull request 03#5381: memcache: use sockets for nutcracker - 13https://github.com/miraheze/mw-config/pull/5381 [17:08:26] [02puppet] 07paladox closed pull request 03#3522: mediawiki::nutcracker: listen on socket - 13https://github.com/miraheze/puppet/pull/3522 [17:08:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d20155840f25...fe68e4452668 [17:08:30] [02miraheze/puppet] 07paladox 03fe68e44 - mediawiki::nutcracker: listen on socket (#3522) [17:08:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-13 [17:08:32] [02puppet] 07paladox deleted branch 03paladox-patch-13 - 13https://github.com/miraheze/puppet [17:08:33] miraheze/mw-config - paladox the build passed. [17:09:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fe68e4452668...3860d3f05f25 [17:09:35] [02miraheze/puppet] 07paladox 033860d3f - Fix [17:11:21] [02mw-config] 07paladox closed pull request 03#5381: memcache: use sockets for nutcracker - 13https://github.com/miraheze/mw-config/pull/5381 [17:11:22] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [17:11:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8a7f896f1bd0...101beca933d1 [17:11:26] [02miraheze/mw-config] 07paladox 03101beca - memcache: use sockets for nutcracker (#5381) [17:11:28] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [17:12:12] miraheze/mw-config - paladox the build passed. [17:12:21] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [17:12:24] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:12:38] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.78, 8.73, 6.12 [17:12:43] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 15.18, 10.40, 6.49 [17:12:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 13.97, 12.02, 10.83 [17:13:58] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 8.15, 5.37, 3.18 [17:14:02] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.66, 2.51, 1.28 [17:14:07] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.86, 9.42, 6.62 [17:14:17] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 124s [17:14:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:14:36] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.96, 10.01, 6.86 [17:14:41] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 8.85, 9.83, 6.76 [17:14:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 12.00, 11.80, 10.89 [17:16:01] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.53, 2.96, 1.61 [17:16:07] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.56, 7.82, 6.36 [17:16:22] !log [@test131] starting deploy of {'config': True} to all [17:16:23] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [17:16:24] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:16:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:16:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:16:34] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.57, 8.25, 6.61 [17:18:00] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.65, 3.43, 1.94 [17:22:31] !log [paladox@mwtask141] starting deploy of {'config': True} to all [17:22:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:07] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 35s [17:23:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:59] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.43, 3.76, 2.65 [17:25:59] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.40, 2.54, 2.33 [17:27:56] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 1.73, 3.31, 3.81 [17:31:56] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.20, 2.10, 3.18 [17:34:22] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.78, 2.99, 2.78 [17:34:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 7.99, 9.17, 10.16 [17:38:16] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.16, 3.25, 2.96 [17:42:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.01, 9.75, 9.83 [17:51:49] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/9882100b848f [17:51:50] [02miraheze/MirahezeMagic] 07paladox 039882100 - Use new hook MessageCacheFetchOverrides on MW 1.41+ [17:51:51] [02MirahezeMagic] 07paladox created branch 03paladox-patch-7 - 13https://github.com/miraheze/MirahezeMagic [17:51:53] [02MirahezeMagic] 07paladox opened pull request 03#443: Use new hook MessageCacheFetchOverrides on MW 1.41+ - 13https://github.com/miraheze/MirahezeMagic/pull/443 [17:52:29] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/9882100b848f...1e8277028037 [17:52:31] [02miraheze/MirahezeMagic] 07paladox 031e82770 - Update extension.json [17:52:32] [02MirahezeMagic] 07paladox synchronize pull request 03#443: Use new hook MessageCacheFetchOverrides on MW 1.41+ - 13https://github.com/miraheze/MirahezeMagic/pull/443 [17:58:13] [02MirahezeMagic] 07paladox synchronize pull request 03#443: Use new hook MessageCacheFetchOverrides on MW 1.41+ - 13https://github.com/miraheze/MirahezeMagic/pull/443 [17:58:14] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/1e8277028037...17b5b470b862 [17:58:15] [02miraheze/MirahezeMagic] 07paladox 0317b5b47 - Update MirahezeMagicHooks.php [18:01:45] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/2b00f680e957 [18:01:48] [02miraheze/MirahezeMagic] 07paladox 032b00f68 - Test against REL1_41 and php8.2 [18:01:50] [02MirahezeMagic] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/MirahezeMagic [18:01:51] [02MirahezeMagic] 07paladox opened pull request 03#444: Test against REL1_41 and php8.2 - 13https://github.com/miraheze/MirahezeMagic/pull/444 [18:04:36] miraheze/MirahezeMagic - paladox the build has errored. [18:06:52] miraheze/MirahezeMagic - paladox the build passed. [18:06:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.06, 11.42, 10.94 [18:07:50] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/2b00f680e957...5d4b31fbbcfb [18:07:52] [02miraheze/MirahezeMagic] 07paladox 035d4b31f - Update globals.php [18:07:53] [02MirahezeMagic] 07paladox synchronize pull request 03#444: Test against REL1_41 and php8.2 - 13https://github.com/miraheze/MirahezeMagic/pull/444 [18:08:07] [02MirahezeMagic] 07paladox closed pull request 03#443: Use new hook MessageCacheFetchOverrides on MW 1.41+ - 13https://github.com/miraheze/MirahezeMagic/pull/443 [18:08:09] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/034e1b3005ad...6584558a2b2b [18:08:12] [02miraheze/MirahezeMagic] 07paladox 036584558 - Use new hook MessageCacheFetchOverrides on MW 1.41+ (#443) [18:08:15] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-7 [18:08:17] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-7 - 13https://github.com/miraheze/MirahezeMagic [18:08:51] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/88fac013c281...01553f510343 [18:08:53] [02miraheze/mediawiki] 07paladox 0301553f5 - Update MirahezeMagic [18:08:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.80, 11.08, 10.87 [18:09:59] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [18:10:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:43] miraheze/MirahezeMagic - paladox the build has errored. [18:12:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 7.21, 8.98, 10.05 [18:14:32] [02MirahezeMagic] 07paladox synchronize pull request 03#444: Test against REL1_41 and php8.2 - 13https://github.com/miraheze/MirahezeMagic/pull/444 [18:14:35] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/5d4b31fbbcfb...b756cd72beff [18:14:38] [02miraheze/MirahezeMagic] 07paladox 03b756cd7 - Update globals.php [18:16:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.94, 11.01, 10.63 [18:17:25] miraheze/MirahezeMagic - paladox the build has errored. [18:18:05] miraheze/MirahezeMagic - paladox the build passed. [18:23:59] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 839s [18:24:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:57] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [18:36:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.71, 11.93, 11.82 [18:40:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.14, 11.93, 11.83 [18:42:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.53, 11.80, 11.80 [18:45:53] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/b47857618b8c [18:45:55] [02miraheze/mw-config] 07paladox 03b478576 - Set mcpMessageCachePerformanceMsgPrefixes [18:45:57] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [18:45:58] [02mw-config] 07paladox opened pull request 03#5382: Set mcpMessageCachePerformanceMsgPrefixes - 13https://github.com/miraheze/mw-config/pull/5382 [18:46:04] [02mw-config] 07paladox closed pull request 03#5382: Set mcpMessageCachePerformanceMsgPrefixes - 13https://github.com/miraheze/mw-config/pull/5382 [18:46:07] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [18:46:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/101beca933d1...134f008f08b9 [18:46:11] [02miraheze/mw-config] 07paladox 03134f008 - Set mcpMessageCachePerformanceMsgPrefixes (#5382) [18:46:13] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [18:46:18] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [18:46:20] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [18:46:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:46:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:46:34] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [18:46:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:46:46] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [18:46:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:46:51] !log [paladox@mwtask141] starting deploy of {'config': True} to all [18:46:55] miraheze/mw-config - paladox the build passed. [18:46:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:46:57] miraheze/mw-config - paladox the build passed. [18:46:58] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [18:47:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:47] !log [paladox@test131] starting deploy of {'config': True} to all [18:47:48] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [18:47:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:52:55] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=hardcorewiki --no-updates /home/macfan/fanhardcore_pages_full.xml (END - exit=0) [18:52:56] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=hardcorewiki (START) [18:53:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 13.19, 11.02, 10.92 [18:58:34] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:00:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-5382-paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/351c3e845092 [19:00:27] [02miraheze/mw-config] 07paladox 03351c3e8 - Revert "Set mcpMessageCachePerformanceMsgPrefixes (#5382)" [19:00:29] [02mw-config] 07paladox created branch 03revert-5382-paladox-patch-4 - 13https://github.com/miraheze/mw-config [19:00:31] [02mw-config] 07paladox opened pull request 03#5383: Revert "Set mcpMessageCachePerformanceMsgPrefixes" - 13https://github.com/miraheze/mw-config/pull/5383 [19:00:32] [02mw-config] 07paladox closed pull request 03#5383: Revert "Set mcpMessageCachePerformanceMsgPrefixes" - 13https://github.com/miraheze/mw-config/pull/5383 [19:00:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/134f008f08b9...c3c401695549 [19:00:37] [02miraheze/mw-config] 07paladox 03c3c4016 - Revert "Set mcpMessageCachePerformanceMsgPrefixes" (#5383) [19:00:38] [02miraheze/mw-config] 07paladox deleted branch 03revert-5382-paladox-patch-4 [19:00:41] [02mw-config] 07paladox deleted branch 03revert-5382-paladox-patch-4 - 13https://github.com/miraheze/mw-config [19:01:29] miraheze/mw-config - paladox the build passed. [19:01:31] miraheze/mw-config - paladox the build passed. [19:06:32] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:06:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:53] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s [19:06:54] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [19:06:55] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [19:06:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:02] !log [paladox@mwtask141] starting deploy of {'config': True} to all [19:07:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:12] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [19:07:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:44] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [19:08:57] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchising.org.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:11:44] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [19:22:25] RECOVERY - gtfo-wiki.cn - LetsEncrypt on sslhost is OK: OK - Certificate 'gtfo-wiki.cn' will expire on Fri 05 Jan 2024 08:31:09 GMT +0000. [19:26:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 9.84, 10.92, 11.94 [19:28:16] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [19:36:29] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fediverse.kr and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:38:54] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp24.miraheze.org - CNAME OK [19:49:52] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=hardcorewiki (END - exit=0) [19:49:53] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=hardcorewiki --active --update (END - exit=0) [19:49:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 6.44, 5.22 [19:54:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.70, 5.73, 5.10 [19:58:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 6.83, 8.65, 9.93 [19:59:53] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:05:08] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.82, 3.27, 2.66 [20:05:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 22.48, 15.42, 12.12 [20:07:05] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 1.50, 2.62, 2.50 [20:22:01] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [20:22:28] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.74, 12.80, 7.96 [20:24:01] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [20:24:28] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.34, 10.06, 7.55 [20:28:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 8.10, 9.80, 11.69 [20:36:29] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 8 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [20:56:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 4.33, 8.16, 10.10 [21:06:28] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.fediverse.kr' expires in 7 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [21:12:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.84, 10.30, 9.91 [21:14:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.48, 11.12, 10.26 [21:16:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.97, 10.98, 10.31 [21:21:06] RECOVERY - yokaiwatchwiki.com - reverse DNS on sslhost is OK: SSL OK - yokaiwatchwiki.com reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [21:24:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.22, 9.82, 10.04 [21:27:44] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [21:31:44] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [21:35:34] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.44, 6.77, 5.68 [21:38:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.50, 8.81, 8.79 [21:39:34] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.02, 7.31, 6.16 [21:41:34] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.10, 7.28, 6.29 [21:45:34] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 6.34, 6.78, 6.31 [21:52:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.75, 6.58, 5.73 [21:54:31] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.63, 6.04, 5.65 [21:56:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 8.01, 9.58, 9.67 [22:10:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/3860d3f05f25...4ce221e7513c [22:10:12]