[00:04:06] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 15.81, 9.44, 4.32 [00:05:43] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:07:56] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 5 backends are down. mw131 mw132 mw133 mw134 mediawiki [00:08:25] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.57, 3.22, 2.03 [00:09:04] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 5 backends are down. mw131 mw132 mw133 mw134 mediawiki [00:09:10] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 5 backends are down. mw131 mw132 mw133 mw134 mediawiki [00:09:10] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 5 backends are down. mw131 mw132 mw133 mw134 mediawiki [00:09:26] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.013 second response time [00:09:42] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:09:44] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.024 second response time [00:10:25] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.64, 3.07, 2.12 [00:11:04] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:11:05] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:11:07] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:11:23] RECOVERY - mw133 MediaWiki Rendering on mw133 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.235 second response time [00:11:38] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.481 second response time [00:11:47] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.526 second response time [00:11:56] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:12:06] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.85, 7.73, 5.95 [00:13:06] PROBLEM - mw143 APT on mw143 is CRITICAL: APT CRITICAL: 391 packages available for upgrade (19 critical updates). [00:13:59] PROBLEM - mw142 APT on mw142 is CRITICAL: APT CRITICAL: 397 packages available for upgrade (19 critical updates). [00:14:06] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 5.28, 6.76, 5.78 [00:15:02] PROBLEM - mw141 APT on mw141 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:15:04] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [00:15:56] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw131 mw132 mw134 [00:16:37] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.64, 3.17, 2.31 [00:16:50] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw133 [00:16:58] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw131 mw133 [00:17:04] PROBLEM - mw141 APT on mw141 is CRITICAL: APT CRITICAL: 650 packages available for upgrade (22 critical updates). [00:17:10] PROBLEM - mw143 APT on mw143 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:17:55] PROBLEM - mw142 APT on mw142 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:18:35] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.29, 3.15, 2.40 [00:21:04] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:21:57] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:22:48] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:22:49] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:24:01] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.19, 3.77, 3.04 [00:25:45] PROBLEM - mw143 APT on mw143 is CRITICAL: APT CRITICAL: 228 packages available for upgrade (13 critical updates). [00:25:56] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw132 [00:26:01] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.93, 3.59, 3.05 [00:26:46] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.05, 3.62, 2.86 [00:26:49] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw133 [00:27:04] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [00:27:55] PROBLEM - mw143 APT on mw143 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:28:42] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 10 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [00:28:44] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.52, 3.57, 2.94 [00:28:49] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:29:04] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:29:58] PROBLEM - mw143 APT on mw143 is CRITICAL: APT CRITICAL: 215 packages available for upgrade (13 critical updates). [00:30:41] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.61, 3.25, 2.90 [00:30:56] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.034 second response time [00:31:58] PROBLEM - mw143 APT on mw143 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:31:58] PROBLEM - mw142 APT on mw142 is CRITICAL: APT CRITICAL: 225 packages available for upgrade (12 critical updates). [00:32:01] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.05, 3.02, 3.02 [00:32:35] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw143 [00:32:49] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw143 [00:33:04] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw143 [00:33:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.53, 6.83, 6.22 [00:33:55] PROBLEM - mw142 APT on mw142 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:35:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.51, 6.42, 6.14 [00:35:08] RECOVERY - mw141 APT on mw141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:35:59] PROBLEM - mw143 APT on mw143 is CRITICAL: APT CRITICAL: 133 packages available for upgrade (8 critical updates). [00:36:02] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.92, 3.40 [00:37:57] PROBLEM - mw142 APT on mw142 is CRITICAL: APT CRITICAL: 188 packages available for upgrade (12 critical updates). [00:37:58] PROBLEM - mw143 APT on mw143 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:38:38] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.023 second response time [00:40:00] PROBLEM - mw143 APT on mw143 is CRITICAL: APT CRITICAL: 91 packages available for upgrade (4 critical updates). [00:45:56] PROBLEM - mw142 APT on mw142 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:48:05] RECOVERY - mw143 APT on mw143 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:48:11] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.333 second response time [00:49:56] PROBLEM - mw142 APT on mw142 is CRITICAL: APT CRITICAL: 11 packages available for upgrade (1 critical updates). [00:51:56] RECOVERY - mw142 APT on mw142 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:52:08] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1046 bytes in 0.212 second response time [00:53:36] RECOVERY - mw141 php-fpm on mw141 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [00:54:08] PROBLEM - mw141 MediaWiki Rendering on mw141 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [00:56:39] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:00:54] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 7.712 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:04:10] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [01:04:55] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 10 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [01:14:04] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [01:17:34] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:18:05] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [01:19:38] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 6.106 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:24:18] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:26:16] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.694 second response time [01:28:15] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:28:26] RECOVERY - mw143 php-fpm on mw143 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [01:29:56] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [01:31:13] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.57, 2.90, 2.39 [01:33:03] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.589 second response time [01:35:13] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.78, 3.02, 2.59 [01:35:18] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:37:01] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [01:48:01] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.75, 3.21, 3.87 [01:50:03] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.51, 3.74, 3.97 [01:54:02] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.70, 3.08, 3.70 [01:59:23] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=modernphenotypeswiki --no-updates --username-prefix=wikia:modernphenotypes /home/macfan/ImportDump/modernphenotypeswiki-20231208161021.xml (START) [02:00:03] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.04, 2.59, 3.28 [02:00:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:04:53] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:11:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:16:10] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=modernphenotypeswiki --no-updates --username-prefix=wikia:modernphenotypes /home/macfan/ImportDump/modernphenotypeswiki-20231208161021.xml (END - exit=0) [02:16:11] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=modernphenotypeswiki (START) [02:16:17] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=modernphenotypeswiki (END - exit=256) [02:16:18] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=modernphenotypeswiki --active --update (END - exit=0) [02:17:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:58:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:58:34] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=cryptworldswiki --no-updates --username-prefix=wikia:cryptworlds /home/macfan/ImportDump/cryptworldswiki-20231210001226.xml (START) [02:59:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:54] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=cryptworldswiki --no-updates --username-prefix=wikia:cryptworlds /home/macfan/ImportDump/cryptworldswiki-20231210001226.xml (END - exit=0) [03:00:55] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=cryptworldswiki (START) [03:01:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:01:47] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=cryptworldswiki (END - exit=0) [03:01:48] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=cryptworldswiki --active --update (END - exit=0) [03:02:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:02:55] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [03:04:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:14] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [03:29:05] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:37:34] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wizardia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:53:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 6.37, 5.74 [03:55:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.66, 6.02, 5.68 [03:58:56] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.023 second response time [04:01:51] PROBLEM - mw142 JobRunner Service on mw142 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [04:02:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e751561887c2...5b2e57556004 [04:02:46] RECOVERY - mw142 php-fpm on mw142 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [04:02:46] [02miraheze/puppet] 07paladox 035b2e575 - mw142: disable jobrunner [04:03:15] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:04:45] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.03, 7.26, 2.98 [04:05:02] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.377 second response time [04:05:32] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.012 second response time [04:05:53] PROBLEM - mw141 php-fpm on mw141 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [04:06:13] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5004974306 secs [04:06:41] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 9.91, 8.22, 3.85 [04:06:47] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.827 second response time [04:07:39] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:07:48] RECOVERY - mw141 php-fpm on mw141 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [04:07:51] RECOVERY - mw142 JobRunner Service on mw142 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [04:09:27] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.928 second response time [04:09:57] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [04:10:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [04:10:49] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [04:11:04] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [04:12:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/81afa12fd2b1 [04:12:42] [02miraheze/puppet] 07paladox 0381afa12 - mw* modify keepalive [04:12:43] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [04:12:46] [02puppet] 07paladox opened pull request 03#3559: mw* modify keepalive - 13https://github.com/miraheze/puppet/pull/3559 [04:13:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/81afa12fd2b1...e5f8f1a3ff9b [04:13:16] [02miraheze/puppet] 07paladox 03e5f8f1a - Update mw131.yaml [04:13:17] [02puppet] 07paladox synchronize pull request 03#3559: mw* modify keepalive - 13https://github.com/miraheze/puppet/pull/3559 [04:13:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e5f8f1a3ff9b...1e99b7e232c0 [04:13:30] [02miraheze/puppet] 07paladox 031e99b7e - Update mw132.yaml [04:13:33] [02puppet] 07paladox synchronize pull request 03#3559: mw* modify keepalive - 13https://github.com/miraheze/puppet/pull/3559 [04:13:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1e99b7e232c0...67097f8c12ae [04:13:42] [02miraheze/puppet] 07paladox 0367097f8 - Update mw133.yaml [04:13:43] [02puppet] 07paladox synchronize pull request 03#3559: mw* modify keepalive - 13https://github.com/miraheze/puppet/pull/3559 [04:13:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/67097f8c12ae...9fd4204d0f33 [04:13:54] [02miraheze/puppet] 07paladox 039fd4204 - Update mw134.yaml [04:13:56] [02puppet] 07paladox synchronize pull request 03#3559: mw* modify keepalive - 13https://github.com/miraheze/puppet/pull/3559 [04:14:05] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:14:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9fd4204d0f33...5b7fe2d4a972 [04:14:11] [02miraheze/puppet] 07paladox 035b7fe2d - Update mw141.yaml [04:14:13] [02puppet] 07paladox synchronize pull request 03#3559: mw* modify keepalive - 13https://github.com/miraheze/puppet/pull/3559 [04:20:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9241dfdb7f52 [04:20:06] [02miraheze/puppet] 07paladox 039241dfd - nginx: increase ssl cache time and size [04:20:09] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [04:20:13] [02puppet] 07paladox opened pull request 03#3560: nginx: increase ssl cache time and size - 13https://github.com/miraheze/puppet/pull/3560 [04:20:28] [02puppet] 07paladox closed pull request 03#3560: nginx: increase ssl cache time and size - 13https://github.com/miraheze/puppet/pull/3560 [04:20:29] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [04:20:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5b2e57556004...7a56c99b9d66 [04:20:31] [02miraheze/puppet] 07paladox 037a56c99 - nginx: increase ssl cache time and size (#3560) [04:20:34] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [04:28:50] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 6.47, 8.93, 11.68 [04:29:02] PROBLEM - mw131 JobRunner Service on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [04:32:19] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.40, 8.59, 11.58 [04:32:27] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [04:32:52] RECOVERY - mw131 JobRunner Service on mw131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [04:34:37] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.66, 7.46, 10.13 [04:34:56] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.05, 8.14, 11.53 [04:35:27] PROBLEM - mw131 Puppet on mw131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [04:37:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d22ca5be22af [04:37:29] [02miraheze/puppet] 07paladox 03d22ca5b - Disable jobrunner on mw131 and mw133 [04:37:32] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [04:37:35] [02puppet] 07paladox opened pull request 03#3561: Disable jobrunner on mw131 and mw133 - 13https://github.com/miraheze/puppet/pull/3561 [04:37:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d22ca5be22af...ba38935c9d51 [04:37:52] [02miraheze/puppet] 07paladox 03ba38935 - Update mw133.yaml [04:37:55] [02puppet] 07paladox synchronize pull request 03#3561: Disable jobrunner on mw131 and mw133 - 13https://github.com/miraheze/puppet/pull/3561 [04:38:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ba38935c9d51...f4afc3547d91 [04:38:11] [02miraheze/puppet] 07paladox 03f4afc35 - Update mw134.yaml [04:38:12] [02puppet] 07paladox synchronize pull request 03#3561: Disable jobrunner on mw131 and mw133 - 13https://github.com/miraheze/puppet/pull/3561 [04:40:13] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.64, 7.84, 10.02 [04:40:40] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.79, 7.20, 10.16 [04:44:14] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.45, 9.80, 11.64 [04:47:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:52:11] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.12, 8.34, 10.18 [04:53:47] [02puppet] 07paladox edited pull request 03#3561: Disable jobrunner on mw131 and mw133, enable on mw134 - 13https://github.com/miraheze/puppet/pull/3561 [04:54:17] [02puppet] 07paladox closed pull request 03#3561: Disable jobrunner on mw131 and mw133, enable on mw134 - 13https://github.com/miraheze/puppet/pull/3561 [04:54:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/7a56c99b9d66...d5da20d93d8d [04:54:19] [02miraheze/puppet] 07paladox 03d5da20d - Disable jobrunner on mw131 and mw133, enable on mw134 (#3561) [04:54:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [04:54:25] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [04:54:52] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:56:09] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.34, 10.84, 10.72 [04:57:35] PROBLEM - mw133 JobRunner Service on mw133 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [04:57:47] PROBLEM - mw131 JobRunner Service on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [04:58:08] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.53, 9.82, 10.37 [05:00:07] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.53, 11.00, 10.71 [05:01:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2ec69a1ebfdc [05:01:34] [02miraheze/puppet] 07paladox 032ec69a1 - disable jobrunner on mw* apart from mwtask141 [05:01:35] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:01:36] [02puppet] 07paladox opened pull request 03#3562: disable jobrunner on mw* apart from mwtask141 - 13https://github.com/miraheze/puppet/pull/3562 [05:01:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2ec69a1ebfdc...bc811118267f [05:01:58] [02miraheze/puppet] 07paladox 03bc81111 - Update mw134.yaml [05:01:59] [02puppet] 07paladox synchronize pull request 03#3562: disable jobrunner on mw* apart from mwtask141 - 13https://github.com/miraheze/puppet/pull/3562 [05:02:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bc811118267f...25752ad6f259 [05:02:04] [02miraheze/puppet] 07paladox 0325752ad - Update mw132.yaml [05:02:06] [02puppet] 07paladox synchronize pull request 03#3562: disable jobrunner on mw* apart from mwtask141 - 13https://github.com/miraheze/puppet/pull/3562 [05:02:06] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.76, 10.41, 10.53 [05:02:10] [02puppet] 07paladox closed pull request 03#3562: disable jobrunner on mw* apart from mwtask141 - 13https://github.com/miraheze/puppet/pull/3562 [05:02:11] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [05:02:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/d5da20d93d8d...0247dd209916 [05:02:15] [02miraheze/puppet] 07paladox 030247dd2 - disable jobrunner on mw* apart from mwtask141 (#3562) [05:02:16] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:03:42] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:05:03] PROBLEM - mw132 JobRunner Service on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [05:08:04] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.57, 9.52, 10.15 [05:16:59] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [05:18:36] PROBLEM - swiftobject122 APT on swiftobject122 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [05:20:37] PROBLEM - swiftobject122 APT on swiftobject122 is CRITICAL: APT CRITICAL: 86 packages available for upgrade (32 critical updates). [05:34:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.85, 3.20, 2.32 [05:36:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.08, 2.93, 2.34 [06:12:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.00, 11.21, 10.09 [06:12:28] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [06:16:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.28, 10.06, 9.86 [06:22:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.32, 10.31, 10.00 [06:24:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.75, 2.71, 1.85 [06:26:49] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.54, 2.75, 1.98 [06:32:15] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 10 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [06:33:43] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [06:40:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.46, 9.30, 9.96 [06:54:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.23, 9.89, 9.49 [06:58:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 10.08, 9.95, 9.61 [07:01:29] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:02:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.94, 10.70, 10.00 [07:05:43] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wizardia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.07, 11.08, 10.68 [07:24:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.42, 11.04, 10.71 [07:38:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.29, 9.59, 10.19 [07:42:38] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:10:38] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:10:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.50, 7.34, 6.27 [08:12:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.67, 6.76, 6.19 [08:24:48] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [08:53:15] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 10 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [08:55:27] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:30:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.08, 10.45, 9.80 [09:33:45] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [09:36:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.36, 9.78, 9.74 [09:52:43] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:05:43] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wizardia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:09:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.77, 3.75, 2.70 [10:11:30] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.02, 3.20, 2.61 [10:13:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.18, 2.84, 2.56 [10:17:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.44, 3.16, 2.75 [10:19:30] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:20:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.49, 9.64, 8.76 [10:21:29] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 1.278 second response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [10:21:31] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.06, 2.88, 2.76 [10:22:27] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 10 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [10:22:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.53, 8.27, 8.38 [10:30:57] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:32:56] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 1.984 second response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [10:34:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.68, 10.29, 9.34 [10:36:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.45, 9.21, 9.06 [10:41:35] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:50:05] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 8.312 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [10:55:26] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.19, 10.03, 9.00 [10:57:23] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.40, 9.31, 8.86 [11:00:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 0.45, 1.68, 3.58 [11:01:16] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.03, 9.93, 9.19 [11:02:42] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.59, 10.15, 9.21 [11:02:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.73, 1.39, 3.24 [11:03:14] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.40, 11.43, 9.82 [11:05:11] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.77, 10.77, 9.77 [11:05:12] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.20, 10.18, 8.66 [11:06:35] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.29, 11.69, 10.05 [11:07:09] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.92, 11.23, 9.21 [11:07:09] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.48, 12.38, 10.47 [11:08:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 16.16, 11.50, 9.39 [11:10:28] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.07, 10.22, 9.38 [11:10:36] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 14.24, 10.78, 8.31 [11:11:02] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 15.53, 11.27, 9.09 [11:12:01] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 20.06, 13.31, 9.67 [11:12:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 10.02, 10.07, 9.42 [11:12:33] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw142 [11:16:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [11:24:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.89, 9.88, 9.48 [11:26:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.46, 9.68, 9.46 [11:33:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:33:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.40, 3.44, 2.89 [11:35:33] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.52, 3.68, 3.05 [11:39:31] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.89, 3.21, 2.99 [11:40:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.25, 10.05, 9.66 [11:46:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 5.93, 9.03, 9.49 [11:46:37] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.95, 10.03, 11.59 [11:48:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.21, 9.74, 11.75 [11:52:33] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.91, 8.24, 10.17 [11:53:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:53:32] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.04, 9.77, 11.83 [11:53:48] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 10 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [11:58:10] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.08, 7.77, 9.93 [11:59:28] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.56, 7.04, 9.93 [12:04:51] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.66, 8.64, 11.91 [12:06:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.81, 9.70, 11.90 [12:08:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.16, 10.46, 9.55 [12:12:42] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.25, 11.71, 10.29 [12:14:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.68, 7.45, 9.98 [12:14:37] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.04, 11.37, 10.33 [12:14:38] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.96, 8.04, 10.17 [12:15:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.88, 9.39, 12.00 [12:18:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.05, 10.18, 11.91 [12:21:22] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:21:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.25, 10.61, 11.57 [12:22:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.28, 10.07, 10.11 [12:23:02] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:23:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.38, 10.53, 11.41 [12:24:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.45, 10.44, 11.33 [12:25:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.02, 11.07, 11.50 [12:26:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.05, 10.08, 11.11 [12:27:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.36, 10.84, 11.37 [12:30:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.84, 9.82, 9.79 [12:31:43] PROBLEM - db101 Backups SQL on db101 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209666 seconds old and 135805 bytes [12:32:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.02, 7.79, 9.75 [12:33:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.36, 8.35, 10.13 [12:34:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 10.00, 10.03, 9.89 [12:36:31] PROBLEM - xomnipedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'xomnipedia.org' expires in 15 day(s) (Tue 26 Dec 2023 12:12:13 GMT +0000). [12:36:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4299e7659a0c...c334487dfa45 [12:36:47] [02miraheze/ssl] 07MirahezeSSLBot 03c334487 - Bot: Update SSL cert for xomnipedia.org [12:36:50] PROBLEM - wiki.gtnewhorizons.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gtnewhorizons.com' expires in 15 day(s) (Tue 26 Dec 2023 12:27:49 GMT +0000). [12:37:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c334487dfa45...c9b5220f033e [12:37:06] [02miraheze/ssl] 07MirahezeSSLBot 03c9b5220 - Bot: Update SSL cert for wiki.gtnewhorizons.com [12:42:12] PROBLEM - wiki.mcsoft.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mcsoft.org' expires in 15 day(s) (Tue 26 Dec 2023 12:36:10 GMT +0000). [12:42:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c9b5220f033e...8797f12abd5b [12:42:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.06, 10.53, 10.11 [12:42:27] [02miraheze/ssl] 07MirahezeSSLBot 038797f12 - Bot: Update SSL cert for wiki.mcsoft.org [12:44:36] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [12:51:06] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 10 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [13:03:43] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [13:06:15] RECOVERY - xomnipedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'xomnipedia.org' will expire on Sat 09 Mar 2024 11:36:37 GMT +0000. [13:06:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 5.24, 8.78, 9.86 [13:11:07] RECOVERY - wiki.mcsoft.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcsoft.org' will expire on Sat 09 Mar 2024 11:42:17 GMT +0000. [13:12:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.44, 9.94, 9.88 [13:15:14] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:20:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.83, 10.06, 10.02 [13:20:50] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:24:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.93, 10.35, 10.13 [13:26:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.76, 10.19, 10.10 [13:30:03] PROBLEM - vmklegacy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vmklegacy.wiki' expires in 15 day(s) (Tue 26 Dec 2023 12:59:33 GMT +0000). [13:30:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8797f12abd5b...01dc698e2f6b [13:30:18] [02miraheze/ssl] 07MirahezeSSLBot 0301dc698 - Bot: Update SSL cert for vmklegacy.wiki [13:30:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.45, 10.33, 10.16 [13:34:35] RECOVERY - wiki.gtnewhorizons.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gtnewhorizons.com' will expire on Sat 09 Mar 2024 11:36:56 GMT +0000. [13:38:51] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.47, 2.29, 1.31 [13:40:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.97, 1.85, 1.27 [13:46:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.99, 9.36, 10.06 [13:48:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.31, 6.36, 5.81 [13:54:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.90, 7.04, 6.41 [13:56:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.35, 8.90, 9.07 [13:56:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.42, 6.09, 6.14 [13:58:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.86, 9.28, 9.19 [14:05:51] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wizardia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:09:06] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.14, 10.43, 9.82 [14:11:48] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [14:12:56] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 10.02, 10.18, 9.85 [14:15:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [14:20:17] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 10 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [14:24:27] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.98, 10.72, 10.24 [14:29:10] RECOVERY - vmklegacy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vmklegacy.wiki' will expire on Sat 09 Mar 2024 12:30:09 GMT +0000. [14:32:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 7.38, 9.60, 9.98 [14:34:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.56, 2.64, 1.68 [14:36:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.86, 2.38, 1.71 [14:40:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.05, 10.33, 9.92 [14:44:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.88, 11.15, 10.35 [14:45:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [14:46:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.33, 11.50, 10.58 [14:48:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.70, 11.71, 10.78 [14:50:01] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:02:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.06, 11.43, 11.03 [15:04:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.83, 6.10, 5.52 [15:06:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.90, 11.16, 11.02 [15:06:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.82, 5.88, 5.51 [15:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:29] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 13.91, 12.01, 11.40 [15:18:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 7.23, 10.29, 10.91 [15:22:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.81, 8.64, 10.10 [15:28:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.19, 10.31, 10.31 [15:36:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 13.01, 11.40, 10.85 [15:38:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.02, 11.09, 10.80 [15:38:29] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.71, 10.89, 7.72 [15:38:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.88, 2.76, 1.75 [15:40:29] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.01, 9.64, 7.64 [15:40:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.36, 3.38, 2.11 [15:42:49] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.22, 2.54, 1.96 [15:48:48] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.74, 3.51, 2.44 [15:49:28] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 10 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [15:52:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 13.56, 11.83, 11.31 [16:00:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.70, 11.66, 11.58 [16:00:54] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.15, 3.11, 3.27 [16:06:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 4.57, 8.00, 10.10 [16:08:02] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.22, 3.39, 2.19 [16:09:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [16:10:02] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.59, 3.16, 2.26 [16:12:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.19, 10.09, 10.33 [16:19:12] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:35:51] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.007861733437 secs [16:48:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 5.46, 8.36, 9.91 [16:54:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.26, 9.62, 9.85 [17:11:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [17:11:51] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1089104712 secs [17:15:13] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [17:21:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.32, 7.23, 6.64 [17:23:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.43, 7.09, 6.67 [17:27:34] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.97, 6.73, 6.64 [17:31:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 6.88, 6.74 [17:32:16] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.42, 3.23, 2.99 [17:34:12] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.02, 2.80, 2.86 [17:35:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.21, 7.30, 6.90 [17:37:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.63, 6.98, 6.82 [17:38:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.11, 6.75, 6.76 [17:42:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.11, 10.95, 10.67 [17:42:38] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [17:42:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 7.37, 7.02 [17:44:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.24, 10.55, 10.55 [17:44:36] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 10 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [17:44:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.79, 6.55, 6.77 [17:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:52:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.40, 7.26, 6.98 [17:58:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.63, 6.00, 6.55 [18:00:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.21, 11.52, 11.08 [18:02:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.75, 11.53, 11.13 [18:03:43] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [18:04:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.90, 3.67, 2.51 [18:06:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.61, 11.83, 11.33 [18:10:38] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:13:50] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:14:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 8.46, 10.78, 11.23 [18:18:07] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 10 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [18:18:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.30, 3.88, 3.53 [18:20:50] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.59, 3.39, 3.39 [18:24:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.95, 3.67, 3.53 [18:26:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.70, 11.63, 11.19 [18:26:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.22, 3.70, 3.54 [18:28:32] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.45, 9.73, 8.93 [18:28:51] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.51, 3.56, 3.51 [18:30:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.29, 11.56, 11.26 [18:30:28] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.63, 8.85, 8.70 [18:30:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.43, 3.17, 3.37 [18:32:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.02, 11.68, 11.34 [18:35:43] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wizardia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:38:23] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [18:38:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.48, 11.71, 11.51 [18:38:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.25, 3.41, 3.33 [18:40:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.87, 12.16, 11.71 [18:40:49] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.50, 3.01, 3.19 [18:44:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.17, 11.61, 11.61 [18:44:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.71, 4.21, 3.64 [18:47:51] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:50:55] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.43, 3.88, 3.70 [18:52:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.75, 4.07, 3.78 [18:54:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.64, 8.20, 10.02 [18:54:50] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.21, 3.76, 3.71 [19:00:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.57, 10.34, 10.26 [19:00:49] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.07, 2.67, 3.27 [19:04:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.20, 11.03, 10.58 [19:10:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.56, 11.72, 11.04 [19:17:35] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 10 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [19:18:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.23, 11.97, 11.53 [19:20:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.90, 7.01, 6.62 [19:20:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.30, 12.08, 11.63 [19:23:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.29, 7.64, 6.94 [19:25:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.51, 7.32, 6.92 [19:27:39] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.49, 6.74, 6.75 [19:31:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.13, 6.93 [19:33:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.89, 6.71, 6.80 [19:33:51] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5043707192 secs [19:34:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 9.02, 10.86, 11.58 [19:36:23] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:39:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [19:40:13] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.42, 3.01, 2.66 [19:40:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 8.71, 8.45, 10.18 [19:42:08] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.29, 2.75, 2.60 [19:43:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.57, 7.09, 6.86 [19:44:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.51, 10.89, 10.80 [19:47:18] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:47:39] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.32, 6.00, 6.50 [19:50:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.48, 11.78, 11.28 [19:53:40] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [20:06:24] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [20:11:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:15:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [20:15:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.14, 6.88, 6.72 [20:17:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.17, 6.24, 6.50 [20:21:40] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.26, 11.30, 10.98 [20:26:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.72, 11.27, 11.00 [20:31:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.71, 7.08, 6.71 [20:33:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.60, 6.54, 6.56 [20:42:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.02, 11.48, 11.08 [20:44:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.26, 11.41, 11.11 [20:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:46:46] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 10 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [20:48:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.84, 8.01, 7.09 [20:52:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.78, 7.89, 7.25 [20:54:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.76, 7.27 [20:56:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 5.76, 8.45, 10.02 [20:56:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.31, 7.18, 7.11 [20:58:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.47, 7.21 [21:02:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 7.19, 7.16 [21:08:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.99, 6.35, 6.79 [21:16:30] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:23:40] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [21:26:19] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.qadrishattari.xyz All nameservers failed to answer the query. [21:39:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [21:40:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 6.76, 6.43 [21:42:16] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.33, 6.68, 6.45 [21:46:14] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 10 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [21:49:40] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [21:55:02] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp24.miraheze.org - CNAME OK [22:11:09] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:14:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 6.30, 5.96 [22:15:58] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:16:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.85, 6.02, 5.89 [22:38:24] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:39:47] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [22:40:46] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.43, 9.69, 9.07 [22:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [22:46:31] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.64, 9.90, 9.38 [22:54:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.93, 10.38, 9.79 [22:54:34] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/commit/9453dc4f1edf [22:54:35] [02miraheze/RottenLinks] 07paladox 039453dc4 - Redesign RottenLinks slightly to depend on a maintenance script less [22:54:37] [02RottenLinks] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/RottenLinks [22:54:40] [02RottenLinks] 07paladox opened pull request 03#73: Redesign RottenLinks slightly to depend on a maintenance script less - 13https://github.com/miraheze/RottenLinks/pull/73 [22:55:17] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/9453dc4f1edf...e8086b377eb8 [22:55:20] [02miraheze/RottenLinks] 07paladox 03e8086b3 - Update extension.json [22:55:23] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to depend on a maintenance script less - 13https://github.com/miraheze/RottenLinks/pull/73 [22:55:31] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/e8086b377eb8...3c6a64d0be45 [22:55:32] [02miraheze/RottenLinks] 07paladox 033c6a64d - Update extension.json [22:55:35] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to depend on a maintenance script less - 13https://github.com/miraheze/RottenLinks/pull/73 [22:56:20] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/3c6a64d0be45...dc941334e447 [22:56:21] [02miraheze/RottenLinks] 07paladox 03dc94133 - Update CHANGELOG.md [22:56:22] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to depend on a maintenance script less - 13https://github.com/miraheze/RottenLinks/pull/73 [22:56:28] [02RottenLinks] 07paladox edited pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [22:56:41] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [22:56:44] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/dc941334e447...8a6d1b0064d0 [22:56:47] [02miraheze/RottenLinks] 07paladox 038a6d1b0 - Update CHANGELOG.md [22:57:46] [02miraheze/RottenLinks] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/8a6d1b0064d0...d8fb23155ea2 [22:57:47] [02miraheze/RottenLinks] 07github-actions 03d8fb231 - CI: lint code to MediaWiki standards [22:57:48] [02RottenLinks] 07github-actions[bot] synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [22:58:25] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [22:58:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.71, 10.13, 9.84 [22:58:27] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/d8fb23155ea2...0cd4d17193cd [22:58:30] [02miraheze/RottenLinks] 07paladox 030cd4d17 - Update RottenLinksHooks.php [22:59:01] miraheze/RottenLinks - paladox the build has errored. [23:00:44] miraheze/RottenLinks - paladox the build has errored. [23:05:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.39, 10.62, 10.13 [23:07:17] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [23:07:49] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/RottenLinks/compare/0cd4d17193cd...4ffc8a26a348 [23:07:52] [02miraheze/RottenLinks] 07paladox 034ffc8a2 - Create RottenLinksJob.php [23:07:55] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:08:20] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/4ffc8a26a348...9dce5d5efb85 [23:08:21] [02miraheze/RottenLinks] 07paladox 039dce5d5 - Update extension.json [23:08:22] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:09:06] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.39, 10.08, 10.02 [23:09:27] [02miraheze/RottenLinks] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/9dce5d5efb85...d44df6ee0277 [23:09:30] [02miraheze/RottenLinks] 07github-actions 03d44df6e - CI: lint code to MediaWiki standards [23:09:31] [02RottenLinks] 07github-actions[bot] synchronize pull request 03#73: Redesign RottenLinks slightly to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:10:43] miraheze/RottenLinks - paladox the build has errored. [23:12:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.04, 10.36, 10.18 [23:18:42] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.13, 10.02, 10.14 [23:21:13] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [23:24:53] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.91, 2.96, 1.69 [23:25:14] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [23:26:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.23, 2.37, 1.63 [23:33:43] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [23:41:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [23:41:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.58, 10.23, 9.70 [23:43:54] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.73, 10.09, 9.72 [23:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [23:48:51] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.69, 3.39, 2.31 [23:50:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.66, 3.16, 2.37 [23:51:13] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [23:55:13] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical]