[00:01:54] RECOVERY - ewgf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ewgf.wiki' will expire on Wed 20 Dec 2023 22:59:05 GMT +0000. [00:03:45] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 17.66, 8.92, 4.02 [00:07:13] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.83, 2.79, 1.93 [00:09:13] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 1.69, 2.51, 1.94 [00:10:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.45, 3.57, 2.01 [00:10:56] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw141 mw142 [00:10:57] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.26, 9.03, 6.39 [00:10:58] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw141 mw142 [00:11:06] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw141 mw142 [00:11:12] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 16.07, 12.49, 7.79 [00:11:41] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw141 [00:12:31] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 14.71, 12.96, 8.05 [00:12:41] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:12:56] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.80, 8.55, 6.55 [00:13:41] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:14:25] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 6.78, 10.70, 7.80 [00:14:38] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.64, 3.14, 2.20 [00:15:13] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 7.18, 11.10, 8.44 [00:16:19] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 7.52, 9.66, 7.76 [00:16:54] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:16:58] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:17:06] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:17:13] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.29, 9.31, 8.12 [00:17:44] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.46, 6.28, 6.91 [00:19:44] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.40, 4.94, 6.33 [00:22:34] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 6.06, 4.04, 2.83 [00:22:35] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 14.37, 11.24, 8.33 [00:22:56] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.14, 11.08, 8.52 [00:23:18] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.93, 4.54, 3.15 [00:24:32] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 10.94, 9.65, 7.66 [00:26:08] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 18.68, 11.99, 7.94 [00:26:35] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.10, 11.73, 9.25 [00:27:16] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:28:32] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 13.82, 11.58, 8.82 [00:28:35] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.93, 12.56, 9.87 [00:29:25] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 9.063 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:30:08] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 9.27, 10.62, 8.36 [00:30:56] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 7.74, 11.10, 9.86 [00:31:46] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw142 mw143 [00:31:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.23, 3.36, 2.59 [00:32:06] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 12.21, 10.94, 8.73 [00:32:35] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 9.38, 11.89, 10.32 [00:32:56] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.97, 9.71, 9.50 [00:33:44] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:33:52] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.46, 3.00, 2.56 [00:33:54] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:34:06] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 4.67, 8.69, 8.18 [00:34:32] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.53, 3.95, 3.80 [00:34:32] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 5.72, 9.95, 9.30 [00:34:35] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.55, 9.63, 9.68 [00:35:17] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.72, 3.38, 3.66 [00:36:33] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:37:52] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.41, 3.63, 2.95 [00:38:32] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 3.081 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:39:07] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.97, 2.84, 3.39 [00:40:32] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.03, 2.55, 3.27 [00:40:43] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:43:51] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.50, 3.11, 2.98 [00:47:53] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.20, 3.94, 3.34 [00:49:52] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.44, 3.42, 3.23 [00:51:18] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.53, 3.46, 3.21 [00:51:54] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.36, 3.57, 3.29 [00:53:54] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.75, 3.78, 3.42 [00:55:53] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.75, 3.01, 3.18 [00:57:14] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.81, 3.22, 3.19 [01:01:53] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.58, 3.47, 3.33 [01:03:53] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.05, 3.15, 3.21 [01:06:59] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.70, 3.45, 3.29 [01:08:55] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.81, 3.23, 3.22 [01:11:22] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [01:14:49] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [01:33:16] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [01:39:22] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [01:44:49] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [01:47:31] PROBLEM - legacygt.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'legacygt.wiki' expires in 15 day(s) (Wed 20 Dec 2023 01:35:44 GMT +0000). [01:47:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e81e2c826625...498664e15633 [01:47:48] [02miraheze/ssl] 07MirahezeSSLBot 03498664e - Bot: Update SSL cert for legacygt.wiki [01:58:03] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.45, 3.13, 3.04 [02:00:27] PROBLEM - test131 HTTPS on test131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [02:01:13] PROBLEM - test131 SSH on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:01:18] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:01:54] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.69, 3.01, 3.01 [02:02:17] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:02:20] PROBLEM - test131 Redis Process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:02:21] PROBLEM - test131 conntrack_table_size on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:03:05] PROBLEM - test131 Current Load on test131 is CRITICAL: LOAD CRITICAL - total load average: 89.70, 62.11, 27.90 [02:03:09] RECOVERY - test131 SSH on test131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [02:03:18] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.624 second response time [02:03:22] PROBLEM - test131 NTP time on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:03:23] PROBLEM - test131 PowerDNS Recursor on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:03:25] PROBLEM - test131 php-fpm on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:04:15] RECOVERY - test131 Redis Process on test131 is OK: PROCS OK: 1 process with args 'redis-server' [02:04:16] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [02:04:19] RECOVERY - test131 conntrack_table_size on test131 is OK: OK: nf_conntrack is 0 % full [02:04:34] RECOVERY - test131 HTTPS on test131 is OK: HTTP OK: HTTP/2 301 - 349 bytes in 0.021 second response time [02:05:04] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:05:19] RECOVERY - test131 PowerDNS Recursor on test131 is OK: DNS OK: 0.477 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [02:05:21] RECOVERY - test131 php-fpm on test131 is OK: PROCS OK: 11 processes with command name 'php-fpm8.2' [02:05:26] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset -0.002079844475 secs [02:12:43] PROBLEM - wiki.evscope.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.evscope.org' expires in 15 day(s) (Wed 20 Dec 2023 02:10:17 GMT +0000). [02:12:57] PROBLEM - test131 Current Load on test131 is WARNING: LOAD WARNING - total load average: 0.10, 8.73, 14.84 [02:12:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/498664e15633...25d69a9b4e01 [02:12:58] [02miraheze/ssl] 07MirahezeSSLBot 0325d69a9 - Bot: Update SSL cert for wiki.evscope.org [02:14:57] RECOVERY - test131 Current Load on test131 is OK: LOAD OK - total load average: 0.17, 5.89, 13.06 [02:16:04] PROBLEM - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipedia.yogfront.ooo' expires in 15 day(s) (Wed 20 Dec 2023 02:13:02 GMT +0000). [02:16:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/25d69a9b4e01...9bf52a00983c [02:16:19] [02miraheze/ssl] 07MirahezeSSLBot 039bf52a0 - Bot: Update SSL cert for wikipedia.yogfront.ooo [02:16:33] RECOVERY - legacygt.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'legacygt.wiki' will expire on Sun 03 Mar 2024 00:47:38 GMT +0000. [02:19:35] PROBLEM - evil.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'evil.wiki' expires in 15 day(s) (Wed 20 Dec 2023 02:03:09 GMT +0000). [02:19:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9bf52a00983c...e66f5e879dd9 [02:19:51] [02miraheze/ssl] 07MirahezeSSLBot 03e66f5e8 - Bot: Update SSL cert for evil.wiki [02:30:45] PROBLEM - test131 HTTPS on test131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [02:31:13] PROBLEM - test131 SSH on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:29] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:19] PROBLEM - test131 Redis Process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:32:42] PROBLEM - test131 conntrack_table_size on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:32:58] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:33:00] PROBLEM - test131 NTP time on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:33:16] PROBLEM - test131 JobChron Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:33:25] PROBLEM - test131 PowerDNS Recursor on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:33:32] PROBLEM - test131 poolcounter process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:33:39] PROBLEM - test131 php-fpm on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:33:46] PROBLEM - test131 Disk Space on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:33:49] PROBLEM - test131 Puppet on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:33:51] PROBLEM - test131 ferm_active on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:33:57] PROBLEM - test131 Current Load on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:34:32] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.71, 3.26, 2.58 [02:36:32] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.52, 2.90, 2.52 [02:40:00] PROBLEM - mystiverse.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mystiverse.wiki' expires in 15 day(s) (Wed 20 Dec 2023 02:23:56 GMT +0000). [02:40:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e66f5e879dd9...568635bd99ab [02:40:16] [02miraheze/ssl] 07MirahezeSSLBot 03568635b - Bot: Update SSL cert for mystiverse.wiki [02:40:30] PROBLEM - wiki.recaptime.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.recaptime.eu.org' expires in 15 day(s) (Wed 20 Dec 2023 02:28:09 GMT +0000). [02:40:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/568635bd99ab...e918170069be [02:40:48] [02miraheze/ssl] 07MirahezeSSLBot 03e918170 - Bot: Update SSL cert for wiki.recaptime.eu.org [02:41:26] RECOVERY - wiki.evscope.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.evscope.org' will expire on Sun 03 Mar 2024 01:12:50 GMT +0000. [02:42:32] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.63, 3.38, 2.83 [02:42:35] PROBLEM - wiki.kachow.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kachow.org' expires in 15 day(s) (Wed 20 Dec 2023 02:35:32 GMT +0000). [02:42:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e918170069be...7033a0a76d0b [02:42:50] [02miraheze/ssl] 07MirahezeSSLBot 037033a0a - Bot: Update SSL cert for wiki.kachow.org [02:45:09] RECOVERY - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipedia.yogfront.ooo' will expire on Sun 03 Mar 2024 01:16:10 GMT +0000. [02:46:11] RECOVERY - test131 Redis Process on test131 is OK: PROCS OK: 1 process with args 'redis-server' [02:46:17] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.392 second response time [02:46:29] RECOVERY - test131 conntrack_table_size on test131 is OK: OK: nf_conntrack is 0 % full [02:46:32] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.39, 3.00, 2.81 [02:46:41] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset -0.002061605453 secs [02:46:52] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [02:47:00] RECOVERY - test131 JobChron Service on test131 is OK: PROCS OK: 1 process with args 'redisJobChronService' [02:47:03] RECOVERY - test131 PowerDNS Recursor on test131 is OK: DNS OK: 0.305 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [02:47:13] RECOVERY - test131 poolcounter process on test131 is OK: PROCS OK: 1 process with UID = 998 (poolcounter), command name 'poolcounterd' [02:47:21] RECOVERY - test131 php-fpm on test131 is OK: PROCS OK: 11 processes with command name 'php-fpm8.2' [02:47:24] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:47:34] RECOVERY - test131 Disk Space on test131 is OK: DISK OK - free space: / 4586MiB (16% inode=67%); [02:47:44] RECOVERY - test131 ferm_active on test131 is OK: OK ferm input default policy is set [02:47:45] RECOVERY - test131 HTTPS on test131 is OK: HTTP OK: HTTP/2 301 - 349 bytes in 0.020 second response time [02:47:51] RECOVERY - test131 SSH on test131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [02:49:09] RECOVERY - evil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'evil.wiki' will expire on Sun 03 Mar 2024 01:19:40 GMT +0000. [02:56:32] PROBLEM - wiki.veloren.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.veloren.net' expires in 15 day(s) (Wed 20 Dec 2023 02:39:13 GMT +0000). [02:56:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7033a0a76d0b...81d774ff97be [02:56:47] [02miraheze/ssl] 07MirahezeSSLBot 0381d774f - Bot: Update SSL cert for wiki.veloren.net [03:00:00] PROBLEM - test131 SSH on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:06] PROBLEM - test131 HTTPS on test131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10008 milliseconds [03:00:31] RECOVERY - db131 Backups SQL on db131 is OK: FILE_AGE OK: /var/log/sql-backup.log is 30 seconds old and 0 bytes [03:00:34] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:57] PROBLEM - wikicompliance.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikicompliance.com' expires in 15 day(s) (Wed 20 Dec 2023 02:53:03 GMT +0000). [03:01:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/81d774ff97be...8b7256d0f6ee [03:01:12] [02miraheze/ssl] 07MirahezeSSLBot 038b7256d - Bot: Update SSL cert for wikicompliance.com [03:01:38] PROBLEM - test131 JobChron Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:01:38] PROBLEM - test131 PowerDNS Recursor on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:01:42] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:01:52] PROBLEM - test131 NTP time on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:01:53] PROBLEM - test131 poolcounter process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:01:59] PROBLEM - test131 Puppet on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:02:03] PROBLEM - test131 php-fpm on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:02:17] PROBLEM - test131 Disk Space on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:02:34] PROBLEM - test131 ferm_active on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:02:38] PROBLEM - test131 Redis Process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:03:36] RECOVERY - test131 JobChron Service on test131 is OK: PROCS OK: 1 process with args 'redisJobChronService' [03:03:38] RECOVERY - test131 PowerDNS Recursor on test131 is OK: DNS OK: 0.193 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:03:41] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [03:03:49] RECOVERY - test131 poolcounter process on test131 is OK: PROCS OK: 1 process with UID = 998 (poolcounter), command name 'poolcounterd' [03:03:57] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset -0.002101302147 secs [03:03:57] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:04:00] RECOVERY - test131 php-fpm on test131 is OK: PROCS OK: 11 processes with command name 'php-fpm8.2' [03:04:02] RECOVERY - test131 SSH on test131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [03:04:13] RECOVERY - test131 HTTPS on test131 is OK: HTTP OK: HTTP/2 301 - 349 bytes in 0.052 second response time [03:04:15] RECOVERY - test131 Disk Space on test131 is OK: DISK OK - free space: / 4585MiB (16% inode=67%); [03:04:32] RECOVERY - test131 ferm_active on test131 is OK: OK ferm input default policy is set [03:04:33] RECOVERY - test131 Redis Process on test131 is OK: PROCS OK: 1 process with args 'redis-server' [03:04:42] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.429 second response time [03:08:43] PROBLEM - dragdown.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dragdown.wiki' expires in 15 day(s) (Wed 20 Dec 2023 02:47:07 GMT +0000). [03:08:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8b7256d0f6ee...5a203c6feaad [03:08:59] [02miraheze/ssl] 07MirahezeSSLBot 035a203c6 - Bot: Update SSL cert for dragdown.wiki [03:08:59] RECOVERY - mystiverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mystiverse.wiki' will expire on Sun 03 Mar 2024 01:40:06 GMT +0000. [03:10:00] RECOVERY - wiki.recaptime.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.eu.org' will expire on Sun 03 Mar 2024 01:40:39 GMT +0000. [03:11:10] RECOVERY - wiki.kachow.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kachow.org' will expire on Sun 03 Mar 2024 01:42:41 GMT +0000. [03:12:35] PROBLEM - persist.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'persist.wiki' expires in 15 day(s) (Wed 20 Dec 2023 02:59:16 GMT +0000). [03:12:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5a203c6feaad...e05054a68406 [03:12:49] [02miraheze/ssl] 07MirahezeSSLBot 03e05054a - Bot: Update SSL cert for persist.wiki [03:19:35] PROBLEM - mariopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mariopedia.org' expires in 15 day(s) (Wed 20 Dec 2023 03:03:36 GMT +0000). [03:19:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e05054a68406...eb3e094e3afb [03:19:50] [02miraheze/ssl] 07MirahezeSSLBot 03eb3e094 - Bot: Update SSL cert for mariopedia.org [03:22:57] PROBLEM - test131 Current Load on test131 is WARNING: LOAD WARNING - total load average: 0.12, 1.34, 14.72 [03:24:57] RECOVERY - test131 Current Load on test131 is OK: LOAD OK - total load average: 0.10, 0.96, 12.96 [03:25:07] PROBLEM - swests.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'swests.org' expires in 15 day(s) (Wed 20 Dec 2023 03:16:46 GMT +0000). [03:25:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/eb3e094e3afb...38e19e34f4cf [03:25:20] [02miraheze/ssl] 07MirahezeSSLBot 0338e19e3 - Bot: Update SSL cert for swests.org [03:28:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 6.31, 6.15 [03:29:44] PROBLEM - test131 SSH on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:29:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.02, 5.69, 5.96 [03:30:26] PROBLEM - test131 HTTPS on test131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10001 milliseconds [03:30:52] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:51] PROBLEM - test131 JobChron Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:31:57] PROBLEM - test131 Current Load on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:32:02] PROBLEM - test131 poolcounter process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:32:05] PROBLEM - test131 Puppet on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:32:18] PROBLEM - test131 php-fpm on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:32:23] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:32:33] PROBLEM - test131 Redis Process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:32:38] PROBLEM - test131 PowerDNS Recursor on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:32:46] PROBLEM - test131 Disk Space on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:33:12] PROBLEM - test131 ferm_active on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:33:26] PROBLEM - test131 conntrack_table_size on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:33:40] PROBLEM - test131 NTP time on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:34:09] PROBLEM - knifepointhorror.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'knifepointhorror.wiki' expires in 15 day(s) (Wed 20 Dec 2023 03:25:52 GMT +0000). [03:34:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/38e19e34f4cf...af35d70d2566 [03:34:25] [02miraheze/ssl] 07MirahezeSSLBot 03af35d70 - Bot: Update SSL cert for knifepointhorror.wiki [03:35:31] RECOVERY - test131 conntrack_table_size on test131 is OK: OK: nf_conntrack is 0 % full [03:35:42] RECOVERY - test131 ferm_active on test131 is OK: OK ferm input default policy is set [03:35:44] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset -0.002927899361 secs [03:35:52] RECOVERY - test131 SSH on test131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [03:36:42] RECOVERY - test131 HTTPS on test131 is OK: HTTP OK: HTTP/2 301 - 349 bytes in 0.014 second response time [03:36:45] RECOVERY - test131 JobChron Service on test131 is OK: PROCS OK: 1 process with args 'redisJobChronService' [03:36:54] RECOVERY - test131 poolcounter process on test131 is OK: PROCS OK: 1 process with UID = 998 (poolcounter), command name 'poolcounterd' [03:37:01] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:37:08] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.517 second response time [03:37:11] RECOVERY - test131 php-fpm on test131 is OK: PROCS OK: 11 processes with command name 'php-fpm8.2' [03:37:20] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [03:37:23] RECOVERY - test131 Redis Process on test131 is OK: PROCS OK: 1 process with args 'redis-server' [03:37:29] RECOVERY - test131 PowerDNS Recursor on test131 is OK: DNS OK: 0.138 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:37:44] RECOVERY - test131 Disk Space on test131 is OK: DISK OK - free space: / 4585MiB (16% inode=67%); [03:37:56] RECOVERY - dragdown.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'dragdown.wiki' will expire on Sun 03 Mar 2024 02:08:49 GMT +0000. [03:41:10] RECOVERY - persist.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'persist.wiki' will expire on Sun 03 Mar 2024 02:12:41 GMT +0000. [03:44:47] PROBLEM - wiki.yahyabd.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yahyabd.xyz' expires in 15 day(s) (Wed 20 Dec 2023 03:41:26 GMT +0000). [03:45:05] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/af35d70d2566...4038724f8913 [03:45:07] [02miraheze/ssl] 07MirahezeSSLBot 034038724 - Bot: Update SSL cert for wiki.yahyabd.xyz [03:49:09] RECOVERY - mariopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mariopedia.org' will expire on Sun 03 Mar 2024 02:19:40 GMT +0000. [03:52:27] PROBLEM - test131 Current Load on test131 is WARNING: LOAD WARNING - total load average: 0.05, 2.60, 14.84 [03:54:14] RECOVERY - swests.org - LetsEncrypt on sslhost is OK: OK - Certificate 'swests.org' will expire on Sun 03 Mar 2024 02:25:13 GMT +0000. [03:54:24] RECOVERY - test131 Current Load on test131 is OK: LOAD OK - total load average: 0.08, 1.80, 13.12 [03:54:36] RECOVERY - wiki.veloren.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.veloren.net' will expire on Sun 03 Mar 2024 01:56:37 GMT +0000. [03:55:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.22, 6.54, 6.11 [03:57:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.75, 6.29, 6.05 [03:58:57] RECOVERY - wikicompliance.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikicompliance.com' will expire on Sun 03 Mar 2024 02:01:03 GMT +0000. [04:00:57] PROBLEM - wiki.aoemods.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.aoemods.com' expires in 15 day(s) (Wed 20 Dec 2023 03:33:23 GMT +0000). [04:00:59] PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'roll-of-arms.com' expires in 15 day(s) (Wed 20 Dec 2023 03:47:23 GMT +0000). [04:01:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4038724f8913...9f759bd64f08 [04:01:12] [02miraheze/ssl] 07MirahezeSSLBot 039f759bd - Bot: Update SSL cert for wiki.aoemods.com [04:01:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9f759bd64f08...d358341f435d [04:01:27] [02miraheze/ssl] 07MirahezeSSLBot 03d358341 - Bot: Update SSL cert for roll-of-arms.com [04:04:17] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [04:10:22] PROBLEM - buildabearwiki.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'buildabearwiki.info' expires in 15 day(s) (Wed 20 Dec 2023 04:06:01 GMT +0000). [04:10:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d358341f435d...0536a358d04a [04:10:37] [02miraheze/ssl] 07MirahezeSSLBot 030536a35 - Bot: Update SSL cert for buildabearwiki.info [04:14:03] RECOVERY - wiki.yahyabd.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yahyabd.xyz' will expire on Sun 03 Mar 2024 02:44:57 GMT +0000. [04:24:47] PROBLEM - wiki.infomedia.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.infomedia.co.uk' expires in 15 day(s) (Wed 20 Dec 2023 04:02:16 GMT +0000). [04:24:49] PROBLEM - squareenixwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'squareenixwiki.com' expires in 15 day(s) (Wed 20 Dec 2023 03:55:30 GMT +0000). [04:24:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0536a358d04a...835b12c7d942 [04:25:01] [02miraheze/ssl] 07MirahezeSSLBot 03835b12c - Bot: Update SSL cert for wiki.infomedia.co.uk [04:25:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/835b12c7d942...85fe39d42b5b [04:25:13] [02miraheze/ssl] 07MirahezeSSLBot 0385fe39d - Bot: Update SSL cert for squareenixwiki.com [04:29:18] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:30:36] PROBLEM - test131 HTTPS on test131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10000 milliseconds [04:31:12] PROBLEM - test131 SSH on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:31:49] PROBLEM - test131 Puppet on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:31:57] PROBLEM - test131 Current Load on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:15] PROBLEM - test131 NTP time on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:19] PROBLEM - test131 Redis Process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:21] PROBLEM - test131 conntrack_table_size on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:25] PROBLEM - test131 Disk Space on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:26] RECOVERY - knifepointhorror.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'knifepointhorror.wiki' will expire on Sun 03 Mar 2024 02:34:15 GMT +0000. [04:32:38] PROBLEM - test131 PowerDNS Recursor on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:41] PROBLEM - test131 ferm_active on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:43] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:48] PROBLEM - iol.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'iol.wiki' expires in 15 day(s) (Wed 20 Dec 2023 04:29:01 GMT +0000). [04:33:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/85fe39d42b5b...c66bb011d07d [04:33:04] [02miraheze/ssl] 07MirahezeSSLBot 03c66bb01 - Bot: Update SSL cert for iol.wiki [04:33:09] PROBLEM - test131 php-fpm on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:33:09] PROBLEM - test131 JobChron Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:33:29] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.036 second response time [04:33:32] PROBLEM - test131 poolcounter process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:33:47] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:34:15] RECOVERY - test131 Redis Process on test131 is OK: PROCS OK: 1 process with args 'redis-server' [04:34:19] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset -0.00373712182 secs [04:34:19] RECOVERY - test131 conntrack_table_size on test131 is OK: OK: nf_conntrack is 0 % full [04:34:23] RECOVERY - test131 Disk Space on test131 is OK: DISK OK - free space: / 4583MiB (16% inode=67%); [04:34:33] RECOVERY - test131 PowerDNS Recursor on test131 is OK: DNS OK: 0.145 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [04:34:41] RECOVERY - test131 ferm_active on test131 is OK: OK ferm input default policy is set [04:34:42] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [04:34:43] RECOVERY - test131 HTTPS on test131 is OK: HTTP OK: HTTP/2 301 - 349 bytes in 0.015 second response time [04:35:05] RECOVERY - test131 php-fpm on test131 is OK: PROCS OK: 11 processes with command name 'php-fpm8.2' [04:35:06] RECOVERY - test131 JobChron Service on test131 is OK: PROCS OK: 1 process with args 'redisJobChronService' [04:35:16] RECOVERY - test131 SSH on test131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [04:35:28] RECOVERY - test131 poolcounter process on test131 is OK: PROCS OK: 1 process with UID = 998 (poolcounter), command name 'poolcounterd' [04:38:20] PROBLEM - en.omniversalis.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.omniversalis.org' expires in 15 day(s) (Wed 20 Dec 2023 04:17:35 GMT +0000). [04:38:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c66bb011d07d...ad648fa17ff2 [04:38:35] [02miraheze/ssl] 07MirahezeSSLBot 03ad648fa - Bot: Update SSL cert for en.omniversalis.org [04:38:44] PROBLEM - royal-wiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'royal-wiki.org' expires in 15 day(s) (Wed 20 Dec 2023 04:21:30 GMT +0000). [04:38:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ad648fa17ff2...1ae98dc40781 [04:38:59] [02miraheze/ssl] 07MirahezeSSLBot 031ae98dc - Bot: Update SSL cert for royal-wiki.org [04:39:44] RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Sun 03 Mar 2024 03:10:27 GMT +0000. [04:43:40] PROBLEM - test131 Current Load on test131 is WARNING: LOAD WARNING - total load average: 0.06, 8.19, 15.76 [04:47:34] RECOVERY - test131 Current Load on test131 is OK: LOAD OK - total load average: 0.18, 3.83, 12.27 [04:53:33] RECOVERY - wiki.infomedia.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.infomedia.co.uk' will expire on Sun 03 Mar 2024 03:24:52 GMT +0000. [04:53:38] RECOVERY - squareenixwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'squareenixwiki.com' will expire on Sun 03 Mar 2024 03:25:04 GMT +0000. [04:58:11] PROBLEM - wiki.virtualjet.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.virtualjet.net' expires in 15 day(s) (Wed 20 Dec 2023 04:45:41 GMT +0000). [04:58:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1ae98dc40781...fd48085266f9 [04:58:30] [02miraheze/ssl] 07MirahezeSSLBot 03fd48085 - Bot: Update SSL cert for wiki.virtualjet.net [04:58:57] RECOVERY - wiki.aoemods.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aoemods.com' will expire on Sun 03 Mar 2024 03:01:04 GMT +0000. [04:58:59] RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Sun 03 Mar 2024 03:01:18 GMT +0000. [05:00:14] PROBLEM - test131 Current Load on test131 is CRITICAL: LOAD CRITICAL - total load average: 24.60, 20.81, 14.02 [05:02:05] RECOVERY - iol.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'iol.wiki' will expire on Sun 03 Mar 2024 03:32:54 GMT +0000. [05:02:11] PROBLEM - test131 Current Load on test131 is WARNING: LOAD WARNING - total load average: 4.00, 14.33, 12.45 [05:04:08] RECOVERY - test131 Current Load on test131 is OK: LOAD OK - total load average: 0.89, 9.70, 10.98 [05:05:40] PROBLEM - wiki.mostly.vet - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mostly.vet' expires in 15 day(s) (Wed 20 Dec 2023 04:59:01 GMT +0000). [05:05:51] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:05:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fd48085266f9...cc3ae36699ad [05:05:58] [02miraheze/ssl] 07MirahezeSSLBot 03cc3ae36 - Bot: Update SSL cert for wiki.mostly.vet [05:07:09] RECOVERY - en.omniversalis.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.omniversalis.org' will expire on Sun 03 Mar 2024 03:38:26 GMT +0000. [05:07:58] RECOVERY - royal-wiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'royal-wiki.org' will expire on Sun 03 Mar 2024 03:38:49 GMT +0000. [05:28:06] RECOVERY - wiki.virtualjet.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.virtualjet.net' will expire on Sun 03 Mar 2024 03:58:20 GMT +0000. [05:33:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5c27a078c04a [05:33:35] [02miraheze/puppet] 07paladox 035c27a07 - rsyslog: set maxDiskSpace [05:33:36] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:33:37] [02puppet] 07paladox opened pull request 03#3540: rsyslog: set maxDiskSpace - 13https://github.com/miraheze/puppet/pull/3540 [05:34:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5c27a078c04a...2090dc936e87 [05:34:38] [02miraheze/puppet] 07paladox 032090dc9 - Update remote_syslog_rule.conf.erb [05:34:40] [02puppet] 07paladox synchronize pull request 03#3540: rsyslog: set maxDiskSpace - 13https://github.com/miraheze/puppet/pull/3540 [05:34:52] RECOVERY - wiki.mostly.vet - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mostly.vet' will expire on Sun 03 Mar 2024 04:05:49 GMT +0000. [05:35:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2090dc936e87...42b531e72ffd [05:35:01] [02miraheze/puppet] 07paladox 0342b531e - Update remote_syslog_rule_parse_json.conf.erb [05:35:02] [02puppet] 07paladox synchronize pull request 03#3540: rsyslog: set maxDiskSpace - 13https://github.com/miraheze/puppet/pull/3540 [05:35:35] [02puppet] 07paladox closed pull request 03#3540: rsyslog: set maxDiskSpace - 13https://github.com/miraheze/puppet/pull/3540 [05:35:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/cba53c58a33d...d5a3d56c87d0 [05:35:41] [02miraheze/puppet] 07paladox 03d5a3d56 - rsyslog: set maxDiskSpace (#3540) [05:35:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [05:35:45] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:37:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/198cdeaf6a6f [05:37:35] [02miraheze/puppet] 07paladox 03198cdea - rsyslog: unset saveonshutdown [05:37:38] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:37:39] [02puppet] 07paladox opened pull request 03#3541: rsyslog: unset saveonshutdown - 13https://github.com/miraheze/puppet/pull/3541 [05:37:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/198cdeaf6a6f...41f0cae7938d [05:37:56] [02miraheze/puppet] 07paladox 0341f0cae - Update remote_syslog_rule.conf.erb [05:37:58] [02puppet] 07paladox synchronize pull request 03#3541: rsyslog: unset saveonshutdown - 13https://github.com/miraheze/puppet/pull/3541 [05:38:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/41f0cae7938d...6b689763b4e0 [05:38:07] [02miraheze/puppet] 07paladox 036b68976 - Update remote_syslog_rule_parse_json.conf.erb [05:38:08] [02puppet] 07paladox synchronize pull request 03#3541: rsyslog: unset saveonshutdown - 13https://github.com/miraheze/puppet/pull/3541 [05:39:22] [02puppet] 07paladox closed pull request 03#3541: rsyslog: unset saveonshutdown - 13https://github.com/miraheze/puppet/pull/3541 [05:39:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/d5a3d56c87d0...a1f365997704 [05:39:26] [02miraheze/puppet] 07paladox 03a1f3659 - rsyslog: unset saveonshutdown (#3541) [05:39:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [05:39:29] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:39:30] [02puppet] 07paladox closed pull request 03#3483: swift: Set "Connection: close" if using HEAD - 13https://github.com/miraheze/puppet/pull/3483 [05:39:32] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [05:39:34] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [05:51:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/e7688d7d761d [05:51:23] [02miraheze/mw-config] 07paladox 03e7688d7 - Set wgTorBlockProxy and wgTorIPs [05:51:25] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [05:51:27] [02mw-config] 07paladox opened pull request 03#5391: Set wgTorBlockProxy and wgTorIPs - 13https://github.com/miraheze/mw-config/pull/5391 [05:52:18] miraheze/mw-config - paladox the build passed. [05:52:47] [02mw-config] 07paladox closed pull request 03#5391: Set wgTorBlockProxy and wgTorIPs - 13https://github.com/miraheze/mw-config/pull/5391 [05:52:50] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [05:52:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/255194e40357...67cf0c6a3cc3 [05:52:52] [02miraheze/mw-config] 07paladox 0367cf0c6 - Set wgTorBlockProxy and wgTorIPs (#5391) [05:52:55] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [05:52:59] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [05:53:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:10] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [05:53:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:16] !log [paladox@mwtask141] starting deploy of {'config': True} to all [05:53:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:25] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [05:53:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:41] miraheze/mw-config - paladox the build passed. [05:53:47] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [05:53:51] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 3s [05:53:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:58] !log [paladox@test131] starting deploy of {'config': True} to all [05:53:59] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:53:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:54:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:54:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:07:41] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu67] [06:13:42] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:11:13] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [07:11:37] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 23.03, 16.95, 8.70 [07:12:15] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw143 [07:13:53] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 29.49, 14.03, 7.17 [07:14:09] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:13] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [07:16:10] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.608 second response time [07:16:54]