[00:01:25] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1195957065 secs [00:03:38] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.86, 2.56, 1.40 [00:05:38] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.15, 2.35, 1.47 [00:08:07] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.19, 6.37, 4.12 [00:17:25] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.01312634349 secs [00:19:32] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:26:20] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:27:05] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.26, 5.57, 3.31 [00:27:39] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 8.46, 11.47, 7.81 [00:28:48] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.13, 7.18, 6.61 [00:29:39] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 2.92, 8.33, 7.10 [00:30:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.81, 7.27, 6.70 [00:32:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.26, 2.10, 3.92 [00:32:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.89, 3.66, 3.26 [00:34:37] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.72, 6.58, 6.56 [00:34:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.56, 3.11, 3.12 [00:36:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.67, 4.15, 4.30 [00:40:29] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.06, 6.97, 6.72 [00:42:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.43, 3.23, 3.89 [00:45:54] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.79, 10.84, 7.02 [00:46:18] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.56, 6.79, 6.74 [00:47:17] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.07, 7.17, 6.60 [00:47:48] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.82, 8.54, 6.62 [00:48:06] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.67, 2.25, 3.23 [00:49:11] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.53, 6.88, 6.56 [00:52:05] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.78, 3.56, 3.56 [00:54:05] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.93, 2.90, 3.32 [01:02:36] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.16, 6.63, 6.67 [01:06:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.85, 3.78, 3.26 [01:07:54] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.01, 3.27, 2.84 [01:08:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.73, 3.36, 3.18 [01:09:48] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.85, 3.06, 2.81 [01:11:12] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:18:05] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [01:20:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.49, 6.75, 6.10 [01:22:18] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.48, 6.26, 6.00 [01:25:25] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:36:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 15.74, 7.82, 4.22 [01:41:11] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [01:44:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.93, 3.69, 3.75 [01:48:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.05, 2.49, 3.26 [01:49:21] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [01:55:19] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:03:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.76, 3.93, 3.21 [02:05:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.52, 4.00, 3.34 [02:11:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.51, 2.83, 3.09 [02:18:35] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:26:20] PROBLEM - drowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'drowiki.com' expires in 13 day(s) (Tue 25 Jul 2023 15:53:31 GMT +0000). [02:39:48] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 6.54, 10.96, 7.38 [02:41:48] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.28, 8.60, 6.94 [02:42:08] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 11.76, 12.89, 7.84 [02:44:09] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 4.74, 9.91, 7.37 [02:47:39] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.67, 7.18, 6.40 [02:51:28] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.71, 6.79, 6.39 [02:56:20] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:01:12] RECOVERY - db101 Backups SQL on db101 is OK: FILE_AGE OK: /var/log/sql-backup.log is 71 seconds old and 0 bytes [03:02:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.57, 3.92, 2.57 [03:04:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.50, 3.46, 2.59 [03:06:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.19, 2.58, 2.36 [03:11:10] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:13:25] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1101992428 secs [03:20:06] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [03:29:25] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09196448326 secs [03:36:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.14, 3.75, 3.04 [03:38:39] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 13.66, 12.47, 7.09 [03:39:02] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 11.28, 12.82, 7.55 [03:39:49] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 14.16, 18.74, 10.20 [03:40:38] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 3.95, 9.19, 6.54 [03:41:01] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.81, 9.58, 7.01 [03:41:11] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [03:42:53] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.25, 3.29, 3.21 [03:43:48] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 3.61, 9.97, 8.50 [03:53:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.98, 7.45, 7.99 [03:57:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.26, 8.07, 8.12 [04:01:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.62, 7.74, 7.97 [04:06:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.84, 4.77, 3.19 [04:10:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.18, 3.81, 3.16 [04:11:11] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:14:48] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.87, 2.98, 2.99 [04:18:37] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:26:20] PROBLEM - drowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'drowiki.com' expires in 13 day(s) (Tue 25 Jul 2023 15:53:31 GMT +0000). [04:39:31] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.64, 6.18, 6.70 [04:43:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.02, 6.71, 6.81 [04:47:56] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=metawiki --user=ManiacACS [04:48:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:48:13] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.85, 6.95, 6.50 [04:49:31] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.11, 6.60, 6.78 [04:50:14] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.09, 6.60, 6.42 [04:56:20] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:13:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.72, 7.26, 6.66 [05:17:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.43, 7.77, 7.03 [05:35:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.01, 7.57, 7.30 [05:36:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.86, 3.05, 2.17 [05:37:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.34, 7.46, 7.29 [05:39:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.36, 7.74, 7.41 [05:41:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.48, 7.67, 7.43 [05:42:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.59, 3.72, 2.84 [05:43:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.06, 7.86, 7.53 [05:44:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.86, 2.95, 2.66 [05:47:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.15, 7.72, 7.57 [06:08:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 17.19, 8.74, 4.62 [06:12:21] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1087893546 secs [06:16:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.50, 3.80, 3.96 [06:17:31] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.38, 6.26, 6.67 [06:20:21] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08526450396 secs [06:20:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.64, 2.54, 3.39 [06:24:41] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.06, 3.39, 2.73 [06:26:41] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.71, 3.13, 2.71 [06:38:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.55, 4.00, 3.14 [06:40:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.40, 3.07, 2.90 [07:08:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.67, 4.98, 3.24 [07:12:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.03, 3.78, 3.25 [07:14:05] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:14:12] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:14:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.80, 2.78, 2.95 [07:18:13] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 15.51, 8.18, 3.73 [07:30:09] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.13, 6.78, 6.49 [07:32:05] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.55, 6.72, 6.51 [07:36:12] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 0.18, 5.21, 7.29 [07:38:13] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 12.08, 7.61, 7.89 [07:49:15] PROBLEM - tudien.como.vn - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tudien.como.vn' expires in 15 day(s) (Fri 28 Jul 2023 07:32:33 GMT +0000). [07:50:45] PROBLEM - wiki.drycleaninglobby.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.drycleaninglobby.net' expires in 15 day(s) (Fri 28 Jul 2023 07:39:47 GMT +0000). [07:50:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/35fd90172c58...8c640c122af7 [07:51:02] [02miraheze/ssl] 07MirahezeSSLBot 038c640c1 - Bot: Update SSL cert for tudien.como.vn [07:58:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8c640c122af7...63afa0113970 [07:58:58] [02miraheze/ssl] 07MirahezeSSLBot 0363afa01 - Bot: Update SSL cert for wiki.drycleaninglobby.net [08:00:21] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1742516458 secs [08:04:21] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.05582439899 secs [08:07:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.75, 2.73, 1.87 [08:09:22] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.89, 5.33, 2.89 [08:13:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.18, 3.51, 2.78 [08:15:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.79, 2.66, 2.56 [08:34:39] PROBLEM - db112 MariaDB Connections on db112 is WARNING: Warning connection usage: 85.2%Current connections: 426 [08:36:56] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:40:37] PROBLEM - db112 MariaDB Connections on db112 is CRITICAL: Critical connection usage: 90.4%Current connections: 452 [08:43:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.02, 6.85, 6.45 [08:45:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.20, 7.29, 6.66 [08:46:34] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.22, 6.35, 5.25 [08:47:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.79, 7.12, 6.68 [08:47:46] RECOVERY - tudien.como.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'tudien.como.vn' will expire on Tue 10 Oct 2023 06:50:49 GMT +0000. [08:48:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.87, 6.31, 5.36 [08:49:15] RECOVERY - wiki.drycleaninglobby.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.drycleaninglobby.net' will expire on Tue 10 Oct 2023 06:58:48 GMT +0000. [08:49:25] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1500240266 secs [08:49:31] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.89, 6.64, 6.55 [08:52:18] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.26, 6.37, 5.60 [08:52:37] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 0.079 second response time [08:52:46] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [08:53:53] PROBLEM - db112 MariaDB on db112 is CRITICAL: Too many connections [08:56:20] PROBLEM - drowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'drowiki.com' expires in 13 day(s) (Tue 25 Jul 2023 15:53:31 GMT +0000). [08:57:53] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 725693 Threads: 500 Questions: 528132381 Slow queries: 5987826 Opens: 175426 Open tables: 2621 Queries per second avg: 727.762 [09:04:50] PROBLEM - db112 MariaDB on db112 is CRITICAL: Too many connections [09:06:12] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [09:07:01] [02ssl] 07Pix1234 opened pull request 03#660: Remove drowiki - no longer points/invalid domain - 13https://github.com/miraheze/ssl/pull/660 [09:07:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.24, 3.10, 1.91 [09:11:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.32, 3.29, 2.33 [09:12:12] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.28, 3.55, 7.40 [09:13:25] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07570248842 secs [09:14:12] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.22, 2.75, 6.64 [09:26:20] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:34:21] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1055672169 secs [09:36:33] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 728013 Threads: 31 Questions: 528318179 Slow queries: 5990588 Opens: 175426 Open tables: 2621 Queries per second avg: 725.698 [09:37:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.35, 3.86, 2.27 [09:37:25] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 30792 MB (23% inode=99%); [09:37:48] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.248 second response time [09:38:14] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6281 bytes in 0.149 second response time [09:38:22] RECOVERY - db112 MariaDB Connections on db112 is OK: OK connection usage: 2.2%Current connections: 11 [09:38:37] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 20261 bytes in 0.173 second response time [09:38:46] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.114 second response time [09:41:25] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1096518636 secs [09:45:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.48, 3.27, 3.25 [09:45:25] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.09838929772 secs [10:02:21] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08609977365 secs [10:26:20] PROBLEM - drowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'drowiki.com' expires in 13 day(s) (Tue 25 Jul 2023 15:53:31 GMT +0000). [10:56:20] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address drowiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:28:32] I’ll teach 10individuals to earn $30k or more within 72 hours but you will pay me 10% of your profit when you receive it. Note only interested people should apply, drop a message let's get started by asking (HOW) [11:36:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.12, 4.71, 2.60 [11:44:54] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.80, 3.27, 3.19 [11:46:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.69, 7.07, 6.66 [11:48:02] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.00, 6.66, 6.56 [11:59:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.66, 6.61, 7.79 [12:06:15] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.06, 6.52, 6.44 [12:07:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.77, 5.49, 6.70 [12:08:50] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.70, 5.60, 3.42 [12:11:59] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.53, 6.71, 6.54 [12:12:41] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.97, 3.56, 2.71 [12:12:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.06, 3.10, 2.89 [12:13:25] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1036044061 secs [12:14:41] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.23, 3.38, 2.75 [12:15:49] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.01, 7.33, 6.83 [12:17:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.71, 7.09, 6.80 [12:18:41] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.36, 3.45, 2.93 [12:20:41] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.09, 3.25, 2.92 [12:21:33] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.93, 6.49, 6.63 [12:26:41] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.93, 3.53, 3.14 [12:34:41] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.17, 3.77, 3.39 [12:35:25] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09062060714 secs [12:36:41] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.62, 3.70, 3.41 [12:36:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.35, 3.22, 2.19 [12:37:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.57, 7.10, 6.71 [12:38:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.49, 3.99, 2.66 [12:39:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.54, 7.13, 6.33 [12:39:19] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 13.16, 9.07, 5.78 [12:39:31] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.32, 6.79, 6.65 [12:40:25] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 17.74, 13.21, 7.26 [12:40:29] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 15.88, 10.09, 6.09 [12:40:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.71, 3.17, 2.52 [12:41:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.53, 6.47, 6.18 [12:41:18] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 9.29, 9.36, 6.32 [12:41:37] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.72, 3.43, 1.69 [12:42:29] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 9.64, 9.10, 6.18 [12:45:21] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 25.33, 18.05, 10.68 [12:45:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.63, 7.15, 6.81 [12:47:31] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.25, 6.71, 6.69 [12:48:41] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.87, 3.28, 3.39 [12:49:37] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 1.61, 3.60, 2.72 [12:51:37] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.28, 2.43, 2.40 [12:52:26] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 3.95, 10.08, 9.84 [12:52:41] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.47, 3.44, 3.44 [12:53:18] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 2.65, 9.05, 9.98 [12:54:41] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.10, 3.66, 3.52 [12:58:41] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.17, 3.21, 3.40 [13:00:33] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:06:57] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:07:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 16.82, 6.18, 3.15 [13:13:11] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [13:15:33] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:16:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 6.60, 6.27 [13:21:43] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.73, 2.47, 3.74 [13:21:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cab4adf34dcb [13:21:59] [02miraheze/puppet] 07paladox 03cab4adf - mem131/141: set threads to two [13:22:01] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [13:22:02] [02puppet] 07paladox opened pull request 03#3288: mem131/141: set threads to two - 13https://github.com/miraheze/puppet/pull/3288 [13:22:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.59, 7.62, 6.80 [13:22:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cab4adf34dcb...88b0b73c39bb [13:22:18] [02miraheze/puppet] 07paladox 0388b0b73 - Update mem141.yaml [13:22:20] [02puppet] 07paladox synchronize pull request 03#3288: mem131/141: set threads to two - 13https://github.com/miraheze/puppet/pull/3288 [13:22:24] [02puppet] 07paladox closed pull request 03#3288: mem131/141: set threads to two - 13https://github.com/miraheze/puppet/pull/3288 [13:22:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/f8cb7e6d6b8a...102c0226f5ef [13:22:27] [02miraheze/puppet] 07paladox 03102c022 - mem131/141: set threads to two (#3288) [13:22:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [13:22:30] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [13:23:42] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.78, 1.94, 3.39 [13:23:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.59, 6.89, 6.64 [13:27:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.95, 6.19, 6.43 [13:31:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7acf3eed349c [13:31:21] [02miraheze/puppet] 07paladox 037acf3ee - Install memcache on test131 [13:31:22] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [13:31:25] [02puppet] 07paladox opened pull request 03#3289: Install memcache on test131 - 13https://github.com/miraheze/puppet/pull/3289 [13:32:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7acf3eed349c...b3dd3151c5da [13:32:05] [02miraheze/puppet] 07paladox 03b3dd315 - Update site.pp [13:32:07] [02puppet] 07paladox synchronize pull request 03#3289: Install memcache on test131 - 13https://github.com/miraheze/puppet/pull/3289 [13:34:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/3b4533cb8058 [13:34:46] [02miraheze/mw-config] 07paladox 033b4533c - Make beta use a different parsercache db [13:34:47] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [13:34:50] [02mw-config] 07paladox opened pull request 03#5280: Make beta use a different parsercache db - 13https://github.com/miraheze/mw-config/pull/5280 [13:35:29] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3b4533cb8058...5085f19e82a8 [13:35:32] [02miraheze/mw-config] 07github-actions 035085f19 - CI: lint code to MediaWiki standards [13:35:35] [02mw-config] 07github-actions[bot] synchronize pull request 03#5280: Make beta use a different parsercache db - 13https://github.com/miraheze/mw-config/pull/5280 [13:36:07] miraheze/mw-config - paladox the build passed. [13:36:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3dd3151c5da...6b06f0fb6354 [13:36:45] [02miraheze/puppet] 07paladox 036b06f0f - Update test131.yaml [13:36:47] [02puppet] 07paladox synchronize pull request 03#3289: Install memcache on test131 - 13https://github.com/miraheze/puppet/pull/3289 [13:36:51] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:37:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6b06f0fb6354...22e947f0df1d [13:37:28] [02miraheze/puppet] 07paladox 0322e947f - Update test131.yaml [13:37:31] [02puppet] 07paladox synchronize pull request 03#3289: Install memcache on test131 - 13https://github.com/miraheze/puppet/pull/3289 [13:38:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/22e947f0df1d...91c507d10d02 [13:38:03] [02miraheze/puppet] 07paladox 0391c507d - Update mediawiki-grants.sql.erb [13:38:04] [02puppet] 07paladox synchronize pull request 03#3289: Install memcache on test131 - 13https://github.com/miraheze/puppet/pull/3289 [13:40:22] [02mw-config] 07paladox synchronize pull request 03#5280: Make beta use a different parsercache db - 13https://github.com/miraheze/mw-config/pull/5280 [13:40:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5085f19e82a8...bfc687bb8b00 [13:40:27] [02miraheze/mw-config] 07paladox 03bfc687b - Update GlobalCache.php [13:41:23] miraheze/mw-config - paladox the build passed. [13:42:25] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:42:58] [02puppet] 07paladox closed pull request 03#3289: Install memcache on test131 - 13https://github.com/miraheze/puppet/pull/3289 [13:42:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/102c0226f5ef...5cf889af290b [13:43:01] [02miraheze/puppet] 07paladox 035cf889a - Install memcache on test131 (#3289) [13:43:02] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [13:43:03] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [13:43:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e5a8cfd7ffac...b94a9fa7b962 [13:43:11] [02miraheze/mw-config] 07paladox 03b94a9fa - Make beta use a different parsercache db (#5280) [13:43:14] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [13:43:17] [02mw-config] 07paladox closed pull request 03#5280: Make beta use a different parsercache db - 13https://github.com/miraheze/mw-config/pull/5280 [13:43:20] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [13:43:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.65, 6.96, 6.47 [13:44:07] miraheze/mw-config - paladox the build passed. [13:44:14] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [13:44:15] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [13:44:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:44:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:45:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5cf889af290b...34e8261e36c2 [13:45:27] [02miraheze/puppet] 07paladox 0334e8261 - Update mediawiki-grants.sql.erb [13:45:53] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:47:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.53, 6.48, 6.40 [14:07:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 16.89, 6.18, 3.14 [14:09:44] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:10:11] !log [@mwtask141] starting deploy of {'config': True} to all [14:10:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:10:50] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw134.miraheze.org [14:10:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:11:36] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 10.89, 11.93, 7.92 [14:13:10] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [14:13:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.25, 3.65, 3.12 [14:13:33] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 5.17, 9.41, 7.48 [14:15:08] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [14:15:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.91, 2.81, 2.88 [14:17:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1b59be050de8 [14:17:14] [02miraheze/puppet] 07paladox 031b59be0 - nutcracker: add test131 as a memcache instance [14:17:16] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [14:17:19] [02puppet] 07paladox opened pull request 03#3290: nutcracker: add test131 as a memcache instance - 13https://github.com/miraheze/puppet/pull/3290 [14:17:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1b59be050de8...a5147b5d2371 [14:17:42] [02miraheze/puppet] 07paladox 03a5147b5 - Update test131.yaml [14:17:43] [02puppet] 07paladox synchronize pull request 03#3290: nutcracker: add test131 as a memcache instance - 13https://github.com/miraheze/puppet/pull/3290 [14:18:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a5147b5d2371...a5971b8f0548 [14:18:04] [02miraheze/puppet] 07paladox 03a5971b8 - Update common.yaml [14:18:05] [02puppet] 07paladox synchronize pull request 03#3290: nutcracker: add test131 as a memcache instance - 13https://github.com/miraheze/puppet/pull/3290 [14:18:17] [02puppet] 07paladox closed pull request 03#3290: nutcracker: add test131 as a memcache instance - 13https://github.com/miraheze/puppet/pull/3290 [14:18:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/34e8261e36c2...333a4429f479 [14:18:22] [02miraheze/puppet] 07paladox 03333a442 - nutcracker: add test131 as a memcache instance (#3290) [14:18:23] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [14:18:24] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [14:19:01] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [14:19:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/910d59062d0c [14:19:55] [02miraheze/mw-config] 07paladox 03910d590 - Add memcached-mem-test for use for beta only [14:19:56] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [14:20:13] [02mw-config] 07paladox opened pull request 03#5281: Add memcached-mem-test for use for beta only - 13https://github.com/miraheze/mw-config/pull/5281 [14:20:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b94a9fa7b962...f2def27ed1c6 [14:20:36] [02miraheze/mw-config] 07paladox 03f2def27 - Add memcached-mem-test for use for beta only (#5281) [14:20:39] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [14:20:40] [02mw-config] 07paladox closed pull request 03#5281: Add memcached-mem-test for use for beta only - 13https://github.com/miraheze/mw-config/pull/5281 [14:20:41] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [14:20:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.86, 6.29, 6.01 [14:21:00] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [14:21:01] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [14:21:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:21:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:21:25] miraheze/mw-config - paladox the build passed. [14:21:34] miraheze/mw-config - paladox the build passed. [14:24:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.79, 6.03, 5.97 [14:31:40] PROBLEM - test131 Disk Space on test131 is WARNING: DISK WARNING - free space: / 2955 MB (10% inode=69%); [14:32:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f2def27ed1c6...071a99be0b81 [14:32:07] [02miraheze/mw-config] 07paladox 03071a99b - Update GlobalCache.php [14:32:19] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [14:32:22] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [14:32:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:04] miraheze/mw-config - paladox the build passed. [14:33:37] RECOVERY - test131 Disk Space on test131 is OK: DISK OK - free space: / 3886 MB (13% inode=69%); [14:35:49] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:36:45] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:49] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown) [14:38:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.85, 4.13, 3.02 [14:40:33] !log [@mwtask141] starting deploy of {'config': True} to all [14:40:37] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw134.miraheze.org [14:40:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:40:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:40:48] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:40:50] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.782 second response time [14:40:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.46, 3.07, 2.76 [14:45:09] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 7.89, 12.32, 8.67 [14:47:09] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 5.80, 10.02, 8.26 [14:51:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.89, 6.84, 6.11 [14:54:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.20, 6.18, 6.02 [15:05:50] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:11:08] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [15:11:49] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 15.07, 11.94, 7.44 [15:13:49] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.50, 9.34, 7.03 [15:15:07] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.98, 7.46, 6.66 [15:17:02] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.41, 7.40, 6.74 [15:28:37] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.25, 6.66, 6.67 [15:31:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/333a4429f479...1c3bf649facf [15:31:50] [02miraheze/puppet] 07paladox 031c3bf64 - swift-container-stats: add dumps-backup [15:33:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1c3bf649facf...c796fd9efc31 [15:33:19] [02miraheze/puppet] 07paladox 03c796fd9 - swift-container-stats: add avatars and awards [15:36:37] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.95, 7.15, 6.83 [15:38:37] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.01, 7.45, 6.98 [15:39:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.68, 3.49, 2.46 [15:40:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.22, 7.23, 6.96 [15:41:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.46, 2.69, 2.29 [15:52:38] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.71, 6.51, 6.79 [16:05:49] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [16:08:46] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [16:15:27] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.22, 6.64, 6.43 [16:21:11] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.54, 6.74, 6.55 [16:25:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c796fd9efc31...0565459c1f30 [16:25:22] [02miraheze/puppet] 07paladox 030565459 - matomo: update to 4.15.0 [16:26:16] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.91, 3.48, 2.54 [16:28:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.25, 7.20, 6.82 [16:30:07] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 2.85, 3.55, 2.83 [16:32:03] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.50, 3.14, 2.76 [16:32:41] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.38, 6.67, 6.68 [16:35:49] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [16:37:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.46, 4.85, 2.72 [16:38:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [16:39:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.30, 3.90, 2.64 [16:43:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.46, 2.89, 2.53 [16:54:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 6.57, 6.08 [16:55:38] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.43, 3.01, 2.79 [16:56:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.80, 6.36, 6.07 [16:57:38]