[00:02:48] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 11 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [00:02:53] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.93, 6.25, 2.92 [00:03:36] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.54, 3.17, 1.94 [00:04:02] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.03, 5.99, 3.82 [00:04:57] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.53, 2.91, 1.74 [00:05:36] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.00, 2.71, 1.93 [00:06:02] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.27, 4.67, 3.61 [00:10:55] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 11 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [00:11:01] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.41, 3.91, 2.57 [00:12:59] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.66, 3.83, 2.70 [00:13:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.80, 6.91, 6.39 [00:14:53] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.04, 7.54, 6.79 [00:15:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.93, 6.18, 6.18 [00:15:27] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.008 second response time [00:15:47] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.019 second response time [00:15:56] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw133 [00:15:56] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:56] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 [00:16:20] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:16:53] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.74, 6.62, 6.57 [00:17:13] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [00:17:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw131 mw133 mw134 [00:17:25] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.327 second response time [00:17:43] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.642 second response time [00:17:53] RECOVERY - mw133 MediaWiki Rendering on mw133 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.803 second response time [00:17:56] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:17:56] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:18:14] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.583 second response time [00:18:20] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Sat 30 Dec 2023 06:22:18 GMT +0000. [00:19:02] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.35, 3.69, 2.95 [00:19:13] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:19:21] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:22:53] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query qadrishattari.xyz. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:32:47] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:35:29] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.58, 3.43, 2.90 [00:39:28] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.13, 3.45, 3.04 [00:39:53] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [00:40:55] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:42:58] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.27, 3.80, 3.84 [00:43:28] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.83, 3.38, 3.13 [00:45:01] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.12, 3.96, 3.89 [00:50:20] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wizardia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:51:00] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.40, 3.67, 3.79 [00:52:54] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp24.miraheze.org - CNAME OK [00:55:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.74, 6.91, 6.38 [00:57:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.12, 6.71, 6.38 [00:57:26] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.35, 3.89, 3.38 [00:59:25] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 2.94, 3.41, 3.26 [01:00:42] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:01:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.22, 7.00, 6.63 [01:02:47] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 11 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [01:03:08] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.01, 3.59, 3.58 [01:03:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.29, 6.37, 6.44 [01:03:24] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.63, 3.91, 3.47 [01:05:08] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.95, 3.21, 3.44 [01:05:24] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.45, 3.58, 3.40 [01:06:59] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [01:07:07] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.68, 3.02, 3.34 [01:09:24] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.03, 3.26, 3.31 [01:10:58] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.03, 3.35, 3.43 [01:12:59] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.92, 2.77, 3.20 [01:37:54] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchise.franchising.org.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:38:29] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:55:42] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:07:54] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp25.miraheze.org - CNAME OK [02:08:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.84, 7.63, 6.71 [02:10:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.54, 7.23, 6.68 [02:10:12] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.55, 3.70, 2.91 [02:10:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.25, 9.56, 7.84 [02:12:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.40, 7.94, 7.00 [02:12:05] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [02:12:11] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.63, 3.18, 2.81 [02:12:31] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.67, 7.15, 6.19 [02:14:43] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 14.26, 11.70, 9.06 [02:15:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 7.88, 7.23 [02:16:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.42, 11.31, 9.26 [02:18:17] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.61, 6.61, 6.29 [02:18:43] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.29, 9.71, 8.92 [02:27:37] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.61, 6.11, 6.66 [02:32:47] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:33:55] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.15, 6.70, 6.27 [02:35:55] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.11, 5.72, 5.96 [02:46:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.84, 6.67, 6.38 [02:48:01] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.85, 6.20, 6.23 [03:01:37] PROBLEM - mon141 Backups Grafana on mon141 is CRITICAL: FILE_AGE CRITICAL: /var/log/grafana-backup.log is 1209675 seconds old and 93 bytes [03:08:05] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:28:57] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:41:19] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 14.24, 11.57, 9.95 [03:41:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.76, 6.89, 6.29 [03:43:13] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.04, 11.04, 9.96 [03:43:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.45, 6.55, 6.25 [03:48:58] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.83, 9.24, 9.59 [03:52:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4bb6cf2a4d64...375298e5d53e [03:52:55] [02miraheze/puppet] 07paladox 03375298e - Revert "jobrunner: don't run parsoidCachePrewarm on main" [03:53:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/375298e5d53e...fed76519e475 [03:53:30] [02miraheze/puppet] 07paladox 03fed7651 - Lower parsoidCachePrewarm memory usage to 500MB [03:55:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fed76519e475...6fca8bb1e3c8 [03:55:47] [02miraheze/puppet] 07paladox 036fca8bb - Revert "Lower parsoidCachePrewarm memory usage to 500MB" [03:58:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6fca8bb1e3c8...f02dbea3a600 [03:58:56] [02miraheze/puppet] 07paladox 03f02dbea - mediawiki: set vm.swappiness to 1 temporarily [03:58:57] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [04:08:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.12, 10.97, 9.98 [04:10:43] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.33, 11.18, 10.16 [04:12:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.84, 10.78, 10.15 [04:15:45] PROBLEM - mw131 Puppet on mw131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [04:19:45] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:20:43] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 1.54, 0.63, 0.24 [04:21:13] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw131 [04:21:23] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 20.52, 12.89, 9.12 [04:22:29] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 20.41, 13.37, 10.34 [04:23:13] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [04:23:23] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.04, 11.57, 9.10 [04:24:29] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 1.85, 0.48, 0.17 [04:27:23] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.92, 8.84, 8.59 [04:32:47] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 11 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [05:02:47] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:28:57] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [05:58:58] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:05:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.23, 3.82, 3.21 [06:07:51] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.32, 3.23, 3.06 [06:12:05] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [06:22:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 7.21, 6.37 [06:23:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.59, 3.56, 3.09 [06:26:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.22, 6.80, 6.39 [06:27:35] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.17, 3.50, 3.19 [06:29:33] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.22, 3.01, 3.05 [06:44:20] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.30, 3.52, 3.21 [06:46:18] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.42, 3.38, 3.19 [06:48:17] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.10, 3.19, 3.14 [06:58:57] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:08:05] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.49, 3.16, 3.06 [07:20:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.29, 6.72, 6.27 [07:20:53] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.12, 3.14, 3.07 [07:22:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.52, 6.79, 6.35 [07:26:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.39, 6.90, 6.52 [07:28:18] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.98, 6.58, 6.44 [07:28:57] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [07:33:46] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.53, 3.58, 3.31 [07:35:45] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.84, 3.32, 3.25 [08:05:20] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.80, 3.33, 3.11 [08:09:17] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.10, 3.36, 3.18 [08:11:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.73, 7.01, 6.32 [08:13:55] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.10, 6.34, 6.16 [08:17:09] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.17, 3.86, 3.41 [08:19:07] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.26, 3.57, 3.35 [08:23:04] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.17, 3.39, 3.34 [08:33:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.95, 3.56, 3.46 [08:35:51] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.05, 3.60, 3.47 [08:37:12] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.45, 3.10, 2.48 [08:37:50] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.05, 3.08, 3.30 [08:39:12] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.05, 2.80, 2.46 [08:52:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 7.09, 6.62 [08:54:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.82, 6.75, 6.55 [08:58:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.98, 6.93, 6.68 [09:00:31] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.15, 6.29, 6.47 [09:04:06] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [09:32:47] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 11 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [09:33:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.69, 7.20, 6.67 [09:36:06] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:37:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.67, 6.54, 6.52 [09:41:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 6.78, 6.62 [09:45:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.34, 6.66, 6.62 [09:58:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.44, 3.15, 3.02 [10:00:53] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.79, 3.06, 3.00 [10:02:47] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:08:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.83, 3.42, 3.23 [10:10:53] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.00, 3.21, 3.17 [10:20:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.63, 3.41, 3.30 [10:22:53] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.60, 3.18, 3.23 [10:28:57] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [10:47:56] PROBLEM - nijigen.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nijigen.wiki' expires in 15 day(s) (Mon 25 Dec 2023 10:26:23 GMT +0000). [10:48:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bf7f1022f1fc...ae3647a8db7c [10:48:11] [02miraheze/ssl] 07MirahezeSSLBot 03ae3647a - Bot: Update SSL cert for nijigen.wiki [11:02:03] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 13.03, 10.55, 8.71 [11:02:57] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.85, 3.15, 2.09 [11:04:03] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.02, 11.01, 9.13 [11:04:28] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.56, 3.28, 3.06 [11:04:57] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.85, 2.68, 2.04 [11:06:02] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.73, 9.44, 8.79 [11:06:27] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.21, 3.24, 3.07 [11:11:51] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.53, 3.98, 2.65 [11:12:02] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.91, 5.17, 3.42 [11:13:50] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 2.66, 3.45, 2.62 [11:14:02] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.15, 5.07, 3.60 [11:15:50] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 1.58, 2.81, 2.48 [11:17:23] RECOVERY - nijigen.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nijigen.wiki' will expire on Fri 08 Mar 2024 09:48:02 GMT +0000. [11:28:57] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [11:34:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.01, 6.64, 6.30 [11:36:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.94, 7.09, 6.49 [11:38:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 6.93, 6.49 [11:39:02] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.42, 3.24, 2.96 [11:40:36] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.03, 6.65, 6.44 [11:41:00] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.62, 3.03, 2.92 [11:53:05] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [12:04:07] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [12:10:55] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 11 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [12:22:44] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [12:24:32] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.49, 3.82, 3.49 [12:30:30] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.47, 4.16, 3.70 [12:30:35] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 15.29, 11.38, 9.32 [12:32:34] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 8.88, 10.54, 9.27 [12:34:28] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.28, 3.86, 3.70 [12:34:34] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 10.18, 10.13, 9.26 [12:36:06] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:38:27] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.30, 3.90, 3.73 [12:41:05] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 14.82, 12.36, 11.33 [12:42:25] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.44, 3.92, 3.81 [12:43:02] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 12.59, 12.42, 11.48 [12:46:00] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 15.31, 14.14, 12.65 [12:49:50] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 10.05, 12.32, 12.29 [12:52:21] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.29, 2.94, 3.38 [12:56:04] PROBLEM - tintinpedia.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tintinpedia.fr' expires in 15 day(s) (Mon 25 Dec 2023 12:43:22 GMT +0000). [12:56:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ae3647a8db7c...e9c0b882093d [12:56:20] [02miraheze/ssl] 07MirahezeSSLBot 03e9c0b88 - Bot: Update SSL cert for tintinpedia.fr [13:08:35] PROBLEM - www.tintinpedia.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tintinpedia.fr' expires in 15 day(s) (Mon 25 Dec 2023 12:43:22 GMT +0000). [13:10:55] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:18:42] PROBLEM - www.burnout.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'burnout.wiki' expires in 15 day(s) (Mon 25 Dec 2023 13:13:20 GMT +0000). [13:19:33] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.000212430954 secs [13:20:48] PROBLEM - burnout.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'burnout.wiki' expires in 15 day(s) (Mon 25 Dec 2023 13:13:20 GMT +0000). [13:21:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e9c0b882093d...2992c354bee6 [13:21:06] [02miraheze/ssl] 07MirahezeSSLBot 032992c35 - Bot: Update SSL cert for burnout.wiki [13:24:38] RECOVERY - tintinpedia.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'tintinpedia.fr' will expire on Fri 08 Mar 2024 11:56:10 GMT +0000. [13:26:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.58, 8.09, 7.10 [13:27:26] PROBLEM - wiki.hspdev.0t0.jp - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.hspdev.0t0.jp' expires in 15 day(s) (Mon 25 Dec 2023 13:02:30 GMT +0000). [13:27:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2992c354bee6...a4e7dd75f908 [13:27:51] [02miraheze/ssl] 07MirahezeSSLBot 03a4e7dd7 - Bot: Update SSL cert for wiki.hspdev.0t0.jp [13:28:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.27, 7.47, 6.99 [13:30:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.10, 6.77, 6.80 [13:30:31] PROBLEM - wiki.corgicam.tv - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.corgicam.tv' expires in 15 day(s) (Mon 25 Dec 2023 13:06:52 GMT +0000). [13:30:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a4e7dd75f908...01d62ca849e9 [13:30:47] [02miraheze/ssl] 07MirahezeSSLBot 0301d62ca - Bot: Update SSL cert for wiki.corgicam.tv [13:36:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.62, 6.87, 6.84 [13:37:41] RECOVERY - www.tintinpedia.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'tintinpedia.fr' will expire on Fri 08 Mar 2024 11:56:10 GMT +0000. [13:38:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.72, 6.61, 6.75 [13:40:55] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 11 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [13:41:00] PROBLEM - kingdomdeath.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kingdomdeath.wiki' expires in 15 day(s) (Mon 25 Dec 2023 13:19:28 GMT +0000). [13:41:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/01d62ca849e9...089ef38f3731 [13:41:18] [02miraheze/ssl] 07MirahezeSSLBot 03089ef38 - Bot: Update SSL cert for kingdomdeath.wiki [13:42:33] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 15.74, 13.58, 12.03 [13:42:54] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.27, 4.03, 3.27 [13:43:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.41, 7.03 [13:44:33] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 13.04, 13.17, 12.06 [13:44:57] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 13.93, 13.09, 12.21 [13:45:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.41, 6.61, 6.78 [13:46:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.85, 3.69, 3.33 [13:47:25] RECOVERY - www.burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Fri 08 Mar 2024 12:20:56 GMT +0000. [13:48:46] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 11.18, 12.31, 12.10 [13:50:06] RECOVERY - burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Fri 08 Mar 2024 12:20:56 GMT +0000. [13:50:53] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.75, 3.25, 3.24 [13:59:03] PROBLEM - wiki.corgicam.tv - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.corgicam.tv and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:02:47] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 11 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [14:02:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.77, 3.42, 3.27 [14:05:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 7.08, 6.76 [14:06:53] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.86, 3.34, 3.27 [14:07:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.21, 6.30, 6.52 [14:09:31] RECOVERY - kingdomdeath.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kingdomdeath.wiki' will expire on Fri 08 Mar 2024 12:41:07 GMT +0000. [14:10:55] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:27:20] RECOVERY - wiki.hspdev.0t0.jp - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hspdev.0t0.jp' will expire on Fri 08 Mar 2024 12:27:42 GMT +0000. [14:28:35] RECOVERY - wiki.corgicam.tv - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.corgicam.tv' will expire on Fri 08 Mar 2024 12:30:37 GMT +0000. [14:29:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.57, 6.82, 6.57 [14:31:37] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.07, 6.53, 6.49 [14:32:47] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:34:33] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [15:06:06] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:21:47] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 19.21, 14.71, 12.96 [15:21:51] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 13.83, 11.73, 10.11 [15:22:50] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 21.39, 16.19, 14.02 [15:23:31] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 20.00, 15.89, 13.31 [15:23:48] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 11.08, 9.25, 7.70 [15:24:16] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 16.18, 13.81, 11.72 [15:25:42] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 14.02, 15.32, 13.67 [15:25:49] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.08, 3.52, 3.18 [15:25:50] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.07, 11.54, 10.47 [15:26:13] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 16.00, 14.69, 12.31 [15:26:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 6.73, 6.46 [15:27:39] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 16.06, 15.76, 14.03 [15:27:48] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.14, 3.37, 3.17 [15:27:50] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.58, 12.10, 10.81 [15:28:11] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 17.60, 15.89, 13.03 [15:28:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.87, 6.49, 6.41 [15:29:28] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 7.02, 13.76, 13.48 [15:29:36] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.31, 12.62, 13.15 [15:29:48] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 9.53, 10.06, 8.69 [15:29:49] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 9.02, 10.91, 10.55 [15:30:09] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 7.20, 12.51, 12.15 [15:30:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 8.89, 12.84, 13.68 [15:31:27] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 13.31, 13.31, 13.31 [15:31:49] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 14.64, 12.16, 11.03 [15:32:43] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 19.28, 15.13, 14.40 [15:33:43] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.38, 3.39, 3.14 [15:35:24] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 16.29, 15.35, 14.15 [15:35:42] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.06, 3.28, 3.14 [15:36:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.73, 6.92, 6.55 [15:37:27] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.12, 3.33, 2.55 [15:37:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 20.11, 16.32, 13.67 [15:38:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.07, 6.60, 6.47 [15:38:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 20.96, 16.56, 14.59 [15:38:37] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 18.36, 13.26, 10.37 [15:38:52] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 15.44, 11.64, 9.23 [15:39:26] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.44, 3.34, 2.65 [15:40:52] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [15:41:25] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.78, 3.76, 2.88 [15:41:52] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 12.74, 15.58, 14.08 [15:42:27] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw141 mw143 [15:43:22] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.70, 3.88, 3.04 [15:43:50] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 17.28, 16.68, 14.69 [15:44:52] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [15:45:19] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.89, 3.20, 2.89 [15:46:16] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [15:46:53] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 9.99, 11.98, 10.65 [15:47:34] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.06, 3.68, 3.37 [15:49:33] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.84, 3.64, 3.39 [15:49:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 7.35, 6.81 [15:50:02] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 9.71, 11.96, 11.68 [15:51:02] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 14.03, 15.75, 15.80 [15:51:33] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.13, 3.40, 3.33 [15:51:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:51:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.91, 7.85, 7.05 [15:51:40] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 9.88, 14.50, 14.79 [15:52:14] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 4.49, 13.02, 15.03 [15:52:53] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 8.29, 9.68, 10.08 [15:53:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.71, 7.34, 6.96 [15:53:55] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.44, 6.77, 5.92 [15:55:28] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.17, 3.48, 3.39 [15:55:48] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 7.27, 8.49, 10.17 [15:55:55] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.06, 6.55, 5.93 [15:57:33] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 9.74, 11.08, 13.12 [15:57:55] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.38, 6.98, 6.14 [15:58:50] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 17.21, 15.52, 15.40 [15:59:26] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.19, 3.34, 3.35 [15:59:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.45, 6.39, 6.03 [16:00:47] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 15.98, 15.37, 15.35 [16:01:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.49, 7.64, 7.13 [16:01:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:02:09] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 17.24, 15.54, 15.16 [16:02:44] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 16.95, 15.54, 15.39 [16:03:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.60, 7.31, 7.08 [16:03:48] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 10.36, 9.80, 10.07 [16:03:55] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.06, 7.10, 6.41 [16:04:08] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 13.94, 15.07, 15.05 [16:04:42] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 15.59, 15.34, 15.33 [16:05:20] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.99, 3.64, 3.44 [16:05:55] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.31, 7.20, 6.50 [16:07:18] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.61, 4.17, 3.65 [16:07:36] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 8.40, 10.96, 11.99 [16:07:48] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 9.22, 9.78, 10.03 [16:07:55] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.43, 6.80, 6.44 [16:09:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.34, 8.22, 7.43 [16:11:15] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.78, 3.57, 3.54 [16:11:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.90, 7.67, 7.33 [16:13:14] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.55, 3.15, 3.38 [16:15:34] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 15.61, 11.97, 11.70 [16:16:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:16:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 15.50, 14.98, 15.97 [16:17:33] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 7.63, 10.36, 11.15 [16:17:33] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1013811529 secs [16:19:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.73, 7.36, 7.18 [16:21:12] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.42, 3.36, 2.74 [16:21:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.93, 6.75, 6.98 [16:23:12] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.39, 3.09, 2.72 [16:23:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.60, 6.32, 6.79 [16:23:31] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 8.10, 8.79, 10.15 [16:23:57] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 10.51, 11.90, 13.39 [16:26:33] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 11.10, 12.05, 13.55 [16:34:54] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [16:38:43] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 16.05, 14.33, 14.25 [16:40:23] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.45, 10.95, 10.35 [16:40:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 14.09, 14.35, 14.28 [16:42:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.30, 7.34, 6.81 [16:44:22] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.31, 10.77, 10.37 [16:46:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 7.33, 6.98 [16:46:21] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.92, 11.18, 10.58 [16:46:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:48:21] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 7.78, 9.91, 10.19 [16:48:53] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.81, 3.73, 3.24 [16:49:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.10, 7.32, 7.02 [16:50:53] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.52, 3.30, 3.14 [16:51:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.39, 6.55, 6.77 [16:54:43] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 11.80, 12.84, 13.58 [17:06:24] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [17:32:32] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/0504e7f904fb [17:32:35] [02miraheze/ssl] 07MacFan4000 030504e7f - Delete certificates/zenith.nsmbu.net.crt [17:32:38] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [17:32:39] [02ssl] 07MacFan4000 opened pull request 03#740: Delete certificates/zenith.nsmbu.net.crt - 13https://github.com/miraheze/ssl/pull/740 [17:33:11] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0504e7f904fb...b6ac4920045f [17:33:14] [02miraheze/ssl] 07MacFan4000 03b6ac492 - Update certs.yaml [17:33:17] [02ssl] 07MacFan4000 synchronize pull request 03#740: Delete certificates/zenith.nsmbu.net.crt - 13https://github.com/miraheze/ssl/pull/740 [17:34:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 13.90, 13.27, 12.45 [17:36:43] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 13.35, 13.47, 12.63 [17:37:08] [02ssl] 07MacFan4000 closed pull request 03#740: Delete certificates/zenith.nsmbu.net.crt - 13https://github.com/miraheze/ssl/pull/740 [17:37:11] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/089ef38f3731...c9a7c3a71632 [17:37:14] [02miraheze/ssl] 07MacFan4000 03c9a7c3a - Delete certificates/zenith.nsmbu.net.crt (#740) [17:37:16] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [17:37:18] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [17:42:43] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 13.77, 13.48, 12.85 [17:42:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.31, 6.84, 6.61 [17:44:43] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 12.54, 13.09, 12.78 [17:44:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/c9a7c3a71632...c4ce9943d9f9 [17:44:50] [02miraheze/ssl] 07MirahezeSSLBot 03c4ce994 - Bot: Add SSL cert for wiki.pixandbit.gay [17:44:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.00, 6.66, 6.57 [17:47:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/c4ce9943d9f9...9c1ebfa56f40 [17:47:41] [02miraheze/ssl] 07MirahezeSSLBot 039c1ebfa - Bot: Add SSL cert for archives.nsgov.org [17:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.22, 7.83, 6.98 [17:52:32] [02dns] 07MacFan4000 opened pull request 03#450: add 2 zones - 13https://github.com/miraheze/dns/pull/450 [17:52:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.21, 7.30, 6.88 [17:52:42] paladox: can you please merge ^^ [17:56:35] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.97, 6.62, 6.70 [17:56:50] [02dns] 07paladox closed pull request 03#450: add 2 zones - 13https://github.com/miraheze/dns/pull/450 [17:56:51] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+2/-0/±0] 13https://github.com/miraheze/dns/compare/3bc86cb431a4...1253035f066a [17:56:52] [02miraheze/dns] 07MacFan4000 031253035 - add 2 zones (#450) [18:00:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 7.03, 6.85 [18:02:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.75, 6.64, 6.73 [18:05:35] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [18:06:26] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=morethanbutterflieswiki --no-updates --username-prefix=wikia:mc-butterfly-mod /home/macfan/ImportDump/morethanbutterflieswiki-20231201005109.xml (START) [18:06:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:54] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=morethanbutterflieswiki --no-updates --username-prefix=wikia:mc-butterfly-mod /home/macfan/ImportDump/morethanbutterflieswiki-20231201005109.xml (END - exit=0) [18:11:55] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=morethanbutterflieswiki (START) [18:12:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:13:34] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=morethanbutterflieswiki (END - exit=0) [18:13:35] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=morethanbutterflieswiki --active --update (END - exit=0) [18:13:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:13:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:17:51] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=theredlakewiki --no-updates --username-prefix=wikia:scp-the-red-lake /home/macfan/ImportDump/theredlakewiki-20231201194432.xml (START) [18:18:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.11, 6.97, 6.75 [18:24:53] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=theredlakewiki --no-updates --username-prefix=wikia:scp-the-red-lake /home/macfan/ImportDump/theredlakewiki-20231201194432.xml (END - exit=0) [18:24:54] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=theredlakewiki (START) [18:24:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/9c1ebfa56f40...de98f57ef513 [18:28:11] [02miraheze/ssl] 07MirahezeSSLBot 03de98f57 - Bot: Add SSL cert for fashiondreamer.wiki [18:29:50] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=theredlakewiki (END - exit=0) [18:29:51] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=theredlakewiki --active --update (END - exit=0) [18:29:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.88, 6.71, 6.79 [18:34:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.86, 6.79, 6.86 [18:38:45] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [18:40:39] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.00, 6.57, 6.75 [18:44:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.23, 6.86, 6.87 [18:46:31] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.59, 6.30, 6.66 [18:47:38] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [18:48:05] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw133 mw134 [18:48:17] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw134 [18:49:38] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [18:50:02] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [18:50:13] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [18:50:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.08, 7.67, 7.13 [18:52:02] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.38, 7.77, 3.57 [18:52:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.28, 7.15, 7.01 [18:53:57] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=mypediawiki --no-updates /home/macfan/ImportDump/mypediawiki-20231205161205.xml (START) [18:54:02] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 10.02, 8.54, 4.37 [18:54:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:12] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=mypediawiki --no-updates /home/macfan/ImportDump/mypediawiki-20231205161205.xml (END - exit=0) [18:56:13] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=mypediawiki (START) [18:56:18] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=mypediawiki (END - exit=256) [18:56:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:19] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=mypediawiki --active --update (END - exit=0) [18:56:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:54] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.40, 10.09, 5.93 [19:01:46] PROBLEM - mwtask141 APT on mwtask141 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:01:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.84, 6.48, 6.78 [19:03:42] RECOVERY - mwtask141 APT on mwtask141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:03:47] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.08, 9.32, 7.05 [19:06:50] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:07:31] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:38] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchising.org.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:08:14] PROBLEM - mwtask141 HTTPS on mwtask141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mwtask141.miraheze.org port 443 after 3069 ms: Couldn't connect to server [19:08:17] PROBLEM - mwtask141 mathoid on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 and port 10044: No route to host [19:08:35] RECOVERY - mw131 APT on mw131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:09:30] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.282 second response time [19:10:07] RECOVERY - mwtask141 HTTPS on mwtask141 is OK: HTTP OK: HTTP/2 301 - 353 bytes in 0.011 second response time [19:10:13] !log upgrade mwtask141 to bookworm [19:10:16] RECOVERY - mwtask141 mathoid on mwtask141 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:504 port 10044 [19:10:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:10:24] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:12:19] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:12:46] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw131.miraheze.org port 443 after 0 ms: Couldn't connect to server [19:13:14] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw131 [19:13:21] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw131 [19:13:36] PROBLEM - mwtask141 APT on mwtask141 is CRITICAL: APT CRITICAL: 306 packages available for upgrade (4 critical updates). [19:13:37] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw131 [19:14:28] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw131 [19:14:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.35, 6.85, 6.64 [19:16:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.13, 7.22, 6.78 [19:16:43] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.024 second response time [19:17:05] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [19:17:19] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [19:17:30] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [19:17:31] RECOVERY - mwtask141 APT on mwtask141 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:18:24] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [19:18:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.39, 7.23, 6.83 [19:19:00] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [19:23:31] PROBLEM - mwtask141 APT on mwtask141 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:24:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.59, 6.38, 6.62 [19:27:50] PROBLEM - mwtask141 APT on mwtask141 is CRITICAL: APT CRITICAL: 221 packages available for upgrade (12 critical updates). [19:29:08] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.212 second response time [19:29:51] PROBLEM - mwtask141 APT on mwtask141 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:30:19] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 10.50, 9.33, 8.24 [19:30:40] PROBLEM - mwtask141 php-fpm on mwtask141 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4' [19:32:00] PROBLEM - mwtask141 APT on mwtask141 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (3 critical updates). [19:32:14] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 10.14, 9.75, 8.54 [19:33:50] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=cogdiswiki --no-updates --username-prefix=wikia:cogdis /home/macfan/ImportDump/cogdiswiki-20231207214638.xml (START) [19:33:51] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=cogdiswiki --no-updates --username-prefix=wikia:cogdis /home/macfan/ImportDump/cogdiswiki-20231207214638.xml (END - exit=256) [19:33:52] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=cogdiswiki (START) [19:33:53] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=cogdiswiki (END - exit=256) [19:33:54] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=cogdiswiki --active --update (END - exit=256) [19:33:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:57] RECOVERY - mwtask141 APT on mwtask141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:34:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:55] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=woriburiwiki --no-updates --username-prefix=dev /home/macfan/ImportDump/woriburiwiki-20231207231541.xml (START) [19:35:56] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=woriburiwiki --no-updates --username-prefix=dev /home/macfan/ImportDump/woriburiwiki-20231207231541.xml (END - exit=256) [19:35:57] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=woriburiwiki (START) [19:35:58] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=woriburiwiki (END - exit=256) [19:35:59] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=woriburiwiki --active --update (END - exit=256) [19:36:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:21] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp24.miraheze.org - CNAME OK [19:36:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:43] MacFan4000: I'm upgrading mwtask [19:36:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:11] I'll be rebooting in just a minute as it's almost finished [19:37:21] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.09, 11.55, 10.21 [19:37:28] ok, I'll pause for a bit [19:38:35] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.23, 3.42, 2.91 [19:39:21] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.67, 11.45, 10.36 [19:43:22] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.50, 12.04, 10.80 [19:43:36] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 50 failures. Last run 3 minutes ago with 50 failures. Failed resources (up to 3 shown): Package[iotop],Package[lvm2],Package[parted],Package[ruby] [19:44:00] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.66, 9.93, 9.26 [19:44:27] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.09, 3.10, 2.99 [19:45:21] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.56, 11.37, 10.70 [19:45:52] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/56313dbe3c87 [19:45:55] [02miraheze/ssl] 07MacFan4000 0356313db - Delete certificates/archives.rezehno.org.crt [19:45:58] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [19:46:25] [02ssl] 07MacFan4000 opened pull request 03#741: Delete certificates/archives.rezehno.org.crt - 13https://github.com/miraheze/ssl/pull/741 [19:46:57] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/de98f57ef513...4112aa9dda97 [19:46:58] [02miraheze/ssl] 07MacFan4000 034112aa9 - Update certs.yaml [19:47:08] [02ssl] 07MacFan4000 closed pull request 03#741: Delete certificates/archives.rezehno.org.crt - 13https://github.com/miraheze/ssl/pull/741 [19:47:10] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [19:47:11] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/4112aa9dda97...4299e7659a0c [19:47:14] [02miraheze/ssl] 07MacFan4000 034299e76 - Delete certificates/archives.rezehno.org.crt (#741) [19:47:15] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [19:47:58] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.47, 8.79, 9.01 [19:48:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f02dbea3a600...08ef2afb01cc [19:48:22] [02miraheze/puppet] 07paladox 0308ef2af - mwtask141: upgrade to php 8.2 [19:49:00] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [19:52:58] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.670 second response time [19:53:00] RECOVERY - mwtask141 php-fpm on mwtask141 is OK: PROCS OK: 7 processes with command name 'php-fpm8.2' [19:55:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.73, 8.27, 6.95 [19:55:31] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.87, 11.93, 10.95 [19:56:18] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 10.37, 9.28, 8.74 [19:56:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/979a4cab27d5 [19:56:40] [02miraheze/puppet] 07paladox 03979a4ca - mediawiki: fix running vendor_psysh_composer [19:56:43] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [19:56:44] [02puppet] 07paladox opened pull request 03#3557: mediawiki: fix running vendor_psysh_composer - 13https://github.com/miraheze/puppet/pull/3557 [19:56:47] [02puppet] 07paladox closed pull request 03#3557: mediawiki: fix running vendor_psysh_composer - 13https://github.com/miraheze/puppet/pull/3557 [19:56:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/08ef2afb01cc...7734a43f7c6c [19:56:53] [02miraheze/puppet] 07paladox 037734a43 - mediawiki: fix running vendor_psysh_composer (#3557) [19:56:55] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [19:56:57] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [19:57:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 7.69, 6.89 [19:57:44] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.35, 7.04, 6.29 [19:59:13] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:01:42] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.11, 6.32, 6.20 [20:02:06] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 9.80, 9.93, 9.19 [20:06:19] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [20:07:11] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.54, 11.28, 11.34 [20:10:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.94, 6.73, 6.77 [20:14:45] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 11.47, 10.46, 9.76 [20:14:57] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 14.52, 12.25, 11.63 [20:15:59] MacFan4000: finished [20:16:41] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 9.64, 9.88, 9.62 [20:16:54] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.60, 11.64, 11.49 [20:20:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.90, 7.45, 6.94 [20:22:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.74, 6.78, 6.75 [20:25:13] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=jokeswiki --no-updates /home/macfan/ImportDump/jokeswiki-20231207234228.xml (START) [20:25:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:28:34] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.78, 8.98, 10.18 [20:30:29] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 11 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [20:38:58] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=jokeswiki --no-updates /home/macfan/ImportDump/jokeswiki-20231207234228.xml (END - exit=0) [20:38:59] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=jokeswiki (START) [20:39:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:39:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7734a43f7c6c...3079f2c6b622 [20:45:00] [02miraheze/puppet] 07paladox 033079f2c - mw131: upgrade to php 8.2 [20:46:09] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.36, 10.42, 10.02 [20:46:24] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 16.82, 11.89, 9.00 [20:47:33] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.047 second response time [20:47:43] !log upgrade mw131 to bookworm [20:47:45] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=jokeswiki (END - exit=0) [20:47:46] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=jokeswiki --active --update (END - exit=0) [20:47:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:57] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw131 [20:48:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.09, 7.11, 6.75 [20:48:05] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.99, 11.75, 10.56 [20:48:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:13] PROBLEM - mw131 php-fpm on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4' [20:48:49] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw131 [20:48:55] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw131 [20:49:04] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw131 [20:49:56] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 12.69, 9.74, 7.30 [20:49:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.30, 6.60, 6.59 [20:50:02] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 7.96, 11.02, 10.48 [20:50:34] PROBLEM - mw131 APT on mw131 is CRITICAL: APT CRITICAL: 354 packages available for upgrade (5 critical updates). [20:51:09] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 18.40, 12.00, 8.43 [20:51:50] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 16.46, 12.72, 10.25 [20:51:58] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 2.79, 8.08, 9.46 [20:52:04] PROBLEM - mw131 JobRunner Service on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [20:52:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.17, 6.27, 5.24 [20:52:23] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.96, 7.21, 6.29 [20:53:14] RECOVERY - mw132 APT on mw132 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:53:25] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=modernphenotypeswiki --no-updates --username-prefix=wikia:modernphenotypes /home/macfan/ImportDump/modernphenotypeswiki-20231208160613.xml (START) [20:53:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.47, 7.21, 6.92 [20:54:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:54:05] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.68, 5.82, 5.19 [20:54:23] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.73, 6.34, 6.10 [20:54:30] RECOVERY - mw131 APT on mw131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:55:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.49, 6.78, 6.80 [21:01:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.56, 6.91, 6.87 [21:05:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.81, 6.50, 6.73 [21:06:38] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:08:38] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: connect to address 2a10:6740::6:404 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [21:09:16] PROBLEM - mw132 NTP time on mw132 is CRITICAL: NTP CRITICAL: Offset -0.5570608675 secs [21:09:40] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw132.miraheze.org port 443 after 0 ms: Couldn't connect to server [21:09:50] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 11.30, 11.48, 11.95 [21:10:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3079f2c6b622...6a581ccac088 [21:10:15] [02miraheze/puppet] 07paladox 036a581cc - mcrouter: install missing packages on bookworm [21:10:32] PROBLEM - mw131 APT on mw131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [21:11:19] RECOVERY - mw132 NTP time on mw132 is OK: NTP OK: Offset -0.001611560583 secs [21:11:35] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.014 second response time [21:12:02] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Puppet has 55 failures. Last run 1 minute ago with 55 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72],Package[php8.2-common],Package[php8.2-opcache],Package[php8.2-cli] [21:12:13] RECOVERY - mw131 php-fpm on mw131 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [21:12:34] PROBLEM - mw131 APT on mw131 is CRITICAL: APT CRITICAL: 402 packages available for upgrade (21 critical updates). [21:12:38] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.137 second response time [21:13:34] PROBLEM - mw131 MediaWiki Rendering on mw131 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [21:13:51] !log upgrade mw132 to bookworm [21:13:57] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [21:13:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:59] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [21:14:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6a581ccac088...e46464bbaf07 [21:14:10] [02miraheze/puppet] 07paladox 03e46464b - mw132: upgrade to php 8.2 [21:14:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [21:14:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.09, 2.90, 1.82 [21:16:32] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.016 second response time [21:16:44] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.34, 3.07, 2.01 [21:17:21] PROBLEM - mw132 APT on mw132 is CRITICAL: APT CRITICAL: 385 packages available for upgrade (5 critical updates). [21:17:23] PROBLEM - mw132 php-fpm on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4' [21:17:50] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 13.86, 11.89, 11.80 [21:17:57] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw132 [21:18:33] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw132 [21:18:40] PROBLEM - mw131 APT on mw131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [21:20:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.11, 7.17, 6.74 [21:20:39] PROBLEM - mw131 APT on mw131 is CRITICAL: APT CRITICAL: 64 packages available for upgrade (3 critical updates). [21:22:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.69, 6.43, 6.51 [21:22:37] RECOVERY - mw131 APT on mw131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:29:38] RECOVERY - mw132 APT on mw132 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:31:43] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:50] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 10.54, 11.43, 11.90 [21:33:07] PROBLEM - mw131 NTP time on mw131 is WARNING: NTP WARNING: Offset 0.1987208426 secs [21:33:33] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw131.miraheze.org port 443 after 3 ms: Couldn't connect to server [21:33:45] RECOVERY - mw131 JobRunner Service on mw131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:34:02] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown) [21:35:12] RECOVERY - mw131 NTP time on mw131 is OK: NTP OK: Offset -0.002038061619 secs [21:35:29] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.014 second response time [21:35:37] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.059 second response time [21:35:51] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [21:35:59] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:37:33] PROBLEM - mw132 APT on mw132 is CRITICAL: APT CRITICAL: 224 packages available for upgrade (11 critical updates). [21:38:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:38:30] RECOVERY - mw133 APT on mw133 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:38:39] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.51, 7.44, 3.13 [21:39:36] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 59 failures. Last run 3 minutes ago with 59 failures. Failed resources (up to 3 shown): Package[openssh-client],Package[openssh-server],Package[libboost-regex1.74.0-icu72],Package[libfmt9] [21:40:07] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 1.46, 0.31, 0.10 [21:41:35] RECOVERY - mw132 APT on mw132 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:42:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e46464bbaf07...f8284090996e [21:42:23] [02miraheze/puppet] 07paladox 03f828409 - mw133: upgrade php to 8.2 [21:42:33] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.46, 9.82, 5.08 [21:42:58] !log upgrade mw133 to bookworm [21:43:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:44:35] PROBLEM - mw133 APT on mw133 is CRITICAL: APT CRITICAL: 387 packages available for upgrade (5 critical updates). [21:46:14] PROBLEM - mw133 php-fpm on mw133 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [21:46:17] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CRITICAL: Puppet has 57 failures. Last run 2 minutes ago with 57 failures. Failed resources (up to 3 shown): Package[openssh-client],Package[openssh-server],Package[libboost-regex1.74.0-icu67],Package[php8.2-common] [21:46:26] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.96, 10.18, 6.30 [21:46:31] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.016 second response time [21:48:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:49:14] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.64, 3.28, 2.71 [21:49:24] PROBLEM - mw132 JobRunner Service on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [21:49:31] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw132.miraheze.org port 443 after 0 ms: Couldn't connect to server [21:50:21] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.87, 10.92, 7.46 [21:50:33] RECOVERY - mw133 APT on mw133 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:51:13] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.16, 2.83, 2.61 [21:51:26] RECOVERY - mw132 php-fpm on mw132 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [21:52:18] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.49, 11.70, 8.15 [21:53:24] RECOVERY - mw132 JobRunner Service on mw132 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:54:14] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.00, 11.54, 8.51 [21:55:31] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.011 second response time [21:55:54] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.361 second response time [21:56:21] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:58:41] RECOVERY - mw134 APT on mw134 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:59:51] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.28, 9.18, 4.05 [22:00:05] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.08, 11.44, 9.38 [22:00:40] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.76, 3.18, 2.16 [22:02:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.68, 11.76, 9.75 [22:02:24] PROBLEM - mw133 APT on mw133 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:03:13] PROBLEM - mw134 NTP time on mw134 is CRITICAL: connect to address 2a10:6740::6:410 port 5666: Connection refusedconnect to host 2a10:6740::6:410 port 5666: Connection refused [22:03:15] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.09, 3.97, 3.15 [22:03:34] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:36] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw134.miraheze.org port 443 after 6102 ms: Couldn't connect to server [22:04:26] !log upgrade mw134 to bookworm [22:04:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f8284090996e...3395bff8900d [22:04:34] [02miraheze/puppet] 07paladox 033395bff - mw134: upgrade php to 8.2 [22:04:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:05:11] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=modernphenotypeswiki --no-updates --username-prefix=wikia:modernphenotypes /home/macfan/ImportDump/modernphenotypeswiki-20231208160613.xml (END - exit=0) [22:05:12] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=modernphenotypeswiki (START) [22:05:28] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=modernphenotypeswiki (END - exit=256) [22:05:30] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=modernphenotypeswiki --active --update (END - exit=0) [22:05:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:05:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:24] PROBLEM - mw133 APT on mw133 is CRITICAL: APT CRITICAL: 189 packages available for upgrade (11 critical updates). [22:06:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:58] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:07:08] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:07:10] PROBLEM - mw134 PowerDNS Recursor on mw134 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:07:15] RECOVERY - mw134 NTP time on mw134 is OK: NTP OK: Offset 0.0004400908947 secs [22:07:30] RECOVERY - mw134 HTTPS on mw134 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.016 second response time [22:09:08] RECOVERY - mw134 PowerDNS Recursor on mw134 is OK: DNS OK: 3.114 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [22:09:31] PROBLEM - os131 PowerDNS Recursor on os131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:09:49] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.12, 11.83, 10.55 [22:09:55] PROBLEM - mw134 APT on mw134 is CRITICAL: APT CRITICAL: 376 packages available for upgrade (5 critical updates). [22:12:29] PROBLEM - mw133 APT on mw133 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:13:38] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.441 second response time [22:13:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.35, 11.93, 9.00 [22:14:14] PROBLEM - mw134 php-fpm on mw134 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [22:14:36] PROBLEM - mw133 APT on mw133 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (2 critical updates). [22:15:39] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.25, 12.27, 9.47 [22:15:56] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 8.40, 10.44, 11.73 [22:15:57] RECOVERY - os131 PowerDNS Recursor on os131 is OK: DNS OK: 7.224 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [22:15:59] PROBLEM - mw134 APT on mw134 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:16:35] RECOVERY - mw133 APT on mw133 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:18:00] PROBLEM - mw134 APT on mw134 is CRITICAL: APT CRITICAL: 295 packages available for upgrade (4 critical updates). [22:20:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/80d3a44d2fa0 [22:20:26] [02miraheze/puppet] 07paladox 0380d3a44 - squid: increase max open file descriptors [22:20:29] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:20:30] [02puppet] 07paladox opened pull request 03#3558: squid: increase max open file descriptors - 13https://github.com/miraheze/puppet/pull/3558 [22:20:43] [02puppet] 07paladox closed pull request 03#3558: squid: increase max open file descriptors - 13https://github.com/miraheze/puppet/pull/3558 [22:20:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3395bff8900d...721abee3b6c1 [22:20:49] [02miraheze/puppet] 07paladox 03721abee - squid: increase max open file descriptors (#3558) [22:20:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [22:20:53] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:21:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.38, 11.90, 10.30 [22:21:56] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 12.35, 10.80, 11.34 [22:22:06] RECOVERY - mw134 APT on mw134 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [22:22:25] PROBLEM - os131 PowerDNS Recursor on os131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:23:39] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 17.25, 13.42, 11.02 [22:24:24] RECOVERY - os131 PowerDNS Recursor on os131 is OK: DNS OK: 0.758 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [22:29:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.35, 11.65, 11.14 [22:31:21] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.22, 10.94, 11.64 [22:32:04] PROBLEM - mw134 APT on mw134 is CRITICAL: APT CRITICAL: 372 packages available for upgrade (16 critical updates). [22:33:19] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1046 bytes in 0.206 second response time [22:33:22] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.14, 12.06, 11.92 [22:34:03] PROBLEM - mw134 APT on mw134 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:35:40] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.48, 12.22, 11.42 [22:38:03] PROBLEM - mw134 APT on mw134 is CRITICAL: APT CRITICAL: 226 packages available for upgrade (12 critical updates). [22:38:25] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:39:58] PROBLEM - mw133 JobRunner Service on mw133 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [22:40:01] PROBLEM - mw134 APT on mw134 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [22:42:05] PROBLEM - mw134 APT on mw134 is CRITICAL: APT CRITICAL: 133 packages available for upgrade (7 critical updates). [22:42:48] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 6.194 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [22:42:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.86, 7.39, 6.54 [22:43:08] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 8.05, 10.31, 11.80 [22:43:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.88, 11.60, 11.75 [22:43:56] PROBLEM - mw143 Current Load on mw143 is WARNING: WARNING - load average: 6.40, 9.50, 11.47 [22:46:02] RECOVERY - mw133 php-fpm on mw133 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [22:46:08] RECOVERY - mw134 APT on mw134 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:48:39] RECOVERY - mw133 MediaWiki Rendering on mw133 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.932 second response time [22:48:41] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:49:08] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.94, 6.86, 9.79 [22:49:56] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 5.95, 7.21, 9.82 [22:50:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.53, 6.64, 6.50 [22:51:38] RECOVERY - mw133 JobRunner Service on mw133 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [22:51:39] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.08, 10.83, 11.25 [22:53:21] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.92, 9.95, 11.73 [22:53:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.52, 9.72, 10.81 [22:55:50] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.74, 8.42, 3.77 [22:57:41] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.34, 8.78, 10.18 [22:57:50] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.52, 8.35, 4.32 [22:58:25] RECOVERY - mw134 php-fpm on mw134 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [22:59:22] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:59:47] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 5.27, 7.89, 11.87 [23:00:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [23:00:37] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:00:49] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [23:00:52] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.92, 2.96, 3.92 [23:01:04] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [23:01:23] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.52, 7.78, 9.99 [23:01:56] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [23:02:58] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.04, 3.65, 4.05 [23:03:25] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.823 second response time [23:04:58] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.55, 3.34, 3.87 [23:05:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 6.52, 6.29 [23:05:47] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.17, 6.55, 9.96 [23:07:36] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.54, 6.58, 6.35 [23:07:39] PROBLEM - mw141 APT on mw141 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [23:07:50] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.60, 10.41, 6.86 [23:09:46] RECOVERY - mw141 APT on mw141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:09:50] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.85, 9.93, 7.11 [23:09:55] RECOVERY - mw142 APT on mw142 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:10:36] RECOVERY - mw143 APT on mw143 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [23:11:13] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.38, 3.09, 3.86 [23:11:39] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.65, 11.05, 9.87 [23:12:48] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.81, 2.45, 3.27 [23:13:13] !log upgrade mw142 to bookworm [23:13:16] !log upgrade mw143 to bookworm [23:13:17] !log upgrade mw141 to bookworm [23:13:21] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 19.46, 13.10, 10.85 [23:13:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:50] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 19.29, 14.35, 9.48 [23:13:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:14:08] PROBLEM - mw142 APT on mw142 is CRITICAL: APT CRITICAL: 355 packages available for upgrade (5 critical updates). [23:14:44] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 18.63, 13.22, 6.96 [23:14:46] PROBLEM - mw143 APT on mw143 is CRITICAL: APT CRITICAL: 335 packages available for upgrade (5 critical updates). [23:15:11] PROBLEM - mw131 JobRunner Service on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [23:15:24] PROBLEM - mw132 JobRunner Service on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [23:15:38] PROBLEM - mw133 JobRunner Service on mw133 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [23:16:17] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: connect to address 2a10:6740::6:502 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [23:17:13] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.25, 3.21, 3.61 [23:17:15] PROBLEM - mw141 Puppet on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [23:17:18] PROBLEM - mw141 SSH on mw141 is CRITICAL: connect to address 2a10:6740::6:502 and port 22: No route to host [23:17:52] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [23:17:57] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw141 [23:18:02] PROBLEM - mw141 APT on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [23:18:08] PROBLEM - mw141 php-fpm on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [23:18:08] PROBLEM - mw141 ferm_active on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [23:18:08] PROBLEM - ping6 on mw141 is CRITICAL: CRITICAL - Host Unreachable (2a10:6740::6:502) [23:18:08] PROBLEM - Host mw141 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:502) [23:19:07] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [23:19:09] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw141 [23:19:13] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.95, 3.04, 3.50 [23:20:08] RECOVERY - Host mw141 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [23:20:08] RECOVERY - mw141 ferm_active on mw141 is OK: OK ferm input default policy is set [23:20:10] RECOVERY - ping6 on mw141 is OK: PING OK - Packet loss = 0%, RTA = 0.58 ms [23:20:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.68, 7.27, 6.61 [23:20:57] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw141.miraheze.org port 443 after 0 ms: Couldn't connect to server [23:21:13] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.32, 2.80, 3.36 [23:21:14] RECOVERY - mw141 SSH on mw141 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u2 (protocol 2.0) [23:21:17] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [23:21:58] RECOVERY - mw142 APT on mw142 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:22:12] RECOVERY - mw141 php-fpm on mw141 is OK: PROCS OK: 25 processes with command name 'php-fpm7.4' [23:22:12] RECOVERY - mw141 APT on mw141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:22:55] RECOVERY - mw143 APT on mw143 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:23:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/721abee3b6c1...12fa7d192e41 [23:23:56] [02miraheze/puppet] 07paladox 0312fa7d1 - mw141: upgrade php to 8.2 [23:23:57] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [23:24:13] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.718 second response time [23:24:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/12fa7d192e41...6b61d6ba66e0 [23:24:21] [02miraheze/puppet] 07paladox 036b61d6b - mw142: upgrade to php 8.2 [23:24:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6b61d6ba66e0...e751561887c2 [23:24:37] [02miraheze/puppet] 07paladox 03e751561 - mw143: upgrade to php 8.2 [23:24:54] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.017 second response time [23:24:54] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [23:24:56] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [23:25:43] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [23:27:23] PROBLEM - mw141 APT on mw141 is CRITICAL: APT CRITICAL: 733 packages available for upgrade (22 critical updates). [23:28:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.93, 6.52, 6.53 [23:29:38] RECOVERY - mw133 JobRunner Service on mw133 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [23:30:08] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1046 bytes in 0.268 second response time [23:32:31] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.52, 3.32, 2.60 [23:34:35] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [23:35:24] RECOVERY - mw132 JobRunner Service on mw132 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [23:37:18] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu67] [23:38:13] PROBLEM - mw142 APT on mw142 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [23:38:30] PROBLEM - mw141 php-fpm on mw141 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [23:41:03] PROBLEM - mw143 APT on mw143 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [23:41:11] RECOVERY - mw131 JobRunner Service on mw131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [23:42:18] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:42:20] PROBLEM - mw142 php-fpm on mw142 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [23:44:16] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.126 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [23:45:41] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 54 failures. Last run 3 minutes ago with 54 failures. Failed resources (up to 3 shown): Package[mcrouter],Package[php8.2-common],Package[php8.2-opcache],Package[php8.2-cli] [23:45:57] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Puppet has 54 failures. Last run 2 minutes ago with 54 failures. Failed resources (up to 3 shown): Package[mcrouter],Package[php8.2-common],Package[php8.2-opcache],Package[php8.2-cli] [23:46:26] PROBLEM - mw143 php-fpm on mw143 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [23:48:07] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 0.28, 2.79, 3.62 [23:50:02] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.37, 2.01, 3.24 [23:56:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1