[00:01:50] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 5 backends are down. mw132 mw142 mw133 mw134 mw143 [00:02:15] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 5 backends are down. mw131 mw132 mw133 mw134 mw143 [00:02:28] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.11, 6.64, 6.56 [00:02:32] PROBLEM - test131 Current Load on test131 is CRITICAL: LOAD CRITICAL - total load average: 4.88, 3.18, 2.14 [00:03:03] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 4 backends are down. mw131 mw132 mw133 mw134 [00:03:11] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 5 backends are down. mw131 mw132 mw142 mw133 mw143 [00:03:13] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 14.18, 11.14, 9.43 [00:03:19] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.96, 6.77, 3.17 [00:03:28] RECOVERY - graylog131 SSH on graylog131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [00:03:33] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 16.67, 11.41, 9.40 [00:04:00] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 13.40, 5.77, 3.13 [00:04:22] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:29] RECOVERY - test131 Current Load on test131 is OK: LOAD OK - total load average: 2.57, 2.94, 2.19 [00:06:26] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.603 second response time [00:06:53] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 20.13, 13.33, 10.00 [00:07:00] RECOVERY - mem131 PowerDNS Recursor on mem131 is OK: DNS OK: 5.273 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:07:55] PROBLEM - graylog131 SSH on graylog131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:08:12] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 38 minutes ago with 0 failures [00:08:47] RECOVERY - mem131 SSH on mem131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u2 (protocol 2.0) [00:09:59] RECOVERY - graylog131 SSH on graylog131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [00:10:00] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.66, 3.72, 3.08 [00:10:44] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 6.99, 11.46, 10.15 [00:11:01] PROBLEM - os141 Puppet on os141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Opensearch_template[graylog-internal] [00:11:13] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 6.85, 10.74, 10.38 [00:11:30] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 5.40, 10.49, 10.23 [00:11:30] PROBLEM - mem131 PowerDNS Recursor on mem131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:11:53] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:12:00] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.16, 2.80, 2.82 [00:12:39] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.17, 9.45, 9.58 [00:13:13] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.89, 9.53, 9.98 [00:13:29] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.33, 9.14, 9.76 [00:13:48] RECOVERY - mem131 PowerDNS Recursor on mem131 is OK: DNS OK: 4.523 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:13:50] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 2.611 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:13:50] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:13:59] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1018386781 secs [00:14:12] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:14:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:14:51] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:15:00] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:15:02] PROBLEM - mem131 Puppet on mem131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:15:19] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.19, 7.71, 6.88 [00:15:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [00:17:19] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.80, 6.05, 6.37 [00:17:34] PROBLEM - graylog131 SSH on graylog131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:31] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:19:33] RECOVERY - graylog131 SSH on graylog131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [00:22:33] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 2.697 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:23:40] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:23:53] PROBLEM - graylog131 Puppet on graylog131 is WARNING: WARNING: Puppet last ran 1 hour ago [00:26:51] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:29:31] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:30:34] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.58, 8.15, 11.34 [00:34:24] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:34:35] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.21, 6.47, 9.89 [00:34:36] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.19, 8.05, 11.34 [00:35:25] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 5.64, 8.29, 11.84 [00:36:11] PROBLEM - mem131 Current Load on mem131 is WARNING: WARNING - load average: 0.80, 1.56, 3.83 [00:36:20] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 1.011 second response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:39:01] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:39:33] PROBLEM - graylog131 SSH on graylog131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:07] RECOVERY - mem131 Current Load on mem131 is OK: OK - load average: 0.42, 0.94, 3.06 [00:41:33] RECOVERY - graylog131 SSH on graylog131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [00:44:36] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.75, 8.08, 9.98 [00:44:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.58, 10.41, 11.98 [00:46:00] PROBLEM - graylog131 SSH on graylog131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:00] RECOVERY - graylog131 SSH on graylog131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [00:49:25] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 5.82, 8.23, 9.86 [00:56:01] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:58:20] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [00:58:58] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.03, 8.31, 9.82 [01:18:09] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 21.87, 12.91, 9.25 [01:18:24] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw131 [01:18:33] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw131 [01:18:41] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw131 [01:19:05] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 16.02, 12.57, 10.60 [01:19:25] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 15.80, 12.01, 10.53 [01:20:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [01:20:40] PROBLEM - graylog131 SSH on graylog131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:17] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:22:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [01:22:39] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [01:22:56] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:22:59] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 4.40, 10.93, 10.64 [01:23:25] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.28, 11.97, 10.99 [01:24:38] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.43, 9.29, 8.45 [01:24:58] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.27, 10.80, 10.56 [01:25:25] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.42, 13.73, 11.73 [01:26:12] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [01:26:58] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.22, 11.41, 10.88 [01:26:59] RECOVERY - graylog131 SSH on graylog131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [01:27:00] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 0.714 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:27:19] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.03, 10.57, 8.31 [01:27:46] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw132 mw134 [01:28:06] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 5 backends are down. mw131 mw132 mw133 mw134 mediawiki [01:28:33] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 3 backends are down. mw131 mw132 mw133 [01:28:58] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.60, 13.37, 11.70 [01:29:14] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 2.22, 7.27, 7.37 [01:30:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [01:30:36] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.44, 11.70, 10.15 [01:30:58] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.00, 11.38, 11.21 [01:31:45] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [01:31:58] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [01:31:58] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.54, 11.35, 11.28 [01:32:37] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 5.18, 9.51, 9.54 [01:33:29] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 4.32, 9.64, 11.06 [01:34:58] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.19, 8.40, 9.99 [01:37:25] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 4.55, 6.81, 9.58 [01:37:53] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 4.07, 7.17, 9.45 [01:41:58] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 2153s [01:42:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:42:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.86, 7.25, 6.46 [01:44:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 6.95, 6.44 [01:47:04] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 0.27, 0.70, 3.67 [01:47:40] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:48:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.17, 7.40, 6.70 [01:49:03] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 0.37, 0.59, 3.28 [01:50:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 7.24, 6.71 [01:54:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.39, 6.57, 6.54 [02:17:40] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 8 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [02:19:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.93, 7.53, 6.78 [02:21:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.10, 7.23, 6.79 [02:23:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.28, 6.64, 6.63 [02:36:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 7.15, 6.84 [02:38:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.00, 6.62, 6.68 [02:47:40] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:53:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.53, 7.13, 6.95 [02:55:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.45, 6.60, 6.78 [02:59:38] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.59, 6.87, 6.12 [03:01:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.92, 7.95, 7.31 [03:01:36] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.95, 6.73, 6.17 [03:03:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 7.58, 7.25 [03:03:21] PROBLEM - db101 Backups SQL on db101 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864200 seconds old and 0 bytes [03:09:23] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.26, 6.45, 6.06 [03:11:21] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.97, 6.24, 6.03 [03:14:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.66, 6.35, 6.78 [03:18:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.77, 7.62, 7.16 [03:20:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 7.63, 7.24 [03:26:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.77, 6.37, 6.79 [03:34:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 6.78, 6.85 [03:42:12] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.12, 6.49, 6.80 [04:43:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.61, 6.64, 6.02 [04:45:18] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.39, 6.37, 5.99 [05:59:53] PROBLEM - alphabirdswiki.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'alphabirdswiki.gq' expires in 7 day(s) (Sat 30 Dec 2023 05:39:12 GMT +0000). [06:16:19] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [06:39:02] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72] [06:49:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.40, 6.78, 6.33 [06:51:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.93, 6.40, 6.24 [07:06:19] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:06:58] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:07:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.23, 10.36, 9.70 [07:15:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.33, 10.17, 9.96 [09:03:25] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:59] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 13.99, 18.62, 12.85 [09:05:30] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.643 second response time [09:08:52] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 4.70, 11.33, 11.16 [09:12:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.66, 8.86, 10.16 [10:15:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [10:46:41] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.14, 10.03, 8.27 [10:48:37] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 7.72, 9.23, 8.20 [11:15:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [11:17:59] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5006245375 secs [12:11:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.97, 10.54, 8.61 [12:13:15] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [12:13:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:13:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.92, 11.36, 9.14 [12:15:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.31, 11.34, 9.41 [12:22:36] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 16.72, 10.64, 6.61 [12:23:25] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.85, 9.63, 6.12 [12:23:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 7.38, 9.30, 9.26 [12:24:58] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 16.46, 12.56, 7.68 [12:25:06] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw132 [12:25:25] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.28, 12.44, 7.58 [12:26:18] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw131 mw133 [12:27:06] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [12:27:25] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.73, 11.79, 7.95 [12:28:18] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [12:31:25] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.11, 12.56, 9.11 [12:31:25] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.38, 10.92, 8.23 [12:32:37] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.24, 11.19, 9.39 [12:33:00] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 5.48, 11.49, 9.66 [12:33:23] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 4.75, 8.70, 7.74 [12:33:25] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 4.98, 9.62, 8.45 [12:34:37] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 4.31, 8.74, 8.70 [12:34:58] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.10, 9.38, 9.09 [12:41:02] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchising.org.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:09:34] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp25.miraheze.org - CNAME OK [13:50:12] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.68, 6.33, 5.69 [13:52:09] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.57, 6.06, 5.68 [14:13:11] PROBLEM - wiki.walkscape.app - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.walkscape.app' expires in 15 day(s) (Sun 07 Jan 2024 13:51:22 GMT +0000). [14:13:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/234d38ac7b36...3d4ff1c5d218 [14:13:28] [02miraheze/ssl] 07MirahezeSSLBot 033d4ff1c - Bot: Update SSL cert for wiki.walkscape.app [14:30:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a33d2f99b39d...a8e2e47097dd [14:30:08] [02miraheze/puppet] 07paladox 03a8e2e47 - varnish: fix if statement [14:42:23] RECOVERY - wiki.walkscape.app - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.walkscape.app' will expire on Thu 21 Mar 2024 13:13:18 GMT +0000. [15:53:41] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 8.82, 4.75, 2.15 [15:55:43] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [15:55:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:01:49] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 365s [16:01:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:18] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.84, 3.32, 2.00 [16:11:15] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.95, 2.88, 2.00 [16:21:15] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.17, 3.40, 2.58 [16:25:15] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.42, 3.76, 2.95 [16:29:15] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.21, 3.37, 2.97 [16:37:15] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.69, 3.90, 3.26 [16:42:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.34, 6.48, 5.91 [16:43:15] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.09, 3.35, 3.31 [16:46:58] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.69, 6.64, 6.09 [16:47:16] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.14, 3.41, 3.35 [16:49:15] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.68, 3.19, 3.28 [17:08:10] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/11db568ab522...4fe91ae7f127 [17:08:12] [02miraheze/mediawiki] 07paladox 034fe91ae - Update Anisa [17:08:57] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [17:09:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:09:41] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 1.95, 2.22, 3.79 [17:10:49] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4fe91ae7f127...eed740f9405a [17:10:50] [02miraheze/mediawiki] 07paladox 03eed740f - Update Flatbox to bd6851996d23db005c64aebb64dda7651de6df5b [17:13:41] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.47, 1.84, 3.27 [17:16:58] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/eed740f9405a...1c0f330fefeb [17:17:01] [02miraheze/mediawiki] 07paladox 031c0f330 - Update Femiwiki [17:19:27] [02miraheze/femiwiki-deploy] 07paladox deleted branch 03REL1_39 [17:19:30] [02femiwiki-deploy] 07paladox deleted branch 03REL1_39 - 13https://github.com/miraheze/femiwiki-deploy [17:23:56] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 898s [17:24:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:25:54] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [17:25:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:27:40] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 105s [17:27:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-5405-paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/702fabe02006 [17:34:04] [02miraheze/mw-config] 07paladox 03702fabe - Revert "Set $wgExtensionInfoMTime to use confActualMtime in MirahezeFunctions (#5405)" [17:34:05] [02mw-config] 07paladox created branch 03revert-5405-paladox-patch-4 - 13https://github.com/miraheze/mw-config [17:34:07] [02mw-config] 07paladox opened pull request 03#5407: Revert "Set $wgExtensionInfoMTime to use confActualMtime in MirahezeFunctions" - 13https://github.com/miraheze/mw-config/pull/5407 [17:34:12] [02mw-config] 07paladox closed pull request 03#5407: Revert "Set $wgExtensionInfoMTime to use confActualMtime in MirahezeFunctions" - 13https://github.com/miraheze/mw-config/pull/5407 [17:34:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/094ad399dafa...91d00b46a985 [17:34:15] [02miraheze/mw-config] 07paladox 0391d00b4 - Revert "Set $wgExtensionInfoMTime to use confActualMtime in MirahezeFunctions" (#5407) [17:34:16] [02miraheze/mw-config] 07paladox deleted branch 03revert-5405-paladox-patch-4 [17:34:18] [02mw-config] 07paladox deleted branch 03revert-5405-paladox-patch-4 - 13https://github.com/miraheze/mw-config [17:35:03] miraheze/mw-config - paladox the build passed. [17:35:12] miraheze/mw-config - paladox the build passed. [17:39:52] !log [@mwtask141] starting deploy of {'config': True} to all [17:39:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:40:03] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s [17:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:40:59] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/1c0f330fefeb...e314333870f2 [17:41:01] [02miraheze/mediawiki] 07paladox 03e314333 - Update Mirage [17:42:09] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/91d00b46a985...e6414126dc89 [17:42:11] [02miraheze/mw-config] 07paladox 03e641412 - Allow mirage for MediaWiki 1.41+ [17:42:54] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [17:42:57] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 3s [17:43:01] miraheze/mw-config - paladox the build passed. [17:43:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:04] !log [paladox@test131] starting deploy of {'config': True} to all [17:43:05] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [17:43:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:23] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [17:43:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:46:05] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 161s [17:46:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:50:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 6.67, 5.83 [17:54:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.67, 6.94, 6.12 [17:56:33] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.47, 6.26, 5.97 [18:09:34] !log [@mwtask141] starting deploy of {'config': True} to all [18:09:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:09:44] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s [18:09:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:51] PROBLEM - mhdh.pj568.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mhdh.pj568.eu.org' expires in 15 day(s) (Sun 07 Jan 2024 18:45:17 GMT +0000). [18:54:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3d4ff1c5d218...f54b3c462997 [18:54:10] [02miraheze/ssl] 07MirahezeSSLBot 03f54b3c4 - Bot: Update SSL cert for mhdh.pj568.eu.org [18:58:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.49, 6.87, 6.02 [19:02:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 4.99, 6.96, 6.37 [19:04:31] PROBLEM - wiki.zamnhacking.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zamnhacking.net' expires in 15 day(s) (Sun 07 Jan 2024 18:44:34 GMT +0000). [19:04:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f54b3c462997...320c7b1c13f9 [19:04:48] [02miraheze/ssl] 07MirahezeSSLBot 03320c7b1 - Bot: Update SSL cert for wiki.zamnhacking.net [19:04:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.74, 7.60, 6.65 [19:05:26] PROBLEM - wiki.moores.tech - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.moores.tech' expires in 15 day(s) (Sun 07 Jan 2024 18:48:34 GMT +0000). [19:05:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/320c7b1c13f9...5de37497a884 [19:05:42] [02miraheze/ssl] 07MirahezeSSLBot 035de3749 - Bot: Update SSL cert for wiki.moores.tech [19:06:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.76, 7.41, 6.68 [19:10:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.12, 7.05, 6.66 [19:12:58] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.85, 6.14, 6.37 [19:21:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.20, 7.00, 6.22 [19:23:51] RECOVERY - mhdh.pj568.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mhdh.pj568.eu.org' will expire on Thu 21 Mar 2024 17:54:02 GMT +0000. [19:23:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.72, 6.81, 6.24 [19:25:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.57, 6.61, 6.22 [19:38:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.90, 6.74, 6.39 [19:40:58] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.20, 6.10, 6.19 [19:46:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.41, 6.64, 6.32 [19:48:58] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.18, 6.63, 6.36 [20:03:26] RECOVERY - wiki.moores.tech - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.moores.tech' will expire on Thu 21 Mar 2024 18:05:31 GMT +0000. [20:03:32] RECOVERY - wiki.zamnhacking.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zamnhacking.net' will expire on Thu 21 Mar 2024 18:04:37 GMT +0000. [20:27:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.71, 6.98, 6.36 [20:29:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.02, 5.96, 6.05 [20:37:27] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.83, 3.83, 2.83 [20:41:17] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.99, 3.77, 3.02 [20:43:12] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 1.68, 2.96, 2.81 [21:41:01] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [22:09:33] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchising.org.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:16:19] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [22:26:21] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.80, 6.70, 5.96 [22:28:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.78, 5.89, 5.76 [22:39:02] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp25.miraheze.org - CNAME OK [23:08:19] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:11:58] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [23:39:58] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures