[00:01:25] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.01358512044 secs [00:03:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.27, 6.57, 4.91 [00:05:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.87, 5.81, 4.84 [00:05:13] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.86, 3.45, 2.06 [00:07:08] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.60, 3.47, 2.23 [00:09:07] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.95, 3.26, 2.29 [00:25:42] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset 0.1032502949 secs [00:30:42] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [00:35:42] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset 0.08895891905 secs [00:43:48] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1818795502 secs [00:45:48] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.008225560188 secs [01:00:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [01:47:15] PROBLEM - wiki.col6.de - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query col6.de. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:15:59] RECOVERY - wiki.col6.de - reverse DNS on sslhost is OK: SSL OK - wiki.col6.de reverse DNS resolves to cp25.miraheze.org - CNAME OK [03:02:37] PROBLEM - mon141 Backups Grafana on mon141 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864140 seconds old and 92 bytes [03:12:46] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:23:17] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 1.791 second response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [03:55:02] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [03:55:04] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:57:05] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 2.521 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [03:59:02] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [04:11:18] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 2.65, 1.80, 0.77 [04:13:18] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 1.19, 1.45, 0.77 [04:22:00] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:23:59] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.981 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [04:48:14] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:54:29] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 1.353 second response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [05:12:16] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:16:25] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 2.825 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [05:23:00] PROBLEM - wiki.gestaltonline.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gestaltonline.ca' expires in 15 day(s) (Mon 06 Nov 2023 05:13:51 GMT +0000). [05:25:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/676195c4982d...9cdcfb4e4cb1 [05:25:30] [02miraheze/ssl] 07MirahezeSSLBot 039cdcfb4 - Bot: Update SSL cert for wiki.gestaltonline.ca [05:30:42] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [05:38:51] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:40:50] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.575 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [05:42:00] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 15.25, 10.49, 8.34 [05:42:58] PROBLEM - theenglishlakes.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'theenglishlakes.uk' expires in 15 day(s) (Mon 06 Nov 2023 05:13:09 GMT +0000). [05:43:55] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.60, 11.42, 8.98 [05:44:01] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1180534959 secs [05:44:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9cdcfb4e4cb1...a7098d248b20 [05:44:43] [02miraheze/ssl] 07MirahezeSSLBot 03a7098d2 - Bot: Update SSL cert for theenglishlakes.uk [05:45:51] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.07, 10.20, 8.85 [05:47:14] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:49:13] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.877 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [06:00:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [06:07:46] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:11:57] RECOVERY - theenglishlakes.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'theenglishlakes.uk' will expire on Fri 19 Jan 2024 04:44:33 GMT +0000. [06:14:02] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 2.202 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [06:17:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.20, 10.81, 9.15 [06:18:26] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:19:31] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.57, 9.81, 9.00 [06:21:00] RECOVERY - wiki.gestaltonline.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gestaltonline.ca' will expire on Fri 19 Jan 2024 04:25:19 GMT +0000. [06:21:27] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:24:49] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 9.242 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [06:32:01] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.08914282918 secs [06:49:27] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:50] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:58:13] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 1.465 second response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [07:01:55] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.73, 10.84, 9.60 [07:02:38] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:05:46] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.66, 9.45, 9.40 [07:21:45] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 5.936 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [07:24:39] PROBLEM - removededm.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for removededm.com could not be found [07:26:09] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:28:13] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 6.356 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [07:29:06] PROBLEM - removededm.com - LetsEncrypt on sslhost is CRITICAL: connect to address removededm.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:29:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.11, 10.85, 9.20 [07:31:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.82, 10.42, 9.23 [07:34:10] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.53, 10.79, 8.70 [07:35:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.63, 12.32, 10.28 [07:44:10] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.08, 11.27, 10.12 [07:46:10] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.85, 10.05, 9.82 [07:52:12] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.04, 3.49, 3.01 [07:53:17] RECOVERY - removededm.com - reverse DNS on sslhost is OK: SSL OK - removededm.com reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK [07:53:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.10, 11.75, 11.82 [07:54:08] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.04, 3.30, 3.00 [07:57:42] RECOVERY - removededm.com - LetsEncrypt on sslhost is OK: OK - Certificate 'removededm.com' will expire on Tue 21 Nov 2023 16:31:05 GMT +0000. [08:03:31] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.05, 8.01, 9.78 [08:10:42] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:19:04] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.560 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [08:23:28] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:30:16] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1589131653 secs [08:32:16] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset 0.02266994119 secs [08:34:04] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 4.736 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [08:40:22] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1634049118 secs [08:44:22] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.0560939014 secs [08:51:46] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:52:28] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1125891209 secs [08:55:56] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 4.004 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [08:56:28] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.04289036989 secs [09:04:16] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:17:01] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 7.758 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [09:23:28] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:25:23] PROBLEM - theagonian.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'theagonian.com' expires in 15 day(s) (Mon 06 Nov 2023 09:22:19 GMT +0000). [09:34:00] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 2.731 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [09:49:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.61, 2.95, 2.33 [09:51:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.03, 2.51, 2.24 [09:54:03] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:56:10] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 9.661 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [10:16:24] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:29:02] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.433 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [10:48:01] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:54:20] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 6.064 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [11:04:57] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:06:56] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.628 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [11:19:41] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:23:51] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 4.187 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [11:32:31] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:34:34] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 5.523 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [11:35:02] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /home/paladox/patch-cu_changes-add-cuc_only_for_read_old.sql [11:35:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:36:09] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /home/paladox/patch-cu_private_event-def.sql [11:36:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:58] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:56:26] PROBLEM - db142 MariaDB on db142 is CRITICAL: Can't connect to MySQL server on 'db142.miraheze.org' (115) [11:57:02] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [11:58:26] RECOVERY - db142 MariaDB on db142 is OK: Uptime: 234 Threads: 9 Questions: 12763 Slow queries: 942 Opens: 1019 Open tables: 989 Queries per second avg: 54.542 [12:00:13] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 5.081 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [12:01:02] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [12:08:46] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:10:49] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 4.892 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [12:13:55] [02miraheze/femiwiki-deploy] 07paladox deleted branch 03REL1_38 [12:13:56] [02femiwiki-deploy] 07paladox deleted branch 03REL1_38 - 13https://github.com/miraheze/femiwiki-deploy [12:15:12] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:19:17] [02miraheze/femiwiki-deploy] 07paladox pushed 031 commit to 03REL1_40 [+2989/-10/±1] 13https://github.com/miraheze/femiwiki-deploy/compare/84b0444b1c3d...7cad81a5666d [12:19:18] [02miraheze/femiwiki-deploy] 07paladox 037cad81a - Update package.json to match REL1_40 branch [12:21:29] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 1.002 second response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [12:25:54] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:36:24] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.480 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [12:39:19] [02miraheze/femiwiki-deploy] 07paladox created branch 03REL1_41 13https://github.com/miraheze/femiwiki-deploy/commit/14e4b5ac5cb7d0f6d346c3b6cf0708fd58e99b42 [12:39:21] [02femiwiki-deploy] 07paladox created branch 03REL1_41 - 13https://github.com/miraheze/femiwiki-deploy [12:42:42] [02miraheze/femiwiki-deploy] 07paladox pushed 031 commit to 03REL1_41 [+0/-2997/±3] 13https://github.com/miraheze/femiwiki-deploy/compare/14e4b5ac5cb7...a7b7e6f1166a [12:42:45] [02miraheze/femiwiki-deploy] 07paladox 03a7b7e6f - Update package.json to match REL1_40 [12:44:21] [02femiwiki-deploy] 07paladox commented on commit 03a7b7e6f1166af96b690db483d51f4463f6c79faa - 13https://github.com/miraheze/femiwiki-deploy/commit/a7b7e6f1166af96b690db483d51f4463f6c79faa#commitcomment-130577582 [12:46:52] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:57:24] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 2.438 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [13:03:48] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:10:02] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.545 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [13:14:26] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:20:25] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [13:20:40] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.260 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [13:20:40] PROBLEM - ns1 Current Load on ns1 is WARNING: LOAD WARNING - total load average: 1.75, 0.55, 0.19 [13:21:07] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:21:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:22:06] RECOVERY - ns1 conntrack_table_size on ns1 is OK: OK: nf_conntrack is 0 % full [13:22:10] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 2691MiB (37% inode=83%); [13:22:13] RECOVERY - ns1 ferm_active on ns1 is OK: OK ferm input default policy is set [13:22:24] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.3518950045 secs [13:22:25] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:22:39] RECOVERY - ns1 Current Load on ns1 is OK: LOAD OK - total load average: 0.40, 0.46, 0.20 [13:22:54] PROBLEM - mail121 SSH on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:35] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:41] PROBLEM - mail121 SMTP on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:14] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 22.33, 14.78, 6.51 [13:25:30] PROBLEM - mail121 PowerDNS Recursor on mail121 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:25:44] RECOVERY - mail121 SMTP on mail121 is OK: SMTP OK - 9.168 sec. response time [13:28:18] PROBLEM - mail121 HTTPS on mail121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10000 milliseconds [13:28:59] PROBLEM - mail121 conntrack_table_size on mail121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:21] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:31] PROBLEM - mail121 APT on mail121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:33:04] PROBLEM - mail121 php-fpm on mail121 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [13:35:27] PROBLEM - mail121 ferm_active on mail121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:38:10] PROBLEM - mail121 SMTP on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:12] PROBLEM - mail121 IMAP on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:43] RECOVERY - mail121 php-fpm on mail121 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [13:40:44] RECOVERY - mail121 conntrack_table_size on mail121 is OK: OK: nf_conntrack is 0 % full [13:42:35] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6169 bytes in 1.391 second response time [13:43:07] RECOVERY - mail121 HTTPS on mail121 is OK: HTTP OK: HTTP/2 301 - 227 bytes in 0.037 second response time [13:43:23] RECOVERY - mail121 SSH on mail121 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [13:43:53] RECOVERY - mail121 PowerDNS Recursor on mail121 is OK: DNS OK: 2.245 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [13:43:54] RECOVERY - mail121 ferm_active on mail121 is OK: OK ferm input default policy is set [13:44:11] RECOVERY - mail121 SMTP on mail121 is OK: SMTP OK - 0.009 sec. response time [13:44:14] RECOVERY - mail121 IMAP on mail121 is OK: IMAP OK - 0.066 second response time on 2a10:6740::6:307 port 143 [* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS LOGINDISABLED] Dovecot (Debian) ready.] [13:44:17] RECOVERY - mail121 APT on mail121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:44:22] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 31 minutes ago with 0 failures [14:09:53] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5042586625 secs [14:14:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f9b0e40f08ae [14:14:32] [02miraheze/puppet] 07paladox 03f9b0e40 - dns: fix issue on ns1 where it can crash if a oom is triggered [14:14:33] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [14:14:35] [02puppet] 07paladox opened pull request 03#3469: dns: fix issue on ns1 where it can crash if a oom is triggered - 13https://github.com/miraheze/puppet/pull/3469 [14:17:26] [02puppet] 07paladox synchronize pull request 03#3469: dns: fix issue on ns1 where it can crash if a oom is triggered - 13https://github.com/miraheze/puppet/pull/3469 [14:17:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f9b0e40f08ae...2bf8c1df6c94 [14:17:29] [02miraheze/puppet] 07paladox 032bf8c1d - Update dns.pp [14:17:53] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.4427922666 secs [14:18:17] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:18:51] [02puppet] 07paladox closed pull request 03#3469: dns: fix issue on ns1 where it can crash if a oom is triggered - 13https://github.com/miraheze/puppet/pull/3469 [14:18:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ab3947cac617...c674ad0b15e0 [14:18:54] [02miraheze/puppet] 07paladox 03c674ad0 - dns: fix issue on ns1 where it can crash if a oom is triggered (#3469) [14:18:56] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [14:18:57] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [14:23:18] PROBLEM - mail121 Current Load on mail121 is WARNING: LOAD WARNING - total load average: 0.18, 0.13, 2.00 [14:27:18] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 0.12, 0.14, 1.58 [14:32:32] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/606f438a5a7e [14:32:34] [02miraheze/dns] 07paladox 03606f438 - Set tcp_clients_per_thread to 1024 [14:32:36] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [14:32:38] [02dns] 07paladox opened pull request 03#445: Set tcp_clients_per_thread to 1024 - 13https://github.com/miraheze/dns/pull/445 [14:32:50] [02dns] 07paladox synchronize pull request 03#445: Set tcp_clients_per_thread to 1024 - 13https://github.com/miraheze/dns/pull/445 [14:32:51] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/606f438a5a7e...ce39a15c8e27 [14:32:52] [02miraheze/dns] 07paladox 03ce39a15 - Update config [14:32:57] [02dns] 07paladox closed pull request 03#445: Set tcp_clients_per_thread to 1024 - 13https://github.com/miraheze/dns/pull/445 [14:33:00] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/82bcffe38f62...3de179ad47f2 [14:33:02] [02miraheze/dns] 07paladox 033de179a - Set tcp_clients_per_thread to 1024 (#445) [14:33:05] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [14:33:08] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [14:35:53] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5003978908 secs [14:46:17] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:03:37] PROBLEM - wiki.songngu.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.songngu.xyz' expires in 15 day(s) (Mon 06 Nov 2023 14:47:09 GMT +0000). [15:04:10] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.01, 10.67, 9.37 [15:04:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/d2f6c8396a37 [15:04:31] [02miraheze/mw-config] 07paladox 03d2f6c83 - Disallow wiki-creations [15:04:33] [02mw-config] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/mw-config [15:04:36] [02mw-config] 07paladox opened pull request 03#5340: Disallow wiki-creations - 13https://github.com/miraheze/mw-config/pull/5340 [15:04:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7369e229ca1f [15:04:58] [02miraheze/puppet] 07paladox 037369e22 - Switch MW branch to REL1_40 [15:05:00] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [15:05:01] [02puppet] 07paladox opened pull request 03#3470: Switch MW branch to REL1_40 - 13https://github.com/miraheze/puppet/pull/3470 [15:05:46] [02mw-config] 07paladox closed pull request 03#5331: Fix mobile - 13https://github.com/miraheze/mw-config/pull/5331 [15:05:49] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [15:05:50] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [15:06:10] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.65, 9.12, 8.96 [15:06:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a7098d248b20...bdb477b1d644 [15:06:52] miraheze/mw-config - paladox the build passed. [15:06:53] [02miraheze/ssl] 07MirahezeSSLBot 03bdb477b - Bot: Update SSL cert for wiki.songngu.xyz [15:15:49] [02mw-config] 07paladox closed pull request 03#5340: Disallow wiki-creations - 13https://github.com/miraheze/mw-config/pull/5340 [15:15:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5879ba5c68d9...433b66251e18 [15:15:55] [02miraheze/mw-config] 07paladox 03433b662 - Disallow wiki-creations (#5340) [15:15:57] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-5 [15:15:58] [02mw-config] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/mw-config [15:16:54] miraheze/mw-config - paladox the build passed. [15:16:59] [02puppet] 07paladox closed pull request 03#3470: Switch MW branch to REL1_40 - 13https://github.com/miraheze/puppet/pull/3470 [15:17:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c674ad0b15e0...9cfa9f2b1083 [15:17:04] [02miraheze/puppet] 07paladox 039cfa9f2 - Switch MW branch to REL1_40 (#3470) [15:17:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [15:17:08] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [15:19:16] !log [@mwtask141] starting deploy of {'config': True} to all [15:19:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:19:27] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 11s [15:19:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:26:42] !log [@test131] starting deploy of {'config': True} to all [15:26:44] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 2s [15:26:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:26:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:32:14] RECOVERY - wiki.songngu.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.songngu.xyz' will expire on Fri 19 Jan 2024 14:06:40 GMT +0000. [15:34:40] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.83, 3.42, 2.89 [15:36:10] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.14, 9.77, 8.37 [15:36:37] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.41, 3.05, 2.82 [15:37:41] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 13.28, 10.16, 6.95 [15:38:14] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.98, 9.30, 6.48 [15:39:17] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.46, 2.93, 1.60 [15:39:39] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.42, 9.89, 7.19 [15:40:10] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 8.62, 10.43, 9.11 [15:41:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 7.53, 6.31, 4.82 [15:41:14] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.99, 2.63, 1.66 [15:41:37] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.55, 8.31, 6.93 [15:41:40] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.50, 11.12, 9.27 [15:42:10] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 15.55, 12.36, 9.97 [15:42:14] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 10.99, 10.97, 7.85 [15:43:36] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.96, 11.41, 9.60 [15:44:10] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.37, 11.48, 9.94 [15:44:14] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.46, 8.80, 7.42 [15:44:55] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [15:45:14] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 5.41, 6.48, 5.29 [15:47:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.40, 11.85, 10.13 [15:48:10] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 15.45, 12.29, 10.51 [15:51:55] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 12.53, 10.68, 9.08 [15:52:06] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.61, 3.14, 2.86 [15:54:02] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.49, 2.83, 2.78 [15:55:43] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 8.81, 11.13, 9.75 [15:57:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 8.84, 11.22, 10.99 [15:58:10] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.65, 11.62, 11.26 [15:59:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 14.81, 12.10, 11.30 [16:00:57] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:01:26] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 6.37, 9.08, 9.37 [16:04:10] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.22, 11.41, 11.21 [16:08:10] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 8.91, 10.93, 11.14 [16:10:51] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1107153594 secs [16:11:57] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.23, 3.31, 2.43 [16:13:54] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.03, 3.29, 2.53 [16:15:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 9.77, 11.19, 11.73 [16:18:10] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.64, 9.09, 10.16 [16:29:31] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.37, 9.14, 10.05 [16:30:21] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.96, 3.62, 3.16 [16:32:18] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.83, 3.34, 3.12 [16:36:51] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.01587751508 secs [16:42:00] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.30, 3.79, 3.37 [16:43:57] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.61, 3.77, 3.42 [16:47:51] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.16, 3.40, 3.34 [16:51:07] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.47, 2.89, 1.67 [16:53:07] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.02, 2.50, 1.68 [16:53:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.25, 10.69, 9.98 [16:53:58] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.19, 10.55, 9.50 [16:55:31] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.97, 10.95, 10.18 [16:55:56] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.80, 10.47, 9.59 [16:57:54] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.51, 9.46, 9.33 [16:59:31] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.57, 9.95, 9.99 [17:06:57] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1262998581 secs [17:10:57] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.04329109192 secs [17:33:03] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1908093691 secs [17:37:03] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset 0.02940750122 secs [17:38:11] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.40, 10.14, 8.51 [17:39:12] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.21, 10.37, 9.13 [17:40:10] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.24, 9.62, 8.51 [17:40:30] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.10, 3.47, 3.29 [17:42:05] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.84, 6.81, 5.83 [17:42:27] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.94, 3.32, 3.26 [17:43:03] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.71, 9.92, 9.28 [17:44:01] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.04, 5.97, 5.63 [18:03:09] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1114489138 secs [18:33:59] PROBLEM - mw142 Puppet on mw142 is UNKNOWN: NRPE: Unable to read output [18:35:10] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset 0.04598298669 secs [18:38:04] PROBLEM - mw134 Puppet on mw134 is UNKNOWN: NRPE: Unable to read output [18:38:56] PROBLEM - mw143 Puppet on mw143 is UNKNOWN: NRPE: Unable to read output [18:41:46] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.12, 3.47, 2.90 [18:42:55] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [18:43:43] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.23, 3.39, 2.94 [18:46:19] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1441561282 secs [18:52:32] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [19:00:19] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.09750819206 secs [19:01:59] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:04:56] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:06:04] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:10:55] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:21:54] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp24.miraheze.org - CNAME OK [19:44:28] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.82, 10.36, 6.77 [19:46:23] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.39, 8.43, 6.50 [20:03:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.77, 3.25, 2.67 [20:05:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.23, 3.07, 2.67 [20:57:12] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [20:57:32] [02mediawiki] 07MacFan4000 created branch 03REL1_41 - 13https://github.com/miraheze/mediawiki [21:01:12] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [22:00:14] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [22:00:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:31] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [22:15:33] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 665 bytes in 1.502 second response time [22:17:31] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [22:20:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9cfa9f2b1083...571f72160134 [22:20:05] [02miraheze/puppet] 07paladox 03571f721 - deploy-mediawiki: fix path [22:21:32] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.017 second response time [22:31:38] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw131.miraheze.org [22:31:38] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 665 bytes in 0.234 second response time [22:31:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:38] PROBLEM - mw131 MediaWiki Rendering on mw131 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [22:34:45] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.97, 4.35, 3.25 [22:35:50] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [22:35:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:37] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.38, 3.78, 3.26 [22:42:29] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.69, 3.36, 3.22 [22:43:53] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw133.miraheze.org [22:43:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:44:25] PROBLEM - mw132 MediaWiki Rendering on mw132 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [22:44:38] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [22:44:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:45:43] PROBLEM - mw133 MediaWiki Rendering on mw133 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [22:46:22] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.27, 3.90, 3.47 [22:47:31] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [22:48:19] !log [paladox@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [22:48:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:17] !log [paladox@mwtask141] starting deploy of {'world': True} to mwtask141 [22:55:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:57] !log [paladox@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [22:58:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:04] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to mwtask141 [22:58:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:19] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to mwtask141 [23:01:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:27] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset 0.1024635434 secs [23:04:26] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 12.90, 8.38, 4.02 [23:07:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 7.11, 5.77, 4.58 [23:08:26] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.96, 6.65, 4.50 [23:09:14] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 8.02, 6.46, 4.97 [23:11:32] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.07, 3.54, 2.15 [23:13:13] !log [paladox@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [23:13:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:13:28] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.15, 3.12, 2.18 [23:15:14] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 5.54, 6.73, 5.69 [23:22:27] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset 0.08640271425 secs [23:23:50] !log [paladox@mwtask141] starting deploy of {'config': True, 'force': True} to mwtask141 [23:23:51] !log [paladox@mwtask141] finished deploy of {'config': True, 'force': True} to mwtask141 - SUCCESS in 0s [23:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:41:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 7.50, 6.91, 6.30 [23:47:46] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 6.03, 6.69, 6.44 [23:52:06] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True, 'extensionlist': True, 'force': True, 'ignoretime': True} to mwtask141 [23:52:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log