[00:02:23] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 3.69, 1.67, 0.71 [00:02:32] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 4.14, 3.22, 1.55 [00:04:19] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.93, 1.35, 0.71 [00:04:32] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.88, 2.26, 1.39 [00:14:32] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 4.55, 3.38, 2.32 [00:15:49] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [00:16:32] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 2.70, 3.13, 2.36 [00:28:16] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [01:06:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.58, 3.16, 2.54 [01:08:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.37, 3.27, 2.66 [01:17:10] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:20:26] PROBLEM - cp32 NTP time on cp32 is CRITICAL: NTP CRITICAL: Offset 0.6045287251 secs [01:29:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 2.03, 5.60, 3.72 [01:33:47] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.73, 3.16, 3.18 [01:39:15] PROBLEM - mem141 NTP time on mem141 is WARNING: NTP WARNING: Offset 0.1112785339 secs [01:39:42] !log [@mwtask141] starting deploy of {'config': True} to all [01:39:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:39:52] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s [01:39:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:45:49] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [02:03:27] RECOVERY - mem141 NTP time on mem141 is OK: NTP OK: Offset 0.0923319757 secs [02:14:41] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [02:34:34] PROBLEM - mw142 Puppet on mw142 is UNKNOWN: NRPE: Unable to read output [02:42:41] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:54:48] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.86, 3.48, 3.05 [02:56:47] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.37, 3.06, 2.95 [03:00:41] RECOVERY - db142 Backups SQL on db142 is OK: FILE_AGE OK: /var/log/sql-backup.log is 40 seconds old and 0 bytes [03:02:33] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:10:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.53, 3.22, 2.90 [03:12:39] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.09, 3.20, 2.94 [03:29:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:29:30] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.58, 3.38, 3.13 [03:33:28] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.74, 2.85, 3.00 [03:58:16] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:18:31] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:23:00] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13235 MB (9% inode=99%); [04:27:16] !log [@test131] starting deploy of {'config': True} to all [04:27:17] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [04:27:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:27:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:29:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:33:32] PROBLEM - mw133 Puppet on mw133 is UNKNOWN: NRPE: Unable to read output [04:34:24] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 9.99, 8.27, 5.13 [04:39:00] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7893 MB (5% inode=99%); [04:42:25] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.81, 6.43, 5.97 [04:46:41] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [04:48:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [04:55:00] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 16513 MB (12% inode=99%); [04:58:16] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [05:01:32] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:02:48] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.44, 7.15, 6.31 [05:04:42] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.28, 8.78, 6.99 [05:05:00] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13765 MB (10% inode=99%); [05:06:36] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.34, 7.08, 6.57 [05:10:24] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.81, 8.06, 7.17 [05:10:30] !log [@mwtask141] starting deploy of {'config': True} to all [05:10:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:39] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [05:10:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:12:25] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.72, 6.50, 6.71 [05:12:41] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [05:17:00] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 18074 MB (13% inode=99%); [05:27:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:44:41] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [05:47:10] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:57:23] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:02:19] PROBLEM - bast141 Current Load on bast141 is WARNING: LOAD WARNING - total load average: 1.98, 1.15, 0.53 [06:04:20] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 1.12, 1.35, 0.70 [06:09:58] PROBLEM - mw134 Puppet on mw134 is UNKNOWN: NRPE: Unable to read output [06:10:41] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:15:50] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [06:16:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.48, 3.20, 2.50 [06:18:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.74, 3.06, 2.53 [06:35:58] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:40:02] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.77, 3.89, 3.28 [06:42:01] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.80, 3.80, 3.32 [06:44:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.58, 4.07, 3.47 [06:45:10] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:45:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.82, 3.58, 3.37 [06:49:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.70, 3.33, 3.34 [06:57:13] !log [@test131] starting deploy of {'config': True} to all [06:57:14] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:57:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:53] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.26, 3.79, 3.51 [06:59:52] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.73, 3.81, 3.56 [07:01:51] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.35, 4.02, 3.67 [07:02:19] PROBLEM - bast141 Current Load on bast141 is WARNING: LOAD WARNING - total load average: 1.99, 1.12, 0.53 [07:04:19] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.30, 0.77, 0.47 [07:05:49] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.15, 3.44, 3.55 [07:07:48] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.05, 2.86, 3.32 [07:56:04] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 3.96, 2.84, 1.79 [07:57:58] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.62, 2.52, 1.81 [08:04:19] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 2.69, 1.77, 0.88 [08:05:32] PROBLEM - mw133 Puppet on mw133 is UNKNOWN: NRPE: Unable to read output [08:06:19] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.58, 1.31, 0.82 [08:09:42] PROBLEM - mw143 Puppet on mw143 is UNKNOWN: NRPE: Unable to read output [08:15:40] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [08:29:32] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [08:35:42] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [08:41:40] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:27:02] !log [@test131] starting deploy of {'config': True} to all [09:27:03] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [09:27:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:25:35] PROBLEM - tudien.como.vn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tudien.como.vn All nameservers failed to answer the query. [10:29:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f555b87db078...c9cf44e5b1c8 [10:29:07] [02miraheze/puppet] 07paladox 03c9cf44e - varnish: don’t cache rest.php [10:40:55] PROBLEM - mw141 Puppet on mw141 is UNKNOWN: NRPE: Unable to read output [10:55:35] PROBLEM - tudien.como.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tudien.como.vn could not be found [11:06:54] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:20:48] [02miraheze/CustomHeader] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/CustomHeader/compare/61120728ce77...8c5717cc1722 [11:20:51] [02miraheze/CustomHeader] 07translatewiki 038c5717c - Localisation updates from https://translatewiki.net. [11:20:52] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/DataDump/compare/6b5dc0c64e62...e9fc9cfd1f94 [11:20:53] [02miraheze/DataDump] 07translatewiki 03e9fc9cf - Localisation updates from https://translatewiki.net. [11:20:56] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/4b3e90b7cca3...c80a8b0c6cc0 [11:20:59] [02miraheze/MirahezeMagic] 07translatewiki 03c80a8b0 - Localisation updates from https://translatewiki.net. [11:26:28] miraheze/CustomHeader - translatewiki the build passed. [11:27:08] miraheze/MirahezeMagic - translatewiki the build has errored. [11:27:36] !log [@test131] starting deploy of {'config': True} to all [11:27:37] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [11:27:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:27:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:28:35] miraheze/DataDump - translatewiki the build passed. [11:45:40] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [12:07:34] PROBLEM - wiki.yuanpi.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yuanpi.eu.org' expires in 15 day(s) (Sat 23 Sep 2023 11:53:14 GMT +0000). [12:08:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8b5574cf1963...babdbb3d7c3b [12:08:19] [02miraheze/ssl] 07MirahezeSSLBot 03babdbb3 - Bot: Update SSL cert for wiki.yuanpi.eu.org [12:10:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.94, 3.74, 2.73 [12:11:40] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:14:44] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [12:18:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.54, 3.84, 3.28 [12:24:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.10, 3.92, 3.47 [12:26:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.83, 3.49 [12:38:51] PROBLEM - zhacg.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zhacg.wiki' expires in 15 day(s) (Sat 23 Sep 2023 12:09:45 GMT +0000). [12:39:43] PROBLEM - mw143 Puppet on mw143 is UNKNOWN: NRPE: Unable to read output [12:40:41] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [12:45:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/babdbb3d7c3b...5d7673ca5ade [12:45:27] [02miraheze/ssl] 07MirahezeSSLBot 035d7673c - Bot: Update SSL cert for zhacg.wiki [12:46:01] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 3.28, 3.41, 2.16 [12:48:01] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.75, 2.40, 1.94 [12:50:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.40, 3.91, 3.73 [12:52:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.00, 3.56, 3.62 [13:00:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.57, 3.85, 3.68 [13:02:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.41, 3.77, 3.68 [13:05:42] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:06:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.11, 3.02, 3.40 [13:06:43] RECOVERY - wiki.yuanpi.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yuanpi.eu.org' will expire on Wed 06 Dec 2023 11:08:08 GMT +0000. [13:08:11] RECOVERY - zhacg.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'zhacg.wiki' will expire on Wed 06 Dec 2023 11:45:18 GMT +0000. [13:10:55] PROBLEM - mw141 Puppet on mw141 is UNKNOWN: NRPE: Unable to read output [13:29:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [13:36:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.80, 3.45, 3.03 [13:36:58] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:42:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.06, 3.38, 3.16 [13:49:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.55, 3.54, 3.31 [13:53:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.27, 3.86, 3.48 [13:55:18] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.11, 3.58, 3.43 [13:57:32] !log [@test131] starting deploy of {'config': True} to all [13:57:33] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [13:57:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:57:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:58:16] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:05:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.03, 3.67, 3.52 [14:07:12] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.28, 3.52, 3.49 [14:09:12] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.13, 3.74, 3.57 [14:10:58] PROBLEM - mw132 Puppet on mw132 is UNKNOWN: NRPE: Unable to read output [14:11:11] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.80, 3.71, 3.57 [14:36:58] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:38:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.25, 3.79, 3.68 [14:40:09] !log [@mwtask141] starting deploy of {'config': True} to all [14:40:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:40:18] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [14:40:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:40:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.64, 3.77, 3.70 [14:44:02] PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 2.46, 3.55, 2.20 [14:46:02] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.72, 2.52, 1.98 [15:02:48] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.03, 3.66, 3.58 [15:06:46] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.47, 3.59, 3.57 [15:10:44] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.29, 3.91, 3.70 [15:12:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.74, 3.83, 3.70 [15:16:41] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.05, 3.85, 3.74 [15:18:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.76, 3.72 [15:22:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 16.21, 12.12, 5.71 [15:24:01] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 4.23, 3.79, 2.20 [15:26:01] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.32, 2.78, 2.01 [15:30:06] PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 0.83, 3.13, 3.75 [15:32:06] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 0.67, 2.27, 3.35 [15:36:32] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.39, 3.83, 3.74 [15:44:29] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.46, 3.83, 3.84 [16:04:16] paladox: when you have a chance, ircrcbot needs a restart [16:05:21] Done [16:07:09] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.85, 2.74, 1.75 [16:09:09] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.31, 2.48, 1.77 [16:14:41] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [16:17:10] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:40:41] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:45:49] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [17:15:40] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [17:40:05] !log [@mwtask141] starting deploy of {'config': True} to all [17:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:40:15] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [17:40:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:41:40] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:45:09] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [17:46:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.05, 3.22, 3.38 [17:58:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.49, 3.36, 3.33 [18:02:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.16, 3.22, 3.28 [18:06:34] PROBLEM - mw142 Puppet on mw142 is UNKNOWN: NRPE: Unable to read output [18:08:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.27, 4.19, 3.66 [18:17:09] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:32:34] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:41:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: LOAD CRITICAL - total load average: 3.84, 4.26, 2.34 [18:43:25] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.51, 3.23, 2.18 [18:56:01] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 3.04, 3.93, 2.17 [18:58:01] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.11, 2.92, 2.01 [19:06:54] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [19:09:58] PROBLEM - mw134 Puppet on mw134 is UNKNOWN: NRPE: Unable to read output [19:10:52] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [19:15:10] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [19:20:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.28, 3.50, 3.95 [19:27:17] !log [@test131] starting deploy of {'config': True} to all [19:27:18] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [19:27:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:58] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:40:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.29, 3.85, 3.77 [19:42:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.92, 3.78, 3.75 [19:47:10] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:50:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.02, 3.76, 3.73 [19:52:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.99, 3.85, 3.77 [20:15:49] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [20:21:49] PROBLEM - cloud13 Puppet on cloud13 is UNKNOWN: NRPE: Unable to read output [20:28:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.79, 2.73, 3.23 [20:41:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: LOAD CRITICAL - total load average: 4.34, 4.99, 2.84 [20:43:25] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 1.12, 3.56, 2.57 [20:45:25] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 0.52, 2.52, 2.31 [20:51:49] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:11:25] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 2.64, 3.74, 2.47 [21:13:25] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.75, 3.15, 2.41 [21:30:14] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [21:32:31] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 3.57, 2.23, 1.04 [21:33:02] PROBLEM - bast141 PowerDNS Recursor on bast141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:34:57] RECOVERY - bast141 PowerDNS Recursor on bast141 is OK: DNS OK: 0.491 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [21:38:19] PROBLEM - bast141 Current Load on bast141 is WARNING: LOAD WARNING - total load average: 0.30, 1.86, 1.42 [21:40:19] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.13, 1.27, 1.26 [21:43:40] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [21:56:48] !log [@test131] starting deploy of {'config': True} to all [21:56:49] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [21:56:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:56:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:59:07] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:04:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 4.85, 4.18, 2.30 [22:06:45] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 1.01, 2.92, 2.06 [22:11:40] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:14:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.80, 3.19, 2.65 [22:16:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.68, 3.14, 2.71 [22:26:40] PROBLEM - mon141 Puppet on mon141 is UNKNOWN: NRPE: Unable to read output [22:42:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 6.39, 5.18 [22:44:38] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.37, 5.58, 5.03 [22:48:41] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [22:52:40] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:04:24] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 13.34, 8.21, 3.89 [23:08:24] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 1.88, 6.87, 4.55 [23:10:24] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.52, 4.68, 4.02 [23:10:41] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:17:10] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:32:29] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 2.40, 1.51, 0.65 [23:38:19] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.65, 1.48, 0.97 [23:40:59] PROBLEM - mw132 Puppet on mw132 is UNKNOWN: NRPE: Unable to read output [23:45:50] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [23:45:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: LOAD CRITICAL - total load average: 3.01, 4.28, 2.81 [23:47:56] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 1.12, 3.14, 2.57 [23:48:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.85, 3.25, 2.56 [23:52:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.40, 3.85, 2.97 [23:54:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.53, 3.79, 3.06 [23:56:40] PROBLEM - mon141 Puppet on mon141 is UNKNOWN: NRPE: Unable to read output