[00:00:18] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.40, 3.67, 3.41 [00:01:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/409ea8e64576...faaa5c52dee1 [00:01:17] [02miraheze/mw-config] 07paladox 03faaa5c5 - Increase shards to 256 for mysql-multiwrite [00:01:40] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [00:01:58] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 17s [00:02:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.01, 3.81, 3.49 [00:02:19] miraheze/mw-config - paladox the build passed. [00:02:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.68, 4.53, 3.39 [00:02:31] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 23.54, 16.25, 9.47 [00:04:26] RECOVERY - db142 Disk Space on db142 is OK: DISK OK - free space: / 43056 MB (12% inode=96%); [00:05:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/61a698dbe2a1 [00:05:45] [02miraheze/mw-config] 07paladox 0361a698d - Stop creating wikis on db142 [00:05:46] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [00:05:48] [02mw-config] 07paladox opened pull request 03#5264: Stop creating wikis on db142 - 13https://github.com/miraheze/mw-config/pull/5264 [00:06:21] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/61a698dbe2a1...1e36e5c099e9 [00:06:24] [02miraheze/mw-config] 07github-actions 031e36e5c - CI: lint code to MediaWiki standards [00:06:25] [02mw-config] 07github-actions[bot] synchronize pull request 03#5264: Stop creating wikis on db142 - 13https://github.com/miraheze/mw-config/pull/5264 [00:06:54] miraheze/mw-config - paladox the build passed. [00:07:00] [02mw-config] 07paladox closed pull request 03#5264: Stop creating wikis on db142 - 13https://github.com/miraheze/mw-config/pull/5264 [00:07:03] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [00:07:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/faaa5c52dee1...190f4d3604a2 [00:07:06] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [00:07:08] [02miraheze/mw-config] 07paladox 03190f4d3 - Stop creating wikis on db142 (#5264) [00:07:11] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [00:07:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:07:32] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [00:07:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:07:42] !log [paladox@mwtask141] starting deploy of {'config': True} to all [00:07:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:07:53] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s [00:08:01] miraheze/mw-config - paladox the build passed. [00:08:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:08:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.59, 3.93, 3.50 [00:08:22] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 6.60, 10.26, 8.98 [00:10:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.42, 3.50, 3.60 [00:10:21] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.26, 8.58, 8.52 [00:12:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.61, 2.83, 3.17 [00:14:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.93, 2.60, 3.20 [00:17:39] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8826 MB (6% inode=99%); [00:26:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.13, 4.69, 3.97 [00:26:25] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:27:30] !log [@test131] starting deploy of {'config': True} to all [00:27:32] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [00:27:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.56, 3.90, 3.77 [00:34:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.33, 2.61, 3.23 [00:34:19] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:36:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.04, 3.57, 2.86 [00:38:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.12, 2.92, 2.71 [00:54:55] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:20] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [00:55:21] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 25.36, 20.00, 11.92 [00:56:50] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.250 second response time [00:56:52] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw142 [00:58:52] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 15 backends are healthy [00:59:15] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 4.65, 11.09, 10.04 [01:01:11] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.83, 9.40, 9.54 [01:05:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.47, 4.87, 3.32 [01:15:43] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.76, 3.89, 3.76 [01:19:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.65, 2.64, 3.26 [01:26:25] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:31:44] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [01:47:51] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.53, 3.39, 3.08 [01:49:51] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.84, 3.10, 3.00 [01:55:22] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [02:05:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.86, 4.31, 3.22 [02:11:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.81, 3.75, 3.47 [02:12:44] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 13.56, 13.17, 8.00 [02:13:14] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.04, 14.05, 8.93 [02:13:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.20, 6.01, 4.28 [02:13:35] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 13.04, 11.39, 6.92 [02:14:43] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 6.12, 10.51, 7.65 [02:15:10] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 5.52, 10.94, 8.39 [02:15:33] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 3.85, 8.45, 6.37 [02:16:01] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 10.33, 11.38, 7.15 [02:16:43] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 4.12, 8.37, 7.22 [02:16:52] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.75, 16.19, 10.42 [02:17:06] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 3.75, 8.60, 7.84 [02:18:00] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 4.04, 8.73, 6.69 [02:19:08] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1060797572 secs [02:19:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.16, 3.86, 3.84 [02:20:49] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 5.06, 10.13, 9.23 [02:23:41] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.57, 3.47, 3.09 [02:24:29] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:25:41] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.71, 2.77, 2.88 [02:27:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.86, 3.54, 3.55 [02:31:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.78, 3.27, 3.45 [02:32:09] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [02:33:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.06, 4.27, 3.84 [02:38:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.94, 3.38, 3.60 [02:42:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.54, 4.09, 3.80 [02:43:39] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7969 MB (5% inode=99%); [02:44:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.89, 3.13, 3.48 [02:46:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.45, 2.84, 3.33 [02:47:08] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09122437239 secs [02:54:26] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:00:48] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:03:32] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 10.16, 5.78, 3.48 [03:06:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.82, 4.40, 3.46 [03:07:30] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [03:07:32] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.83, 3.76, 3.14 [03:09:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.77, 3.09, 2.97 [03:10:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.80, 3.97, 3.55 [03:11:25] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [03:12:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 3.32, 3.36 [03:13:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.90, 4.37, 3.58 [03:17:06] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 11.90, 10.77, 7.10 [03:19:01] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.11, 8.58, 6.73 [03:19:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.26, 3.43, 3.49 [03:21:29] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.84, 3.22, 3.40 [03:23:39] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8747 MB (6% inode=99%); [03:28:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.29, 3.48, 3.00 [03:30:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.40, 3.91, 3.21 [03:30:09] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:32:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.60, 3.60, 3.20 [03:34:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.75, 3.00, 3.02 [03:56:25] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [04:25:21] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:39:02] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 16.69, 15.00, 9.14 [04:40:57] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 6.82, 11.92, 8.70 [04:42:52] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.11, 9.90, 8.33 [05:03:42] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [05:19:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.44, 3.25, 2.53 [05:21:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.06, 2.47, 2.34 [05:22:18] PROBLEM - teamballs.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'teamballs.polandball.wiki' expires in 15 day(s) (Fri 14 Jul 2023 05:18:35 GMT +0000). [05:22:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/298bdc7f17bb...d81de172a929 [05:22:45] [02miraheze/ssl] 07MirahezeSSLBot 03d81de17 - Bot: Update SSL cert for teamballs.polandball.wiki [05:32:25] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:33:57] PROBLEM - mercy.theking1543.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mercy.theking1543.eu.org' expires in 15 day(s) (Fri 14 Jul 2023 05:13:02 GMT +0000). [05:41:32] PROBLEM - raras.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'raras.polandball.wiki' expires in 15 day(s) (Fri 14 Jul 2023 05:12:19 GMT +0000). [05:43:39] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7962 MB (5% inode=99%); [05:49:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d81de172a929...aaaac15f18dc [05:49:13] [02miraheze/ssl] 07MirahezeSSLBot 03aaaac15 - Bot: Update SSL cert for raras.polandball.wiki [06:00:55] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 10.67, 12.86, 8.39 [06:01:10] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 16.35, 18.00, 10.82 [06:02:55] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.75, 9.72, 7.77 [06:03:01] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 40.07, 21.26, 10.29 [06:05:04] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 4.16, 10.48, 9.35 [06:07:00] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.14, 8.47, 8.75 [06:07:01] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 5.37, 11.94, 8.92 [06:09:00] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 3.09, 8.89, 8.16 [06:20:52] RECOVERY - teamballs.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'teamballs.polandball.wiki' will expire on Tue 26 Sep 2023 04:22:32 GMT +0000. [06:27:16] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [06:28:41] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.21, 3.49, 2.09 [06:28:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.44, 6.44, 4.27 [06:30:41] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.72, 2.59, 1.93 [06:31:43] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:32:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.50, 3.61, 3.58 [06:34:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.14, 3.66, 3.59 [06:35:08] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1170980334 secs [06:37:39] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8692 MB (6% inode=99%); [06:39:37] RECOVERY - raras.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'raras.polandball.wiki' will expire on Tue 26 Sep 2023 04:49:05 GMT +0000. [06:47:08] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08423879743 secs [06:52:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.06, 2.73, 3.92 [06:56:11] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:58:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.27, 2.10, 3.26 [07:22:26] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 9.98, 13.77, 8.96 [07:24:26] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 4.51, 10.41, 8.31 [07:26:26] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 3.65, 8.19, 7.75 [07:47:02] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 11.45, 15.26, 8.98 [07:47:03] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.25, 16.68, 9.97 [07:48:57] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 4.86, 11.49, 8.33 [07:50:52] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 3.47, 8.91, 7.75 [07:50:53] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 5.84, 10.67, 8.98 [07:52:49] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 3.84, 8.46, 8.38 [07:54:56] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:04:20] PROBLEM - es.countryhumans.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'es.countryhumans.polandball.wiki' expires in 15 day(s) (Fri 14 Jul 2023 08:01:18 GMT +0000). [08:07:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.93, 3.41, 2.37 [08:09:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/aaaac15f18dc...f1d08a98151e [08:09:08] [02miraheze/ssl] 07MirahezeSSLBot 03f1d08a9 - Bot: Update SSL cert for es.countryhumans.polandball.wiki [08:09:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.14, 3.31, 2.46 [08:12:14] PROBLEM - natiospherae.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'natiospherae.polandball.wiki' expires in 15 day(s) (Fri 14 Jul 2023 07:52:30 GMT +0000). [08:12:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f1d08a98151e...dd75282e3683 [08:12:49] [02miraheze/ssl] 07MirahezeSSLBot 03dd75282 - Bot: Update SSL cert for natiospherae.polandball.wiki [08:16:20] PROBLEM - ru.countryhumans.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ru.countryhumans.polandball.wiki' expires in 15 day(s) (Fri 14 Jul 2023 07:57:27 GMT +0000). [08:18:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dd75282e3683...60c941e98360 [08:18:22] [02miraheze/ssl] 07MirahezeSSLBot 0360c941e - Bot: Update SSL cert for ru.countryhumans.polandball.wiki [08:22:56] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:33:40] RECOVERY - es.countryhumans.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'es.countryhumans.polandball.wiki' will expire on Tue 26 Sep 2023 07:08:57 GMT +0000. [08:37:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.62, 2.58, 1.97 [08:39:06] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.74, 2.18, 1.89 [08:44:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.80, 4.16, 2.91 [08:45:17] PROBLEM - db131 Backups SQL on db131 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864151 seconds old and 1879 bytes [08:45:40] RECOVERY - ru.countryhumans.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ru.countryhumans.polandball.wiki' will expire on Tue 26 Sep 2023 07:18:14 GMT +0000. [08:46:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.35, 3.10, 2.67 [08:49:39] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7963 MB (5% inode=99%); [08:55:42] [02dns] 07jchristin closed pull request 03#419: Add TXT field for arru.xyz - 13https://github.com/miraheze/dns/pull/419 [08:56:25] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [09:08:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [09:09:00] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.75, 6.34, 3.70 [09:11:42] RECOVERY - natiospherae.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'natiospherae.polandball.wiki' will expire on Tue 26 Sep 2023 07:12:38 GMT +0000. [09:16:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.88, 3.99, 3.54 [09:18:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.51, 3.12, 3.27 [09:38:16] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [09:54:25] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:57:39] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8624 MB (6% inode=99%); [10:43:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 16.17, 9.78, 4.79 [10:50:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.76, 3.46, 3.73 [10:52:55] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 10.12, 10.90, 6.87 [10:54:24] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 18.44, 17.99, 10.08 [10:54:55] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.72, 8.59, 6.51 [10:54:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.01, 2.15, 3.13 [10:56:02] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 7.46, 12.00, 7.72 [10:58:02] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 3.74, 9.17, 7.21 [10:58:21] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 4.03, 10.25, 8.67 [11:00:21] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.75, 8.02, 8.04 [11:02:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/167d29fa7100...c34055bedbe6 [11:02:24] [02miraheze/puppet] 07paladox 03c34055b - Create matomo121.yaml [11:32:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/efd1718c568a [11:32:17] [02miraheze/puppet] 07paladox 03efd1718 - matomo: use systemd timer to run archiver [11:32:18] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [11:32:21] [02puppet] 07paladox opened pull request 03#3251: matomo: use systemd timer to run archiver - 13https://github.com/miraheze/puppet/pull/3251 [11:35:33] [02puppet] 07paladox synchronize pull request 03#3251: matomo: use systemd timer to run archiver - 13https://github.com/miraheze/puppet/pull/3251 [11:35:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/efd1718c568a...472983d93801 [11:35:38] [02miraheze/puppet] 07paladox 03472983d - Update init.pp [11:36:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.65, 3.59, 2.22 [11:37:08] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.181304127 secs [11:43:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/472983d93801...acff5c91e8dd [11:43:58] [02miraheze/puppet] 07paladox 03acff5c9 - Update init.pp [11:44:01] [02puppet] 07paladox synchronize pull request 03#3251: matomo: use systemd timer to run archiver - 13https://github.com/miraheze/puppet/pull/3251 [11:44:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.25, 3.66, 3.22 [11:46:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.78, 3.04, 3.04 [11:47:22] [02puppet] 07paladox closed pull request 03#3251: matomo: use systemd timer to run archiver - 13https://github.com/miraheze/puppet/pull/3251 [11:47:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c34055bedbe6...940fff039b03 [11:47:26] [02miraheze/puppet] 07paladox 03940fff0 - matomo: use systemd timer to run archiver (#3251) [11:47:29] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [11:47:31] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [11:53:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/940fff039b03...ab3cda31729d [11:53:31] [02miraheze/puppet] 07paladox 03ab3cda3 - db112: increase buffer pool to 8g [11:53:39] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 27466 MB (20% inode=99%); [11:55:08] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08152523637 secs [12:01:59] !log install prometheus-mysqld-exporter from debian backports (bullseye) on db112 [12:02:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:17:37] !log dist-upgrade db112 [12:17:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:19:08] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1116107702 secs [12:20:09] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2397 bytes in 0.082 second response time [12:20:38] PROBLEM - db112 MariaDB Connections on db112 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Connection failed: Connection refused [12:20:46] PROBLEM - db112 MariaDB on db112 is CRITICAL: Can't connect to MySQL server on 'db112.miraheze.org' (115) [12:21:27] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [12:21:29] RECOVERY - db112 APT on db112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:21:46] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 0.100 second response time [12:22:52] !log reboot db112 [12:22:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:16] Is phabricator down ? [12:24:18] I think paladox is working on that [12:24:38] RECOVERY - db112 MariaDB Connections on db112 is OK: OK connection usage: 0.4%Current connections: 2 [12:24:47] should be back now [12:25:11] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 36 Threads: 34 Questions: 25889 Slow queries: 474 Opens: 535 Open tables: 529 Queries per second avg: 719.138 [12:25:18] Thanks [12:25:27] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.092 second response time [12:25:46] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 20261 bytes in 0.221 second response time [12:26:09] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.224 second response time [12:31:08] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.08689925075 secs [12:36:21] !log (db121/db131/db141/db112): MariaDB [(none)]> set global table_open_cache = 50000; [12:36:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:36:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.42, 2.33, 1.61 [12:37:21] Paladox I noticed last night cloud11’s IPMI sensors alerted on icinga is that something to worry about? [12:37:29] ^ dunno why it didn't pickup the config [12:37:37] Zppix that's been going off since last year [12:38:05] Do we know the cause? [12:38:24] it's something that does need sorting (one of the ssds disks are faulty it seems or at least everytime we reboot, it works for a week or 2 then just stops working) [12:38:46] we have backups of AC now. [12:38:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.65, 3.28, 2.06 [12:40:40] Hm yeah… that should probably be looked into [12:40:58] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.65, 2.59, 1.95 [12:41:29] i have no idea why mariadb isn't picking up https://github.com/miraheze/puppet/blob/master/modules/mariadb/templates/config/mw.cnf.erb#L45 [12:41:37] rather having to set it manually [12:42:05] hmm [12:42:16] exiting and doing mysql again shows it as 16k??? [12:42:23] Well in your sal log you used _ but in that its -, so is - correct? [12:42:51] oh [12:42:54] ok it works manually [12:43:29] i've tried to use _ on db112 to see, didn't help it seems [12:44:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8e9cd4800394 [12:44:26] [02miraheze/puppet] 07paladox 038e9cd48 - mariadb: use underscores for 2 configs [12:44:28] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [12:44:34] [02puppet] 07paladox opened pull request 03#3252: mariadb: use underscores for 2 configs - 13https://github.com/miraheze/puppet/pull/3252 [12:49:42] [1/2] > Jun 28 12:47:35 db112 mariadbd[6252]: 2023-06-28 12:47:35 0 [Warning] Could not increase number of max_open_files to more than 32768 (request: 800540) [12:49:43] [2/2] > Jun 28 12:47:35 db112 mariadbd[6252]: 2023-06-28 12:47:35 0 [Warning] Changed limits: max_open_files: 32768 max_connections: 500 (was 500) table_cache: 16119 (was 50000) [12:49:43] hmm [12:51:56] Something else must be overriding the config you changed in puppet is my guess [12:52:10] Is there somehow two db configs being applied by puppet? [12:52:42] oh i found it [12:52:45] [1/2] # Number of files limit. previously [mysqld_safe] open-files-limit [12:52:45] [2/2] LimitNOFILE=32768 [12:52:52] in the mariadb systemd file [12:54:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b691a5979209 [12:54:29] [02miraheze/puppet] 07paladox 03b691a59 - mariadb: increase LimitNOFILE to 200000 [12:54:31] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [12:54:34] [02puppet] 07paladox opened pull request 03#3253: mariadb: increase LimitNOFILE to 200000 - 13https://github.com/miraheze/puppet/pull/3253 [12:56:06] works \o/ [12:59:36] [02puppet] 07paladox closed pull request 03#3253: mariadb: increase LimitNOFILE to 200000 - 13https://github.com/miraheze/puppet/pull/3253 [12:59:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ab3cda31729d...55c22f1a5eb4 [12:59:39] [02miraheze/puppet] 07paladox 0355c22f1 - mariadb: increase LimitNOFILE to 200000 (#3253) [12:59:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [12:59:45] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [13:01:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/55c22f1a5eb4...8c048951204d [13:01:14] [02miraheze/puppet] 07paladox 038c04895 - mariadb: use underscores for 2 configs (#3252) [13:01:15] [02puppet] 07paladox closed pull request 03#3252: mariadb: use underscores for 2 configs - 13https://github.com/miraheze/puppet/pull/3252 [13:01:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [13:01:18] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [13:02:09] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [13:06:18] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:08:35] !log db101: MariaDB [(none)]> set global table_open_cache = 50000; [13:08:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:35:36] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [13:37:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.23, 2.91, 2.13 [13:39:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.64, 3.44, 2.44 [13:41:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.91, 3.74, 2.63 [13:45:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.02, 3.11, 2.62 [13:57:22] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 15.16, 15.12, 9.55 [14:00:09] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:01:14] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.03, 10.03, 8.69 [14:03:32] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 15.39, 13.82, 7.63 [14:04:55] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:04:55] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.51, 11.76, 7.80 [14:06:56] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 8.60, 10.43, 7.79 [14:07:28] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 5.58, 9.79, 7.41 [14:08:55] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.52, 8.87, 7.56 [14:25:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/56b915d4a75f [14:25:53] [02miraheze/puppet] 07paladox 0356b915d - mariadb: innodb_flush_neighbors set to 0 [14:25:55] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [14:25:58] [02puppet] 07paladox opened pull request 03#3254: mariadb: innodb_flush_neighbors set to 0 - 13https://github.com/miraheze/puppet/pull/3254 [14:26:00] [02puppet] 07paladox closed pull request 03#3254: mariadb: innodb_flush_neighbors set to 0 - 13https://github.com/miraheze/puppet/pull/3254 [14:26:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [14:26:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8c048951204d...cd4489bf4794 [14:26:07] [02miraheze/puppet] 07paladox 03cd4489b - mariadb: innodb_flush_neighbors set to 0 (#3254) [14:26:10] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [14:27:08] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:28:53] !log db*: set innodb_flush_neighbors to 0 [14:28:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:31:40] !log dist-upgrade on prometheus131 [14:31:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:01] !log dist-upgrade on mon141 [14:32:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:41] RECOVERY - prometheus131 APT on prometheus131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:33:27] !log dist-upgrade on puppet141 [14:33:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:37:03] [Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:37:15] RECOVERY - puppet141 APT on puppet141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:37:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.82, 2.86, 2.02 [14:37:51] PROBLEM - prometheus131 Puppet on prometheus131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:38:30] [Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:39:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.47, 3.41, 2.32 [14:40:30] !log dist-upgrade on mail121 [14:40:33] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:40:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:40:46] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:40:48] PROBLEM - swiftproxy131 Puppet on swiftproxy131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:40:52] !log dist-upgrade on bast121 [14:41:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:41:01] PROBLEM - puppet141 puppetserver on puppet141 is CRITICAL: connect to address 2a10:6740::6:507 and port 8140: Connection refused [14:41:17] !log dist-upgrade on bast141 [14:41:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.66, 3.25, 2.41 [14:41:27] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Puppet has 399 failures. Last run 3 minutes ago with 399 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/smafallfest.org.key],File[/etc/ssl/localcerts/creativecommons.id.crt],File[/etc/ssl/private/creativecommons.id.key],File[/etc/ssl/localcerts/elsterclopedie.marithaime.nl.crt] [14:41:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:41:31] PROBLEM - os141 Puppet on os141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:41:39] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:41:45] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:41:51] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:41:55] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:42:16] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:42:28] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:43:30] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:43:36] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:43:38] !log dist-upgrade on phab121 [14:43:42] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:43:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:43:48] RECOVERY - bast121 APT on bast121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:43:56] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Puppet has 493 failures. Last run 2 minutes ago with 493 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/persist.wiki.key],File[/etc/ssl/localcerts/wiki.kachow.org.crt],File[/etc/ssl/private/wiki.kachow.org.key],File[/etc/ssl/localcerts/wiki.veloren.net.crt] [14:43:59] RECOVERY - bast141 APT on bast141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:44:08] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:44:21] PROBLEM - cp22 Puppet on cp22 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:44:59] RECOVERY - puppet141 puppetserver on puppet141 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:507 port 8140 [14:45:02] PROBLEM - swiftobject113 Puppet on swiftobject113 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:45:20] PROBLEM - mail121 Current Load on mail121 is WARNING: WARNING - load average: 1.76, 1.27, 0.63 [14:45:23] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:45:53] PROBLEM - mem141 Puppet on mem141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:46:16] RECOVERY - phab121 APT on phab121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:47:09] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.57, 1.42, 0.81 [14:47:19]