[00:00:17] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 10523MiB (11% inode=98%); [00:02:56] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 8936MiB (10% inode=98%); [00:04:56] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 22207MiB (25% inode=98%); [00:06:51] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 18779MiB (20% inode=98%); [00:24:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.69, 6.57, 6.17 [00:30:44] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.76, 6.76, 6.46 [00:31:46] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 21.44, 19.75 [00:33:45] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.73, 22.50, 20.39 [00:34:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.48, 7.86, 6.95 [00:35:44] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.45, 23.75, 21.11 [00:36:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.58, 7.63, 7.00 [00:39:43] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.17, 23.16, 21.56 [00:40:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.55, 8.14, 7.36 [00:41:42] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.10, 23.82, 21.98 [00:42:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.03, 7.73, 7.29 [00:43:41] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.54, 23.07, 21.92 [00:44:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.04, 7.99, 7.42 [00:46:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.28, 7.87, 7.46 [00:47:39] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.24, 22.34, 21.78 [00:47:41] PROBLEM - thelonsdalebattalion.co.uk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query thelonsdalebattalion.co.uk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:48:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.12, 7.68, 7.42 [00:50:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.43, 7.27, 7.31 [00:51:37] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.27, 22.18, 21.90 [00:52:16] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query private.yahyabd.xyz. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:54:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.22, 7.52, 7.37 [00:57:34] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.35, 22.86, 22.25 [00:58:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.24, 7.19, 7.28 [01:01:57] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 8.41, 19.20, 21.23 [01:03:57] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.25, 17.33, 20.23 [01:04:11] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:04:43] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.03, 6.05, 6.78 [01:07:54] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.50, 21.52, 21.37 [01:10:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.65, 7.64, 7.18 [01:14:18] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.yahyabd.xyz. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:15:50] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 12.17, 17.64, 19.84 [01:16:51] PROBLEM - thelonsdalebattalion.co.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - thelonsdalebattalion.co.uk All nameservers failed to answer the query. [01:18:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.97, 21.11, 23.48 [01:24:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.41, 7.50, 7.72 [01:26:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.51, 8.04, 7.89 [01:28:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.36, 7.41, 7.68 [01:30:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.31, 17.36, 20.15 [01:32:04] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [01:44:43] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.54, 6.23, 6.76 [01:46:00] RECOVERY - thelonsdalebattalion.co.uk - reverse DNS on sslhost is OK: SSL OK - thelonsdalebattalion.co.uk reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [01:51:35] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.36, 6.81, 6.93 [01:54:44] PROBLEM - webkinzguide.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - webkinzguide.com All nameservers failed to answer the query. [01:59:15] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.59, 6.59, 6.79 [02:03:07] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.35, 7.53, 7.16 [02:06:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.42, 20.22, 18.70 [02:06:57] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.81, 7.49, 7.17 [02:08:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.36, 19.98, 18.80 [02:08:52] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.96, 7.71, 7.28 [02:15:07] PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [02:20:43] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.09, 5.79, 6.60 [02:23:24] RECOVERY - webkinzguide.com - reverse DNS on sslhost is OK: SSL OK - webkinzguide.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [02:44:31] RECOVERY - antiguabarbudacalypso.com - reverse DNS on sslhost is OK: SSL OK - antiguabarbudacalypso.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [03:07:26] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.24, 7.40, 6.00 [03:09:22] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.19, 7.83, 6.31 [03:23:35] PROBLEM - mockgovernments.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mockgovernments.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:26:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.51, 7.25, 7.56 [03:34:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.51, 7.90, 7.71 [03:38:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.51, 7.78, 7.71 [03:52:27] RECOVERY - mockgovernments.com - reverse DNS on sslhost is OK: SSL OK - mockgovernments.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [03:52:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.31, 7.81, 7.44 [03:55:39] PROBLEM - roll-of-arms.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roll-of-arms.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:56:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.87, 7.83, 7.52 [04:06:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.11, 7.62, 7.39 [04:10:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.16, 6.86, 7.24 [04:15:53] PROBLEM - otherkin.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query otherkin.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:16:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.71, 9.10, 8.06 [04:20:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.37, 7.86, 7.77 [04:22:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.99, 8.46, 7.98 [04:25:10] RECOVERY - roll-of-arms.com - reverse DNS on sslhost is OK: SSL OK - roll-of-arms.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [04:36:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.00, 7.60, 7.95 [04:42:45] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.75, 8.30, 8.06 [04:44:35] RECOVERY - otherkin.wiki - reverse DNS on sslhost is OK: SSL OK - otherkin.wiki reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [04:47:05] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.72, 20.20, 16.81 [04:52:59] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.13, 22.24, 19.07 [04:56:55] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.37, 18.59, 18.37 [05:00:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.49, 7.32, 7.90 [05:02:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.72, 7.68, 7.95 [05:04:45] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.00, 7.03, 7.68 [05:08:13] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:18:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.05, 7.28, 7.28 [05:20:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.48, 7.20, 7.25 [05:24:43] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.84, 5.83, 6.66 [05:34:31] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:36:11] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [05:36:45] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.83, 6.86, 6.61 [05:40:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.81, 8.81, 7.38 [05:44:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.04, 21.36, 14.88 [05:46:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.12, 22.49, 16.14 [05:48:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 23.27, 17.21 [05:52:06] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.84, 21.96, 16.15 [05:53:34] PROBLEM - osfirsttimer.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query osfirsttimer.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:56:04] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.25, 24.11, 18.20 [05:57:43] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.47, 20.11, 15.51 [05:58:24] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.21, 21.88, 16.66 [05:59:40] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.36, 21.48, 16.54 [06:01:36] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.19, 20.29, 16.72 [06:03:45] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [06:10:24] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.89, 19.09, 18.12 [06:12:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:13:56] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.32, 23.67, 22.72 [06:23:27] RECOVERY - osfirsttimer.wiki - reverse DNS on sslhost is OK: SSL OK - osfirsttimer.wiki reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [06:25:00] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.93, 20.76, 18.57 [06:25:50] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.69, 21.89, 21.57 [06:26:56] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.29, 20.10, 18.57 [06:27:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.80, 21.92, 21.61 [06:31:48] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.59, 17.86, 19.93 [06:35:47] PROBLEM - burnout.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - burnout.wiki All nameservers failed to answer the query. [06:48:13] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.46, 21.86, 20.05 [06:50:12] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.17, 20.67, 19.79 [06:52:12] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.48, 19.60, 19.51 [06:57:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:02:29] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 11.92, 16.80, 23.02 [07:04:58] RECOVERY - burnout.wiki - reverse DNS on sslhost is OK: SSL OK - burnout.wiki reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [07:06:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 10.70, 13.68, 20.35 [07:18:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.47, 6.43, 7.70 [07:22:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.90, 7.86, 7.95 [07:32:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.23, 7.80, 8.00 [07:34:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.84, 8.65, 8.29 [07:58:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.53, 6.97, 7.81 [08:00:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.08, 7.24, 7.79 [08:02:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.83, 7.09, 7.66 [08:04:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 12.13, 8.72, 8.17 [08:11:47] PROBLEM - wiki.hacklabsr.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.hacklabsr.org All nameservers failed to answer the query. [08:32:46] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.55, 6.68, 7.69 [08:38:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.52, 7.55, 7.68 [08:40:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.33, 7.53, 7.66 [08:40:53] RECOVERY - wiki.hacklabsr.org - reverse DNS on sslhost is OK: SSL OK - wiki.hacklabsr.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:44:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.66, 8.03, 7.83 [09:01:56] PROBLEM - cp41 PowerDNS Recursor on cp41 is CRITICAL: No response from DNS ::1 [09:03:55] RECOVERY - cp41 PowerDNS Recursor on cp41 is OK: DNS OK: 2.272 seconds response time. wikitide.net returns 109.123.230.163,2400:d320:2161:9775::1 [09:30:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.69, 6.80, 7.83 [09:32:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.53, 7.26, 7.84 [09:34:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.76, 7.17, 7.75 [09:42:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.69, 7.95, 7.81 [09:46:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.43, 7.67, 7.73 [09:52:45] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.74, 7.58, 7.52 [09:58:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.27, 7.85, 7.72 [10:00:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.25, 8.22, 7.86 [10:02:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.46, 7.50, 7.64 [10:04:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.60, 8.34, 7.91 [10:30:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.83, 7.43, 7.90 [10:42:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.01, 9.07, 8.35 [10:52:45] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.22, 7.68, 7.95 [10:54:58] PROBLEM - agesofconflict.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query agesofconflict.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:56:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.29, 7.66, 7.84 [11:24:41] RECOVERY - agesofconflict.wiki - reverse DNS on sslhost is OK: SSL OK - agesofconflict.wiki reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [11:56:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.99, 19.39, 14.96 [12:02:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.35, 18.84, 16.32 [12:08:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.42, 22.93, 18.81 [12:12:11] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query. [12:12:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.67, 23.05, 19.88 [12:14:41] PROBLEM - metroidpedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - metroidpedia.com All nameservers failed to answer the query. [12:14:48] PROBLEM - wiki.yahyabd.xyz - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:15:08] PROBLEM - private.yahyabd.xyz - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:16:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.95, 24.71, 21.22 [12:19:38] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:20:07] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [12:26:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 8.28, 20.20, 21.45 [12:28:03] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 22.69, 34.04, 15.78 [12:30:14] PROBLEM - ncuwiki.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query ncuwiki.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:32:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.77, 22.19, 21.55 [12:33:45] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.51, 11.22, 11.08 [12:35:39] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.28, 7.73, 9.82 [12:43:48] RECOVERY - metroidpedia.com - reverse DNS on sslhost is OK: SSL OK - metroidpedia.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [12:53:21] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.76, 20.54, 18.48 [12:54:53] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10207MiB (10% inode=98%); [12:55:18] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.13, 20.55, 18.74 [12:57:14] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.37, 20.01, 18.73 [12:59:49] PROBLEM - ncuwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - ncuwiki.com All nameservers failed to answer the query. [13:04:41] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:08:14] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:08:31] [02miraheze/mediawiki-repos] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/d70ff5066852...e13df8c6c226 [13:08:32] [02miraheze/mediawiki-repos] 07AgentIsai 03e13df8c - T12059: Install MultiPurge [13:10:00] PROBLEM - www.tintinpedia.fr - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query tintinpedia.fr. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:12:52] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.45, 21.40, 19.98 [13:14:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:14:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.88, 20.17, 19.71 [13:18:05] PROBLEM - thelonsdalebattalion.co.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - thelonsdalebattalion.co.uk All nameservers failed to answer the query. [13:21:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:21:29] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.79 ms [13:21:46] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.41, 21.69, 20.63 [13:24:18] PROBLEM - wiki.site97.fr - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.site97.fr All nameservers failed to answer the query. [13:25:32] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.54 ms [13:29:20] RECOVERY - ncuwiki.com - reverse DNS on sslhost is OK: SSL OK - ncuwiki.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [13:31:29] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.22, 19.40, 20.24 [13:33:58] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [13:36:10] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:38:50] RECOVERY - www.tintinpedia.fr - reverse DNS on sslhost is OK: SSL OK - www.tintinpedia.fr reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [13:41:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:45:40] PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [13:47:16] PROBLEM - thelonsdalebattalion.co.uk - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [13:48:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 13.22, 19.12, 23.29 [13:48:52] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.31 ms [13:51:48] PROBLEM - worldtriggerwiki.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query worldtriggerwiki.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:52:55] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.26 ms [13:53:35] RECOVERY - wiki.site97.fr - reverse DNS on sslhost is OK: SSL OK - wiki.site97.fr reverse DNS resolves to cp37.wikitide.net - CNAME OK [13:54:05] !log [@test151] starting deploy of {'config': True} to test151 [13:54:06] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [13:54:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 10.72, 13.88, 19.54 [14:03:14] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [14:12:42] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [14:15:04] RECOVERY - antiguabarbudacalypso.com - reverse DNS on sslhost is OK: SSL OK - antiguabarbudacalypso.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [14:16:25] RECOVERY - thelonsdalebattalion.co.uk - reverse DNS on sslhost is OK: SSL OK - thelonsdalebattalion.co.uk reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [14:20:31] RECOVERY - worldtriggerwiki.com - reverse DNS on sslhost is OK: SSL OK - worldtriggerwiki.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [14:47:32] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [14:47:42] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.24 ms [14:49:22] PROBLEM - dragonquestwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dragonquestwiki.com All nameservers failed to answer the query. [14:49:43] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.26 ms [15:02:16] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:10:34] PROBLEM - fallofsanctuary.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.409 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [15:18:32] RECOVERY - dragonquestwiki.com - reverse DNS on sslhost is OK: SSL OK - dragonquestwiki.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [15:22:15] PROBLEM - familiacorsi.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - familiacorsi.com All nameservers failed to answer the query. [15:22:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:30:45] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.25, 7.28, 8.00 [15:32:13] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [15:40:26] RECOVERY - fallofsanctuary.com - reverse DNS on sslhost is OK: SSL OK - fallofsanctuary.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [15:42:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:46:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.46, 7.76, 7.61 [15:50:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.06, 7.42, 7.54 [15:52:08] RECOVERY - familiacorsi.com - reverse DNS on sslhost is OK: SSL OK - familiacorsi.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [15:52:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.17, 7.29, 7.43 [15:56:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.82, 6.80, 7.18 [16:00:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.18, 7.50, 7.37 [16:02:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.31, 7.07, 7.23 [16:10:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.76, 7.89, 7.52 [16:12:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.75, 7.49, 7.43 [16:20:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.63, 8.14, 7.63 [16:26:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.60, 7.46, 7.59 [16:33:54] PROBLEM - wizardia.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wizardia.wiki All nameservers failed to answer the query. [16:34:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.68, 7.90, 7.66 [16:36:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.66, 7.82, 7.66 [16:38:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.10, 7.82, 7.67 [16:40:45] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.63, 7.60, 7.60 [16:44:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.59, 8.57, 7.98 [16:47:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:50:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.31, 7.94, 7.93 [16:54:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.22, 7.82, 7.86 [16:56:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.51, 7.47, 7.71 [17:00:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.37, 7.85, 7.75 [17:02:35] RECOVERY - wizardia.wiki - reverse DNS on sslhost is OK: SSL OK - wizardia.wiki reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [17:04:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.00, 7.96, 7.85 [17:07:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:08:13] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:09:47] PROBLEM - evilgeniuswiki.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query evilgeniuswiki.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:14:43] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.10, 5.62, 6.66 [17:15:00] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/e13df8c6c226...ff047200def9 [17:15:03] [02miraheze/mediawiki-repos] 07Universal-Omega 03ff04720 - Fix MultiPurge [17:16:05] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.806B045F0D150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1586:SSL alert number 40 [17:28:05] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Fri 19 Jul 2024 04:26:37 PM GMT +0000. [17:32:47] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:36:10] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [18:01:18] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:07:36] RECOVERY - evilgeniuswiki.com - reverse DNS on sslhost is OK: SSL OK - evilgeniuswiki.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [18:08:12] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:36:10] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:55:45] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.82, 20.49, 18.30 [18:59:41] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.04, 18.95, 18.23 [19:10:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.17, 21.10, 19.32 [19:16:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.59, 19.18, 19.26 [19:54:07] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 197.53 ms [19:56:08] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.00 ms [20:08:12] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:15:11] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03multipurge [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f17bc0428b19 [20:15:12] [02miraheze/mw-config] 07AgentIsai 03f17bc04 - T12059: Enable MultiPurge on all wikis [20:15:14] [02mw-config] 07AgentIsai created branch 03multipurge - 13https://github.com/miraheze/mw-config [20:15:32] [02mw-config] 07AgentIsai opened pull request 03#5542: T12059: Prepare for CloudFlare CDN enabling - 13https://github.com/miraheze/mw-config/pull/5542 [20:16:35] miraheze/mw-config - AgentIsai the build passed. [20:20:17] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9702MiB (10% inode=98%); [20:22:44] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03multipurge [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f17bc0428b19...dd438d396722 [20:22:46] [02miraheze/mw-config] 07AgentIsai 03dd438d3 - Add further CloudFlare configurations [20:22:49] [02mw-config] 07AgentIsai synchronize pull request 03#5542: T12059: Prepare for CloudFlare CDN enabling - 13https://github.com/miraheze/mw-config/pull/5542 [20:23:37] miraheze/mw-config - AgentIsai the build passed. [20:27:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:27:59] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03multipurge [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e1db4818739d [20:28:00] [02miraheze/puppet] 07AgentIsai 03e1db481 - T12059: Add MultiPurge configurations to PrivateSettings [20:28:01] [02puppet] 07AgentIsai created branch 03multipurge - 13https://github.com/miraheze/puppet [20:28:05] [02puppet] 07AgentIsai opened pull request 03#3837: T12059: Add MultiPurge configurations to PrivateSettings - 13https://github.com/miraheze/puppet/pull/3837 [20:30:40] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03multipurge [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e1db4818739d...80a8b2325185 [20:30:43] [02miraheze/puppet] 07AgentIsai 0380a8b23 - Add lookups for values for MultiPurge [20:30:45] [02puppet] 07AgentIsai synchronize pull request 03#3837: T12059: Add MultiPurge configurations to PrivateSettings - 13https://github.com/miraheze/puppet/pull/3837 [20:32:43] [02puppet] 07AgentIsai closed pull request 03#3837: T12059: Add MultiPurge configurations to PrivateSettings - 13https://github.com/miraheze/puppet/pull/3837 [20:32:45] [02miraheze/puppet] 07AgentIsai deleted branch 03multipurge [20:32:46] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/e7c7218c52a7...2a55fd47052b [20:32:49] [02miraheze/puppet] 07AgentIsai 032a55fd4 - T12059: Add MultiPurge configurations to PrivateSettings (#3837) [20:32:51] [02puppet] 07AgentIsai deleted branch 03multipurge - 13https://github.com/miraheze/puppet [20:36:09] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:39:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.47, 22.74, 18.35 [20:41:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.91, 22.87, 18.96 [20:44:35] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03cloudflare [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7cda7d3587a4 [20:44:37] [02miraheze/puppet] 07AgentIsai 037cda7d3 - Add CloudFlare IPs to firewall [20:44:39] [02puppet] 07AgentIsai created branch 03cloudflare - 13https://github.com/miraheze/puppet [20:44:42] [02puppet] 07AgentIsai opened pull request 03#3838: Add CloudFlare IPs to firewall - 13https://github.com/miraheze/puppet/pull/3838 [20:45:12] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03cloudflare [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7cda7d3587a4...4ea94c268e25 [20:45:14] [02miraheze/puppet] 07github-actions 034ea94c2 - CI: lint puppet code to standards [20:45:17] [02puppet] 07github-actions[bot] synchronize pull request 03#3838: Add CloudFlare IPs to firewall - 13https://github.com/miraheze/puppet/pull/3838 [20:46:10] [02puppet] 07AgentIsai closed pull request 03#3838: Add CloudFlare IPs to firewall - 13https://github.com/miraheze/puppet/pull/3838 [20:46:11] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2a55fd47052b...a7f4d046db55 [20:46:13] [02miraheze/puppet] 07AgentIsai 03a7f4d04 - Add CloudFlare IPs to firewall (#3838) [20:46:15] [02puppet] 07AgentIsai deleted branch 03cloudflare - 13https://github.com/miraheze/puppet [20:46:16] [02miraheze/puppet] 07AgentIsai deleted branch 03cloudflare [20:47:48] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:49:36] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:51:00] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.72, 20.30, 19.37 [20:52:47] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:53:39] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a7f4d046db55...304fead67afd [20:53:42] [02miraheze/puppet] 07AgentIsai 03304fead - Remove duplicate [20:54:55] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.87, 21.49, 20.05 [20:55:03] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:55:18] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:55:35] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:55:50] [02mw-config] 07AgentIsai closed pull request 03#5542: T12059: Prepare for CloudFlare CDN enabling - 13https://github.com/miraheze/mw-config/pull/5542 [20:55:53] [02miraheze/mw-config] 07AgentIsai deleted branch 03multipurge [20:55:54] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/d79ee04ba748...6c93ceace236 [20:55:56] [02miraheze/mw-config] 07AgentIsai 036c93cea - T12059: Prepare for CloudFlare CDN enabling (#5542) [20:55:58] [02mw-config] 07AgentIsai deleted branch 03multipurge - 13https://github.com/miraheze/mw-config [20:56:44] miraheze/mw-config - AgentIsai the build passed. [20:57:03] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:57:12] !log [@mwtask171] starting deploy of {'config': True} to all [20:57:13] !log [@mwtask171] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [20:57:15] !log [@mwtask181] starting deploy of {'config': True} to all [20:57:16] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [20:57:18] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:57:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:34] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [20:57:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:48] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:57:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:51] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:58:51] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.38, 22.47, 20.73 [20:59:19] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [20:59:20] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [21:00:34] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:00:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:46] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [21:00:49] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.06, 22.53, 21.00 [21:00:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:01:06] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [21:01:19] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.064 second response time [21:01:20] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.390 second response time [21:01:35] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [21:03:07] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:04:06] PROBLEM - largedu.eu.org - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [21:04:07] PROBLEM - wiki.walkscape.app - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.walkscape.app could not be found [21:04:09] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [21:04:10] PROBLEM - wiki.potabi.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.potabi.com could not be found [21:04:11] PROBLEM - celeste.ink - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [21:06:21] [02puppet] 07Universal-Omega reviewed pull request 03#3837 commit - 13https://github.com/miraheze/puppet/pull/3837#discussion_r1573402288 [21:06:35] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.28, 20.85, 16.41 [21:06:43] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.12, 24.54, 22.36 [21:07:03] PROBLEM - pokemon.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pokemon.miraheze.org could not be found [21:07:05] PROBLEM - projectsekai.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for projectsekai.miraheze.org could not be found [21:07:35] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:08:01] PROBLEM - squareenix.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for squareenix.miraheze.org could not be found [21:08:04] PROBLEM - wiki.cosmicdeathfungus.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cosmicdeathfungus.com could not be found [21:08:12] PROBLEM - commonswiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for commonswiki.miraheze.org could not be found [21:08:39] PROBLEM - donkeykong.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for donkeykong.miraheze.org could not be found [21:08:39] PROBLEM - smashbros.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for smashbros.miraheze.org could not be found [21:08:39] PROBLEM - yoshi.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for yoshi.miraheze.org could not be found [21:08:45] PROBLEM - status.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for status.miraheze.org could not be found [21:09:26] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.91, 23.40, 18.31 [21:10:40] PROBLEM - wiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.miraheze.org could not be found [21:10:44] PROBLEM - patrykstan.veles-crimea.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for patrykstan.veles-crimea.ru could not be found [21:10:55] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.15, 21.20, 15.69 [21:10:56] PROBLEM - schizoidnightmares.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for schizoidnightmares.miraheze.org could not be found [21:11:06] PROBLEM - 321nailswiki.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for 321nailswiki.miraheze.org could not be found [21:11:21] PROBLEM - wiki.jerryly.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jerryly.top could not be found [21:11:53] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.02, 20.33, 13.84 [21:12:01] PROBLEM - nintendo.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nintendo.miraheze.org could not be found [21:13:12] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.02, 22.84, 16.75 [21:13:25] PROBLEM - polcompball.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for polcompball.miraheze.org could not be found [21:13:26] PROBLEM - rare.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rare.miraheze.org could not be found [21:13:26] PROBLEM - wiki.autocountsoft.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.autocountsoft.com could not be found [21:14:02] PROBLEM - kaylaandrachel.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kaylaandrachel.miraheze.org could not be found [21:14:33] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gab.pt.eu.org could not be found [21:14:47] PROBLEM - phabricator.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phabricator.miraheze.org could not be found [21:14:47] PROBLEM - wiki.rlab.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.807B0E7827150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1586:SSL alert number 40 [21:17:05] PROBLEM - dragonquest2.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dragonquest2.miraheze.org could not be found [21:17:28] PROBLEM - closinglogosgroup.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for closinglogosgroup.miraheze.org could not be found [21:17:31] PROBLEM - piwik.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for piwik.miraheze.org could not be found [21:18:07] PROBLEM - sims.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sims.miraheze.org could not be found [21:18:07] PROBLEM - wiki.jerryly.top - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80AB417844150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1586:SSL alert number 40 [21:18:32] PROBLEM - ajhalili2006.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ajhalili2006.miraheze.org could not be found [21:18:52] PROBLEM - wiki.worldx.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80CBD94F11150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1586:SSL alert number 40 [21:20:54] PROBLEM - mario.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mario.miraheze.org could not be found [21:20:54] PROBLEM - phorge.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for phorge.miraheze.org could not be found [21:21:10] PROBLEM - wiki.rlab.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rlab.org.uk could not be found [21:22:01] PROBLEM - sega.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sega.miraheze.org could not be found [21:22:11] PROBLEM - tintinpedia.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tintinpedia.miraheze.org could not be found [21:23:07] PROBLEM - croc.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for croc.miraheze.org could not be found [21:23:47] !log [@test151] starting deploy of {'config': True} to test151 [21:23:48] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [21:23:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:13]