[00:03:58] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.09, 19.82, 14.86 [00:05:58] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.16, 18.52, 15.03 [00:17:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:57:58] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.64, 20.61, 16.24 [00:59:58] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.49, 19.99, 16.57 [01:00:58] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.35, 22.68, 15.85 [01:02:58] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.08, 19.86, 15.68 [01:03:58] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 16.86, 21.03, 17.99 [01:05:58] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.24, 19.98, 17.98 [01:07:41] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.49, 21.81, 16.95 [01:08:58] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.57, 23.29, 18.28 [01:11:33] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.51, 21.86, 17.91 [01:11:58] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.16, 20.10, 18.45 [01:12:58] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.49, 21.26, 18.84 [01:13:29] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.76, 21.42, 18.25 [01:14:36] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.57, 22.44, 16.96 [01:14:58] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.50, 23.91, 20.11 [01:15:24] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.66, 23.68, 19.43 [01:15:58] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.69, 22.57, 19.69 [01:16:34] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.42, 21.80, 17.44 [01:18:32] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.94, 20.25, 17.40 [01:19:58] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.93, 23.33, 20.75 [01:20:58] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.24, 24.00, 21.52 [01:21:58] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.95, 25.43, 21.83 [01:22:26] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 22.23, 18.75 [01:22:58] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.32, 25.49, 22.35 [01:23:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.66, 21.08, 16.68 [01:24:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:24:24] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.59, 23.41, 19.66 [01:25:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.42, 23.66, 18.15 [01:27:07] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.49, 20.30, 15.69 [01:27:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.50, 22.17, 18.33 [01:28:19] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.04, 24.84, 20.94 [01:29:03] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.97, 22.43, 18.02 [01:29:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.53, 24.09, 19.50 [01:30:19] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.52, 22.91, 18.34 [01:31:01] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.79, 23.08, 18.73 [01:33:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.28, 22.54, 18.45 [01:39:02] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.33, 25.25, 20.92 [01:57:07] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:59:45] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 14.18, 20.53, 23.73 [02:00:16] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 12.54, 20.01, 23.52 [02:01:41] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 13.57, 20.57, 23.42 [02:01:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 11.51, 18.78, 22.53 [02:02:52] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 15.29, 19.97, 23.38 [02:05:38] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 10.24, 14.53, 20.04 [02:05:49] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.87, 15.70, 20.39 [02:07:34] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 10.55, 14.31, 19.70 [02:07:36] PROBLEM - aryavartpedia.online - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aryavartpedia.online' expires in 15 day(s) (Wed 12 Jun 2024 01:45:35 AM GMT +0000). [02:07:47] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ca42a0ad23af...bf5372285d20 [02:07:50] [02ssl] 07WikiTideSSLBot 03bf53722 - Bot: Update SSL cert for aryavartpedia.online [02:08:04] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 12.21, 15.36, 19.99 [02:08:49] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 8.66, 13.54, 19.46 [02:09:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:10:56] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 11.60, 15.58, 22.69 [02:10:58] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 13.64, 17.75, 22.85 [02:11:58] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 12.59, 17.60, 23.20 [02:14:56] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 10.40, 13.33, 20.24 [02:14:58] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 10.96, 14.08, 20.17 [02:19:58] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.30, 15.24, 19.68 [02:26:31] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:35:58] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.83, 19.01, 18.18 [02:37:20] RECOVERY - aryavartpedia.online - LetsEncrypt on sslhost is OK: OK - Certificate 'aryavartpedia.online' will expire on Sun 25 Aug 2024 01:07:41 AM GMT +0000. [02:37:58] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.33, 18.43, 18.04 [02:38:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.53, 20.68, 18.80 [02:38:56] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.50, 19.28, 17.45 [02:40:33] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.91, 18.41, 18.22 [02:42:56] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 9.27, 16.64, 17.10 [03:26:56] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.46, 18.67, 14.00 [03:27:58] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.39, 21.48, 16.38 [03:29:38] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.76, 17.73, 13.60 [03:30:56] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.70, 18.66, 15.02 [03:33:58] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.71, 23.04, 18.84 [03:34:56] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.29, 22.05, 17.22 [03:35:32] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.01, 22.27, 16.78 [03:35:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.34, 19.37, 13.84 [03:36:56] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.92, 22.85, 18.09 [03:38:18] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 29.13, 21.43, 15.44 [03:39:48] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.22, 18.03, 14.62 [03:39:58] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.83, 25.06, 21.05 [03:42:13] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.70, 18.97, 15.88 [03:42:56] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.17, 22.47, 19.68 [03:44:43] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.14, 22.92, 17.04 [03:44:56] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.50, 24.80, 20.87 [03:51:55] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.31, 21.65, 18.13 [03:54:37] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.85, 23.66, 18.62 [03:55:59] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.63, 22.43, 17.96 [03:58:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:04:21] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.74, 23.56, 22.91 [04:06:04] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.89, 23.14, 21.76 [04:06:19] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.56, 24.05, 23.10 [04:07:00] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.16, 23.37, 19.89 [04:07:54] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.14, 23.39, 21.46 [04:09:52] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.43, 23.25, 21.88 [04:09:53] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.23, 23.71, 21.78 [04:49:34] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 2 backends are down. mw181 mw182 [04:51:34] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [05:02:44] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw172 [05:04:44] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [05:14:15] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw182 [05:20:07] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [05:26:23] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:44:18] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw162 [05:46:14] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [06:04:11] PROBLEM - wiki.barengreza.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.barengreza.my.id All nameservers failed to answer the query. [06:18:00] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw172 [06:19:58] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [06:54:52] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [07:06:00] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw161 [07:07:56] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [07:17:46] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw181 [07:21:35] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [07:27:20] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [07:28:07] PROBLEM - crustypedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'crustypedia.org' expires in 15 day(s) (Wed 12 Jun 2024 07:03:50 AM GMT +0000). [07:28:19] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bf5372285d20...af69430c985c [07:28:22] [02ssl] 07WikiTideSSLBot 03af69430 - Bot: Update SSL cert for crustypedia.org [07:29:19] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [07:32:48] PROBLEM - wiki.barengreza.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.barengreza.my.id could not be found [07:56:14] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.34, 18.87, 23.17 [07:57:14] RECOVERY - crustypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'crustypedia.org' will expire on Sun 25 Aug 2024 06:28:13 AM GMT +0000. [07:58:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:01:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.36, 18.90, 23.16 [08:03:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.24, 21.70, 23.66 [08:04:11] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 21.86, 22.47 [08:05:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.42, 19.52, 22.58 [08:06:10] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 16.91, 20.21, 21.83 [08:12:08] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.12, 17.81, 20.25 [08:16:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.70, 23.24, 21.88 [08:18:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 16.94, 19.81, 20.73 [08:19:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.41, 23.02, 21.88 [08:22:05] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 12.31, 18.19, 20.10 [08:26:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.95, 21.08, 20.82 [08:29:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.12, 23.16, 23.19 [08:30:05] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.02, 18.97, 20.04 [08:34:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.60, 24.87, 22.22 [08:35:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.91, 24.60, 23.53 [08:40:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 18.06, 21.01, 23.55 [08:42:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.31, 23.43, 24.12 [08:59:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 13.57, 19.10, 23.47 [09:01:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.01, 20.63, 23.47 [09:03:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.52, 19.61, 22.80 [09:09:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.59, 23.38, 23.29 [09:25:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.75, 21.12, 23.30 [09:29:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 22.26, 23.18 [10:05:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 13.33, 19.45, 23.16 [10:07:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.50, 22.13, 23.66 [10:09:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.68, 20.70, 22.95 [10:09:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.87, 21.73, 23.88 [10:11:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.78, 23.16, 24.16 [10:19:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.45, 21.99, 23.41 [10:21:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.40, 23.59, 23.82 [10:22:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.35, 22.86, 23.99 [10:22:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.43, 21.04, 23.61 [10:23:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.28, 20.98, 21.54 [10:24:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.47, 24.36, 24.43 [10:25:33] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.73, 22.40, 23.70 [10:25:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.52, 19.31, 20.88 [10:25:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.03, 21.76, 23.21 [10:26:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.20, 23.00, 23.95 [10:26:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.75, 21.65, 23.12 [10:28:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.70, 22.45, 23.25 [10:29:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.65, 22.68, 23.15 [10:31:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.89, 22.08, 22.90 [10:32:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.04, 22.76, 23.04 [10:34:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.96, 21.70, 22.45 [10:34:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.07, 21.49, 22.53 [10:35:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.27, 23.56, 23.27 [10:36:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.78, 21.69, 22.36 [10:36:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.35, 24.14, 23.38 [10:37:33] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.95, 22.41, 22.47 [10:37:59] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [10:38:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 21.66, 22.19 [10:39:34] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.02, 17.43, 19.67 [10:41:33] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.68, 23.72, 23.18 [10:42:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.19, 23.31, 22.82 [10:43:33] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.79, 24.87, 23.68 [10:46:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 18.93, 22.22, 23.39 [10:47:48] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.53, 23.17, 23.74 [10:49:33] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.96, 23.60, 23.53 [10:51:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.23, 23.14, 23.48 [10:53:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.54, 21.44, 22.76 [10:54:05] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.54, 17.14, 19.77 [10:54:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 22.41, 22.63 [10:56:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.53, 22.69, 22.73 [10:59:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.44, 21.51, 22.14 [11:00:59] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 22.40, 22.54 [11:01:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.95, 22.07, 20.17 [11:01:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.05, 21.56, 22.06 [11:02:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.54, 18.74, 19.11 [11:03:33] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.67, 17.15, 20.00 [11:03:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.78, 22.55, 20.65 [11:07:33] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.53, 22.88, 21.64 [11:07:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.44, 22.23, 21.91 [11:08:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.50, 21.63, 20.28 [11:10:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.85, 20.99, 20.23 [11:11:33] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.81, 23.63, 22.40 [11:11:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.12, 23.46, 22.66 [11:12:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.08, 22.92, 20.99 [11:14:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.38, 21.28, 20.65 [11:14:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.99, 23.35, 23.60 [11:15:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.54, 23.72, 22.90 [11:16:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.96, 22.33, 21.08 [11:16:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.61, 24.11, 23.84 [11:17:33] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.18, 23.04, 22.39 [11:17:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.25, 22.84, 21.76 [11:17:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.75, 21.95, 22.42 [11:18:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.72, 22.49, 23.30 [11:19:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.00, 20.24, 20.91 [11:19:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.02, 22.67, 22.57 [11:20:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 23.18, 23.45 [11:21:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.91, 22.10, 21.50 [11:22:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.07, 23.94, 23.75 [11:23:33] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 17.76, 23.25, 23.06 [11:23:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.16, 19.83, 20.78 [11:23:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.95, 22.52, 22.60 [11:24:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.73, 23.35, 22.29 [11:24:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.43, 25.04, 24.15 [11:27:34] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.01, 18.86, 20.22 [11:28:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.71, 23.87, 23.98 [11:30:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.18, 23.70, 23.87 [11:31:33] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.42, 22.34, 22.47 [11:31:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.14, 20.25, 20.55 [11:32:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.94, 23.33, 23.77 [11:33:33] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.24, 22.23, 22.44 [11:33:48] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.18, 22.62, 22.36 [11:34:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 22.76, 23.43 [11:36:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.75, 22.94, 22.15 [11:36:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.09, 22.81, 23.42 [11:37:33] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.27, 22.94, 22.56 [11:40:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.59, 23.02, 22.51 [11:42:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 23.73, 24.12, 23.01 [11:43:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.67, 23.21, 23.52 [11:44:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 16.09, 21.06, 22.02 [11:45:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 23.33, 23.47 [11:46:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.88, 21.75, 22.08 [11:48:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.45, 20.52, 21.63 [11:48:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.94, 22.35, 22.60 [11:49:34] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 17.37, 19.31, 20.26 [11:51:48] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.42, 21.98, 23.14 [11:52:27] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [11:53:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.65, 20.01, 20.41 [11:53:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.45, 23.77, 23.65 [11:54:05] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.08, 17.82, 20.08 [11:55:34] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.32, 18.87, 19.91 [11:55:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.89, 23.52, 23.57 [11:58:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.94, 20.82, 20.88 [11:58:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:59:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.56, 21.73, 20.95 [11:59:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.09, 24.35, 23.84 [12:01:33] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.84, 22.93, 23.82 [12:01:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.18, 22.43, 21.26 [12:02:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.33, 23.92, 23.89 [12:03:33] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.33, 24.77, 24.40 [12:03:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.84, 23.20, 21.73 [12:04:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.04, 21.48, 21.03 [12:04:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.84, 23.87, 23.85 [12:05:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.31, 24.30, 22.28 [12:06:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.00, 20.28, 20.59 [12:06:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.99, 23.80, 23.85 [12:08:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.74, 22.48, 21.34 [12:08:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.65, 25.37, 24.40 [12:12:44] [02ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/6917022422b7...0b6eb659a17b [12:12:46] [02ImportDump] 07translatewiki 030b6eb65 - Localisation updates from https://translatewiki.net. [12:12:47] [02RequestSSL] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/compare/3c711da73bcb...ddf329b5e1f4 [12:12:49] [02RequestSSL] 07translatewiki 03ddf329b - Localisation updates from https://translatewiki.net. [12:12:51] [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/f6afacc445f2...edc3cf64494c [12:12:52] [02MirahezeMagic] 07translatewiki 03edc3cf6 - Localisation updates from https://translatewiki.net. [12:16:27] miraheze/MirahezeMagic - translatewiki the build has errored. [12:19:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.02, 22.14, 23.40 [12:20:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.72, 23.79, 23.29 [12:21:17] miraheze/ImportDump - translatewiki the build passed. [12:21:49] miraheze/RequestSSL - translatewiki the build passed. [12:21:56] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [12:22:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.64, 23.87, 23.32 [12:23:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.31, 24.47, 23.95 [12:23:46] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:23:47] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:23:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.99, 20.37, 23.21 [12:23:58] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:23:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:59] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:13] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [12:24:14] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.95, 22.83, 23.98 [12:25:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:25:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:25:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.70, 23.34, 23.61 [12:25:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:26:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.52, 23.15, 23.94 [12:31:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.84, 22.92, 23.06 [12:31:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.90, 21.95, 22.71 [12:32:50] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:33:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:38] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 48s [12:33:53] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:34:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:15] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 21s [12:34:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:35:33] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.06, 22.49, 23.90 [12:37:33] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.77, 23.48, 24.07 [12:37:37] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:38:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:15] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 38s [12:38:28] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:38:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:58] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 30s [12:39:04] [1/2] `{'folders': '1.41/skins/Citizen'}` when- [12:39:04] [2/2] /just me being me tho [12:39:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:39:38] Never [12:39:44] Muhahaha [12:51:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.37, 22.62, 23.56 [12:53:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.45, 25.23, 24.41 [13:11:19] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [13:15:19] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [13:27:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 13.31, 19.68, 23.57 [13:33:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.03, 20.17, 22.17 [13:39:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.24, 22.09, 22.63 [13:41:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.66, 22.41, 22.62 [13:43:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.94, 23.66, 23.10 [13:45:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.60, 24.23, 23.38 [13:45:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.14, 21.37, 23.58 [13:47:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.17, 23.82, 24.20 [14:01:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.60, 22.21, 23.30 [14:04:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.83, 21.98, 23.40 [14:05:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.75, 21.49, 22.57 [14:07:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.46, 19.10, 21.55 [14:08:05] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.11, 23.94, 23.74 [14:09:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.65, 22.25, 22.38 [14:15:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 14.89, 21.29, 22.31 [14:17:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.37, 23.21, 22.87 [14:19:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.01, 22.24, 22.58 [14:23:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.25, 22.75, 22.54 [14:24:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.57, 21.67, 23.72 [14:26:54] !log block some scrapers [14:26:58] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.49, 22.80, 23.85 [14:27:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 14.30, 19.63, 21.39 [14:28:58] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 10.66, 18.11, 22.01 [14:29:34] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 7.60, 15.52, 19.69 [14:29:58] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 7.68, 16.09, 22.09 [14:30:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 6.77, 17.44, 22.22 [14:30:58] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 8.08, 14.75, 20.31 [14:31:33] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 7.44, 16.83, 22.99 [14:31:49] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 7.40, 15.81, 22.05 [14:31:58] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 6.78, 12.80, 20.14 [14:32:05] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 5.71, 13.50, 20.19 [14:32:56] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 8.61, 15.91, 23.38 [14:32:58] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 7.37, 14.87, 22.84 [14:33:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:33:49] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 7.04, 12.76, 20.17 [14:35:33] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 7.42, 11.58, 19.41 [14:36:56] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 9.97, 12.54, 20.27 [14:36:58] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 8.00, 10.94, 19.38 [14:38:08] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [14:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:52:35] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:54:25] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=holidayswiki --procs=12 --memory-limit=max (START) [14:54:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:00:46] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=holidayswiki --procs=12 --memory-limit=max (END - exit=0) [15:00:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:01:01] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=holidayswiki --procs=12 --memory-limit=max (START) [15:01:03] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=holidayswiki --procs=12 --memory-limit=max (END - exit=0) [15:01:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:01:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:01:33] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=caprifatawiki --procs=12 --memory-limit=max (START) [15:01:34] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=caprifatawiki --procs=12 --memory-limit=max (END - exit=0) [15:01:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:01:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:02:50] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=holidayswiki --procs=12 --memory-limit=max (START) [15:02:51] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=holidayswiki --procs=12 --memory-limit=max (END - exit=0) [15:02:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:02:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:22:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:26:51] [02mw-config] 07RhinosF1 opened pull request 03#5578: Bing: increase crawl-delay - 13https://github.com/miraheze/mw-config/pull/5578 [15:27:48] miraheze/mw-config - RhinosF1 the build passed. [15:43:21] [02mw-config] 07RhinosF1 commented on pull request 03#5578: Bing: increase crawl-delay - 13https://github.com/miraheze/mw-config/pull/5578#issuecomment-2133722622 [15:43:37] [02mw-config] 07RhinosF1 deleted a comment on pull request 03#5578: Bing: increase crawl-delay - 13https://github.com/miraheze/mw-config/pull/5578#issuecomment-2133722622 [15:43:46] [02mw-config] 07RhinosF1 commented on pull request 03#5578: Bing: increase crawl-delay - 13https://github.com/miraheze/mw-config/pull/5578#issuecomment-2133723292 [15:44:16] [02mw-config] 07RhinosF1 deleted a comment on pull request 03#5578: Bing: increase crawl-delay - 13https://github.com/miraheze/mw-config/pull/5578#issuecomment-2133723292 [15:44:20] [02mw-config] 07RhinosF1 commented on pull request 03#5578: Bing: increase crawl-delay - 13https://github.com/miraheze/mw-config/pull/5578#issuecomment-2133724193 [15:44:27] [02mw-config] 07coderabbitai[bot] commented on pull request 03#5578: Bing: increase crawl-delay - 13https://github.com/miraheze/mw-config/pull/5578#issuecomment-2133724358 [15:45:04] [02mw-config] 07coderabbitai[bot] edited pull request 03#5578: Bing: increase crawl-delay - 13https://github.com/miraheze/mw-config/pull/5578 [15:45:06] [02mw-config] 07coderabbitai[bot] edited a comment on pull request 03#5578: Bing: increase crawl-delay - 13https://github.com/miraheze/mw-config/pull/5578#issuecomment-2133724358 [15:51:29] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [16:21:00] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:40:16] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [16:50:28] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [17:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:36:27] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [18:04:27] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:10:20] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [21:32:25]