[00:16:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:27:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.84, 22.74, 23.81 [00:41:44] PROBLEM - christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.christipedia.nl' expires in 7 day(s) (Sun 28 Jul 2024 12:21:09 AM GMT +0000). [00:42:12] PROBLEM - www.christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.christipedia.nl' expires in 7 day(s) (Sun 28 Jul 2024 12:21:09 AM GMT +0000). [00:43:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:47:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.33, 23.08, 22.74 [00:49:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.14, 22.85, 22.72 [00:51:32] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [00:51:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.03, 23.55, 22.99 [00:53:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:53:32] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 203 system event log (SEL) entries present] [00:53:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.90, 23.39, 23.02 [00:55:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.80, 24.67, 23.50 [01:05:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.29, 23.46, 23.92 [01:07:52] PROBLEM - trollpasta.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'trollpasta.com' expires in 7 day(s) (Sun 28 Jul 2024 12:52:16 AM GMT +0000). [01:09:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.28, 22.68, 23.37 [01:11:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 14.89, 20.09, 22.37 [01:13:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.00, 22.58, 22.98 [01:15:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.18, 21.32, 22.52 [01:17:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.62, 22.52, 22.77 [01:19:18] PROBLEM - www.trollpasta.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'trollpasta.com' expires in 7 day(s) (Sun 28 Jul 2024 12:52:16 AM GMT +0000). [01:24:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:13:01] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/60defefa91b5...ee8e74afeee9 [02:13:04] [02ssl] 07WikiTideSSLBot 03ee8e74a - Bot: Update SSL cert for worldtriggerwiki.com [02:32:04] RECOVERY - worldtriggerwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'worldtriggerwiki.com' will expire on Fri 18 Oct 2024 01:12:55 AM GMT +0000. [02:33:58] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [02:35:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.21, 17.95, 22.84 [02:35:59] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 205 system event log (SEL) entries present] [02:37:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.44, 21.22, 23.44 [02:39:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:39:28] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.77, 19.33, 16.60 [02:41:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.14, 22.24, 23.50 [02:41:26] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.25, 17.88, 16.42 [02:43:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.23, 23.97, 23.97 [02:49:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.62, 23.06, 23.86 [02:57:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.83, 24.64, 23.89 [03:04:40] gaslight gatekeep girlboss [03:04:51] slay [03:05:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.91, 22.27, 23.96 [03:06:05] jingle bells [03:06:30] yass [03:07:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.29, 22.57, 23.77 [03:11:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.05, 22.20, 22.94 [03:13:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.04, 22.68, 23.18 [03:15:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.17, 21.33, 22.65 [03:15:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.50, 21.73, 22.68 [03:19:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.11, 22.30, 22.54 [03:19:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.98, 24.33, 23.43 [03:23:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.51, 23.93, 23.30 [03:25:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.65, 23.61, 23.22 [03:25:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:41:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.10, 22.99, 23.95 [03:43:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 23.17, 23.61, 24.10 [03:50:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:53:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.01, 23.46, 23.98 [03:55:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.64, 24.50, 24.29 [03:57:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.56, 22.25, 23.47 [04:01:10] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.95, 24.00, 23.78 [04:18:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:29:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.62, 21.49, 23.73 [04:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.09, 21.35, 23.99 [04:38:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:39:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.49, 23.39, 24.38 [04:39:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.08, 22.00, 22.67 [04:41:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.32, 21.15, 22.28 [04:43:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.89, 23.37, 22.93 [04:45:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.98, 21.73, 22.42 [04:47:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.46, 22.12, 22.42 [04:49:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.62, 21.61, 23.68 [04:49:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.88, 21.90, 22.32 [04:53:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.05, 22.85, 23.66 [04:55:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.77, 22.37, 23.88 [04:55:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.44, 22.65, 23.55 [05:01:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.33, 23.63, 23.82 [05:01:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.10, 24.33, 24.02 [05:04:20] [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:05:33] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.80, 3.46, 1.47 [05:07:33] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.97, 2.65, 1.42 [05:09:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.83, 23.41, 23.95 [05:09:20] [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:11:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 23.42, 23.85 [05:11:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.51, 24.45, 22.35 [05:12:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:21:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.81, 22.62, 22.92 [05:25:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 11.72, 18.23, 22.52 [05:27:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:27:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.07, 22.96, 22.89 [05:29:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 9.94, 13.44, 19.58 [05:31:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:35:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.67, 22.52, 21.47 [05:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.23, 22.49, 21.60 [05:39:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.81, 23.48, 22.08 [05:45:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.68, 22.45, 22.39 [05:45:32] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:47:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.41, 23.70, 22.85 [05:51:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.75, 23.25, 22.95 [06:05:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.10, 24.25, 23.46 [06:06:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:09:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.95, 23.57, 23.51 [06:09:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.86, 21.69, 23.78 [06:11:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.56, 24.41, 23.80 [06:11:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.18, 23.54, 24.17 [06:13:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.63, 23.34, 23.50 [06:19:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.69, 24.05, 23.70 [06:21:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.49, 22.99, 23.33 [06:23:00] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [06:25:01] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [207 system event log (SEL) entries present] [06:27:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:29:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.64, 22.28, 22.15 [06:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.09, 22.73, 22.77 [06:37:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:42:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:47:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.76, 21.90, 21.84 [06:47:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.45, 20.25, 23.55 [06:49:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.73, 22.20, 23.87 [06:51:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.36, 22.37, 23.73 [06:53:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.18, 23.89, 24.11 [06:55:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.14, 22.87, 22.67 [06:57:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.33, 23.32, 22.84 [06:59:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.76, 23.32, 22.94 [07:03:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.90, 24.67, 23.41 [07:07:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.38, 22.05, 23.55 [07:13:34] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:13:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.52, 23.56, 23.68 [07:15:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.13, 23.58, 23.70 [07:17:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:19:34] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:21:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 24.13, 23.67 [07:25:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.66, 23.88, 23.72 [07:27:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.59, 25.42, 24.29 [07:47:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:49:08] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [08:18:43] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:21:31] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [08:23:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.29, 18.44, 22.93 [08:23:31] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [208 system event log (SEL) entries present] [08:23:33] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 109.123.230.163/cpweb, 2400:d320:2161:9775::1/cpweb [08:23:34] PROBLEM - cp41 SSH on cp41 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 109.123.230.163/cpweb, 2400:d320:2161:9775::1/cpweb [08:24:12] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 100% [08:25:05] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [08:25:42] PROBLEM - Host cp41 is DOWN: PING CRITICAL - Packet loss = 100% [08:27:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 11.53, 18.67, 23.88 [08:31:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.82, 16.79, 20.18 [08:31:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 12.09, 17.37, 23.19 [08:32:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:35:14] RECOVERY - Host cp41 is UP: PING OK - Packet loss = 0%, RTA = 110.97 ms [08:35:26] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3841 bytes in 0.745 second response time [08:35:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:36:20] RECOVERY - cp41 SSH on cp41 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [08:36:45] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 102.67 ms [08:37:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.77, 20.31, 21.73 [08:37:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:37:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.95, 21.32, 22.62 [08:39:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.14, 20.75, 19.88 [08:39:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.07, 20.05, 22.03 [08:41:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.28, 21.90, 22.42 [08:45:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.83, 23.19, 22.83 [08:47:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.53, 23.05, 21.49 [08:47:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.22, 23.97, 23.19 [08:49:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.05, 23.28, 23.05 [08:53:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.25, 23.41, 23.03 [08:55:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.66, 23.95, 22.11 [09:01:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.09, 23.30, 23.59 [09:03:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.84, 22.52, 22.45 [09:06:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:07:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.81, 23.50, 22.83 [09:09:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.56, 23.16, 23.04 [09:11:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.17, 22.05, 22.66 [09:13:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.79, 23.47, 23.11 [09:26:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:46:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:51:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:52:00] PROBLEM - www.journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.journeytheword.wiki' expires in 7 day(s) (Sun 28 Jul 2024 09:51:43 AM GMT +0000). [09:55:53] PROBLEM - patternarchive.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'patternarchive.online' expires in 7 day(s) (Sun 28 Jul 2024 09:50:16 AM GMT +0000). [10:09:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.48, 21.29, 23.78 [10:15:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.21, 20.81, 23.93 [10:17:26] PROBLEM - journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.journeytheword.wiki' expires in 7 day(s) (Sun 28 Jul 2024 09:51:43 AM GMT +0000). [10:19:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.36, 23.51, 24.19 [10:19:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.34, 24.96, 24.11 [10:25:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.84, 22.46, 23.77 [10:26:02] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [10:26:50] [02mw-config] 07redbluegreenhat closed pull request 03#5613: index.php: remove backwards compatibility - 13https://github.com/miraheze/mw-config/pull/5613 [10:26:53] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5e50cf85d60a...15dac3a6d83c [10:26:54] [02mw-config] 07Universal-Omega 0315dac3a - index.php: remove backwards compatibility (#5613) [10:26:57] [02mw-config] 07redbluegreenhat deleted branch 03Universal-Omega-patch-2 [10:27:00] [02mw-config] 07redbluegreenhat deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [10:27:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.41, 19.16, 23.00 [10:27:44] miraheze/mw-config - redbluegreenhat the build passed. [10:28:02] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [209 system event log (SEL) entries present] [10:29:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.25, 22.20, 23.67 [10:29:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 22.78, 23.44 [10:31:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.05, 21.50, 23.26 [10:31:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.48, 21.26, 22.79 [10:32:42] !log [@mwtask181] starting deploy of {'config': True} to all [10:32:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:32:55] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 13s [10:33:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:35:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.76, 24.90, 24.12 [10:37:47] !log [@mwtask171] starting deploy of {'config': True} to all [10:37:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:38:00] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 12s [10:38:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:40:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:43:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.07, 22.46, 23.57 [10:45:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 11.42, 19.17, 22.98 [10:47:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.33, 21.78, 22.89 [10:49:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.70, 22.76, 23.14 [10:49:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.21, 14.88, 20.35 [10:51:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.96, 24.78, 23.84 [10:53:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.81, 22.12, 21.64 [10:53:22] !log [@test151] starting deploy of {'config': True} to test151 [10:53:23] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [10:53:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:53:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:57:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.26, 22.15, 23.16 [10:59:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.22, 25.06, 24.10 [11:01:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.75, 23.28, 22.99 [11:03:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.30, 25.53, 23.86 [11:03:33] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.57, 2.70, 1.17 [11:03:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.10, 19.46, 19.43 [11:05:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:05:33] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.31, 2.86, 1.39 [11:05:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.13, 18.06, 18.92 [11:09:33] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.24, 3.07, 1.92 [11:09:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.56, 20.63, 19.73 [11:11:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.13, 22.27, 20.39 [11:13:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.83, 21.90, 20.46 [11:23:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.21, 23.23, 21.63 [11:35:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.52, 23.43, 23.26 [11:37:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.59, 25.50, 24.04 [11:39:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.40, 23.97, 23.70 [11:40:26] PROBLEM - wiki.misslessfighters.club - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.misslessfighters.club' expires in 7 day(s) (Sun 28 Jul 2024 11:18:10 AM GMT +0000). [11:41:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 23.33, 23.45 [12:04:10] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:10:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:13:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.29, 21.77, 23.79 [12:15:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:19:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.40, 23.62, 23.81 [12:25:30] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:33:20] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [12:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.53, 19.68, 23.42 [12:37:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 12.72, 18.20, 22.97 [12:39:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.17, 21.82, 23.76 [12:39:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.35, 21.37, 23.63 [12:43:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.50, 21.16, 23.16 [12:49:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.37, 17.09, 20.29 [12:51:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.45, 21.88, 22.43 [12:51:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.72, 22.21, 22.43 [12:53:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.06, 20.31, 20.75 [12:53:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.54, 19.75, 21.49 [12:59:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.32, 23.90, 22.58 [13:01:10] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.60, 23.87, 22.67 [13:01:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.30, 21.01, 21.23 [13:05:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 14.97, 21.98, 23.45 [13:05:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.58, 23.06, 22.87 [13:05:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.20, 22.80, 21.97 [13:07:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.00, 24.97, 23.57 [13:07:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.09, 23.39, 22.30 [13:09:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:09:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.30, 23.83, 23.80 [13:09:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.48, 23.12, 23.04 [13:11:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.74, 22.11, 23.17 [13:13:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.20, 23.08, 23.39 [13:13:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.54, 22.95, 22.57 [13:14:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:15:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 14.00, 19.80, 22.17 [13:21:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.95, 18.35, 20.23 [13:21:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 30.12, 24.54, 22.94 [13:23:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.82, 16.54, 19.84 [13:37:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.14, 21.44, 19.36 [13:37:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.84, 19.08, 17.89 [13:41:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 12.95, 19.38, 19.24 [13:41:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.94, 18.58, 18.13 [13:42:47] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [13:43:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.88, 20.28, 23.71 [13:44:48] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [210 system event log (SEL) entries present] [13:53:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 21.75, 22.58 [13:55:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.34, 21.88, 22.56 [14:01:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.33, 19.57, 17.18 [14:03:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 21.17, 18.04 [14:05:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.74, 21.23, 18.44 [14:05:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.11, 23.08, 22.52 [14:05:52] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.26, 20.90, 18.18 [14:07:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.64, 20.21, 18.38 [14:11:40] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.31, 17.80, 17.84 [14:14:36] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o