[00:01:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.84, 3.87, 3.39
[00:03:56] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.31, 3.16, 3.19
[00:13:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.82, 22.39, 23.43
[00:18:32] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.57, 3.67, 3.18
[00:18:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.91, 21.65, 23.60
[00:20:28] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.03, 3.15, 3.03
[00:25:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.66, 20.56, 21.19
[00:27:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:28:09] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.42, 3.81, 3.24
[00:29:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.53, 22.04, 21.73
[00:30:03] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.31, 3.44, 3.19
[00:32:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:32:50] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.50, 18.29, 20.31
[00:33:55] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.58, 3.12, 3.14
[00:35:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.22, 22.30, 21.87
[00:37:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.42, 21.15, 21.46
[00:40:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.22, 4.04, 3.44
[00:42:41] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.99, 3.34, 3.24
[00:45:03] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.20, 19.29, 20.38
[00:46:07] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[00:47:26] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.74, 20.00, 20.10
[00:48:07] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 389 system event log (SEL) entries present]
[00:49:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.26, 20.08, 20.58
[00:49:21] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.05, 19.22, 19.78
[00:51:03] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.59, 19.26, 20.23
[00:55:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.93, 22.72, 21.32
[00:57:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:57:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.83, 23.03, 21.62
[00:58:01] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.27, 4.55, 3.63
[00:59:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.72, 22.45, 21.50
[00:59:56] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.02, 3.68, 3.41
[01:00:46] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152
[01:01:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.49, 22.82, 21.81
[01:01:55] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.88, 21.46, 20.66
[01:01:55] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.01, 2.83, 3.14
[01:02:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:04:43] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[01:05:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.13, 24.04, 22.48
[01:05:32] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152
[01:05:46] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.62, 23.95, 21.77
[01:07:37] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.91, 21.61, 17.65
[01:07:41] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.66, 23.92, 22.03
[01:09:33] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.27, 21.31, 18.02
[01:09:37] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.58, 25.15, 22.72
[01:11:19] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[01:11:32] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.90, 23.00, 22.24
[01:13:28] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.23, 24.79, 22.98
[01:15:33] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.76, 20.14, 18.65
[01:17:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.90, 23.95, 23.56
[01:17:19] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.27, 22.88, 22.69
[01:21:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 22.80, 23.09
[01:21:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.98, 22.32, 22.36
[01:23:05] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.14, 22.74, 22.56
[01:25:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.84, 22.32, 22.98
[01:30:16] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[01:32:17] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 391 system event log (SEL) entries present]
[01:35:03] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.02, 17.85, 20.12
[01:42:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.34, 21.88, 21.77
[01:44:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.74, 22.12, 21.85
[01:48:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.36, 22.73, 22.00
[01:52:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.38, 23.36, 22.48
[01:57:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.19, 20.19, 19.65
[02:04:50] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.97, 18.00, 20.39
[02:07:03] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.22, 18.87, 19.70
[02:07:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.51, 22.12, 23.97
[02:16:16] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:18:22] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.735 second response time
[02:19:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.04, 19.90, 19.05
[02:21:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.02, 21.74, 21.93
[02:23:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.45, 22.42, 20.24
[02:25:05] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152
[02:26:48] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152
[02:27:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.36, 22.74, 21.05
[02:28:44] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[02:28:55] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[02:30:28] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[02:31:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.01, 21.99, 21.00
[02:31:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 13.03, 20.19, 22.14
[02:32:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:32:28] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 393 system event log (SEL) entries present]
[02:33:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.62, 20.50, 20.59
[02:33:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.81, 22.32, 22.69
[02:35:03] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.22, 19.86, 20.34
[02:37:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.14, 23.32, 23.14
[02:39:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.82, 23.91, 23.38
[02:41:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.07, 22.34, 22.89
[02:47:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:49:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 21.34, 22.01
[02:51:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.06, 21.07, 21.83
[03:02:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:03:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 22.04, 21.79
[03:03:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.35, 2.64, 1.10
[03:05:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.22, 21.97, 21.85
[03:05:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.49, 2.51, 1.22
[03:07:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.14, 22.82, 22.17
[03:07:25] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:07:57] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.76, 3.62, 1.79
[03:08:24] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[03:09:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.59, 21.66, 21.81
[03:10:21] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.073 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[03:11:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.44, 3.99, 2.43
[03:12:25] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:13:55] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.44, 3.20, 2.32
[03:17:25] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:23:19] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 11.98, 17.48, 19.91
[03:25:56] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.21, 3.56, 2.77
[03:27:25] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:27:55] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.34, 3.09, 2.69
[03:30:42] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:35:42] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:35:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:35:53] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.98, 22.97, 21.30
[03:38:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.57, 3.76, 3.14
[03:39:43] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.38, 22.13, 21.43
[03:40:32] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.13, 3.80, 3.22
[03:44:24] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.09, 3.34, 3.21
[03:45:26] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.29, 23.29, 22.01
[03:45:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:47:21] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.14, 22.90, 21.99
[03:47:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:50:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.23, 4.45, 3.67
[03:52:07] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.50, 3.81, 3.51
[03:54:01] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.49, 3.12, 3.30
[03:55:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.04, 22.88, 22.16
[03:57:50] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:59:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.62, 21.21, 21.72
[04:02:50] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:05:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.00, 22.34, 21.83
[04:07:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.68, 21.99, 21.75
[04:07:50] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:08:33] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.85, 4.39, 3.48
[04:09:16] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[04:09:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.48, 22.35, 21.88
[04:11:13] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.089 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[04:11:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.13, 22.86, 22.17
[04:12:23] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.62, 3.03, 3.22
[04:12:50] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:17:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.78, 22.44, 22.08
[04:19:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.01, 21.99, 21.99
[04:21:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.85, 22.77, 22.22
[04:25:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.16, 22.94, 22.53
[04:27:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.13, 23.18, 22.66
[04:29:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.91, 22.42, 22.44
[04:35:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.37, 23.75, 22.90
[04:37:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.08, 22.60, 22.58
[04:39:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 23.64, 22.95
[04:41:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.55, 23.94, 23.17
[04:47:50] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:55:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.57, 22.72, 22.19
[04:57:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.69, 22.67, 18.40
[04:58:11] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.25, 22.35, 16.16
[05:01:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.97, 22.62, 22.63
[05:03:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.54, 23.98, 23.14
[05:07:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.69, 22.78, 22.93
[05:11:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.65, 23.78, 22.95
[05:18:11] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 13.85, 21.97, 22.87
[05:19:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:21:03] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.16, 17.29, 20.31
[05:21:19] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.20, 18.05, 20.04
[05:22:11] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 10.70, 15.62, 20.03
[05:46:26] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[05:52:52] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:57:00] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.000215023756 secs
[06:14:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:16:27] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:16:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.69, 20.32, 18.16
[06:18:50] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.90, 19.92, 18.29
[06:35:21] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.04, 21.29, 19.12
[06:37:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.63, 20.49, 19.08
[06:39:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.91, 22.14, 19.82
[06:41:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.56, 22.52, 20.24
[06:45:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:47:11] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:49:19] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.14, 19.08, 19.53
[06:55:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.74, 20.42, 19.88
[06:59:19] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 20.04, 20.25, 19.92
[07:02:29] <icinga-miraheze>	 PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2]
[07:03:28] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:05:28] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001509666443 secs
[07:15:15] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[07:23:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.18, 19.39, 17.56
[07:23:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.28, 23.21, 20.39
[07:25:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.44, 23.11, 19.15
[07:25:56] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.86, 20.68, 18.34
[07:27:52] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.02, 23.40, 19.60
[07:29:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.51, 23.93, 20.51
[07:29:47] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.86, 23.02, 19.98
[07:30:29] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[07:33:03] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.50, 18.82, 19.23
[07:33:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.92, 23.57, 22.64
[07:33:38] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.21, 19.51, 19.27
[07:45:22] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:51:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.28, 21.83, 21.43
[07:53:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.52, 20.82, 21.14
[07:55:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:55:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.71, 23.09, 21.89
[08:01:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.93, 23.58, 22.72
[08:03:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 24.46, 23.16
[08:11:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.98, 22.95, 23.53
[08:13:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 23.68, 23.69
[08:14:23] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[08:15:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.66, 23.09, 23.48
[08:17:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.25, 24.24, 23.82
[08:17:21] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:17:56] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152
[08:21:34] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.834 second response time
[08:21:53] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.36, 19.77, 18.75
[08:21:56] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[08:22:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:23:49] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.80, 17.66, 18.12
[08:25:36] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.57, 20.02, 18.62
[08:33:22] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.90, 18.80, 18.88
[08:39:14] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.20, 22.36, 20.37
[08:43:07] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.20, 24.18, 21.44
[08:45:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.84, 22.95, 21.34
[08:47:15] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[08:53:03] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.16, 18.00, 19.81
[08:59:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.72, 19.49, 22.99
[09:03:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 20.64, 22.52
[09:06:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.95, 21.18, 18.41
[09:09:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.88, 21.09, 19.67
[09:11:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.68, 23.92, 20.89
[09:11:43] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:13:48] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0007724165916 secs
[09:14:11] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.56, 19.66, 16.12
[09:18:11] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 20.07, 19.59, 16.87
[09:18:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.92, 23.47, 21.77
[09:22:22] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.79, 19.92, 17.69
[09:24:19] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.64, 18.79, 17.57
[09:27:41] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:31:56] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.572 second response time
[09:34:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.77, 21.58, 21.31
[09:36:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.89, 21.82, 21.45
[09:40:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.26, 22.75, 21.87
[09:42:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.68, 22.03, 21.73
[09:43:07] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:43:29] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152
[09:43:31] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152
[09:44:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.21, 23.38, 22.26
[09:45:06] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0006036758423 secs
[09:45:24] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[09:45:31] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[09:46:12] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[09:46:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.37, 22.90, 22.22
[09:47:56] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw161
[09:48:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.72, 23.78, 22.58
[09:49:52] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[09:50:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.72, 22.92, 22.43
[09:54:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.97, 24.93, 23.22
[09:58:58] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.69, 20.60, 16.42
[10:00:55] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.09, 18.37, 16.12
[10:06:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.52, 22.06, 23.45
[10:10:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.59, 23.84, 23.77
[10:19:44] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.77, 21.60, 18.62
[10:21:41] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.78, 20.04, 18.41
[10:24:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.53, 22.88, 23.89
[10:31:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.08, 18.94, 23.51
[10:32:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.26, 24.16, 23.75
[10:45:13] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:46:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.54, 21.40, 23.55
[10:52:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.51, 22.44, 23.10
[10:54:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.50, 22.45, 23.02
[10:58:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.67, 24.59, 23.65
[11:02:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:02:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.30, 23.86, 23.65
[11:05:34] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152
[11:07:29] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[11:08:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.51, 22.76, 23.06
[11:14:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.80, 22.78, 23.34
[11:15:03] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.28, 19.15, 20.36
[11:15:11] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.15, 19.32, 18.09
[11:15:50] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[11:19:07] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.20, 18.57, 18.18
[11:19:49] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152
[11:21:43] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[11:22:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:24:59] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.47, 21.12, 19.37
[11:26:57] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.22, 20.00, 19.18
[11:27:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:28:50] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.98, 17.39, 20.09
[11:29:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.29, 4.03, 2.88
[11:33:56] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.90, 3.42, 2.86
[11:35:56] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.89, 4.40, 3.29
[11:37:42] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.66, 22.14, 20.21
[11:37:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.50, 3.62, 3.15
[11:39:56] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.60, 2.97, 2.97
[11:41:38] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.59, 19.19, 19.49
[11:42:35] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.09, 20.77, 19.71
[11:44:32] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.58, 21.99, 20.27
[11:45:58] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 19.51, 15.59
[11:46:24] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:48:56] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.78, 4.41, 3.58
[11:49:58] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.00, 20.76, 17.08
[11:50:21] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.27, 23.49, 21.70
[11:50:51] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.56, 3.43, 3.31
[11:51:58] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 13.90, 18.23, 16.60
[11:52:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.26, 2.81, 3.11
[11:56:11] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.38, 22.99, 21.73
[11:57:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:58:33] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.48, 4.68, 3.73
[12:00:04] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.29, 22.14, 21.77
[12:00:29] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.95, 3.77, 3.50
[12:02:25] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.68, 4.19, 3.69
[12:02:53] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:04:47] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[12:06:14] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.79, 3.51, 3.54
[12:07:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:07:49] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.19, 18.34, 19.98
[12:08:09] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.53, 2.98, 3.33
[12:14:56] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[12:16:57] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152
[12:17:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:18:52] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[12:28:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:29:40] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.67, 4.02, 3.35
[12:31:35] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.78, 3.26, 3.16
[12:32:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.61, 21.80, 20.45
[12:33:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:34:06] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.22, 19.58, 19.79
[12:41:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:46:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:48:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:53:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:57:28] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.01, 21.31, 19.84
[12:58:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:58:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.38, 4.39, 3.24
[13:00:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.94, 20.79, 19.66
[13:02:50] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.59, 19.82, 19.41
[13:03:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:07:11] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.99, 19.33, 19.91
[13:08:23] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.34, 3.69, 3.80
[13:12:12] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.11, 2.64, 3.37
[13:15:45] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[13:16:03] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.30, 3.67, 3.66
[13:19:56] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.61, 3.39, 3.50
[13:21:56] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.21, 3.22, 3.45
[13:23:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.94, 4.25, 3.80
[13:28:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:29:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.43, 3.53, 3.70
[13:31:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.28, 4.21, 3.93
[13:35:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.60, 3.94, 3.91
[13:38:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:39:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.86, 4.18, 3.97
[13:41:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.96, 3.72, 3.80
[13:43:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:44:30] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[13:45:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.47, 3.81, 3.80
[13:47:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.71, 3.44, 3.64
[13:48:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:49:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.16, 21.56, 23.77
[13:51:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.76, 23.01, 24.01
[13:51:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.00, 3.77, 3.67
[13:53:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:57:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.25, 3.43, 3.61
[13:58:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:59:55] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.32, 2.82, 3.35
[14:03:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:03:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 14.65, 20.89, 23.34
[14:03:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.13, 3.77, 3.67
[14:04:22] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[14:05:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.67, 4.12, 3.79
[14:06:21] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0003278255463 secs
[14:07:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.92, 3.36, 3.54
[14:08:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:09:55] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.77, 2.89, 3.33
[14:13:19] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.10, 17.14, 20.10
[14:15:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.28, 3.55, 3.51
[14:18:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:19:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.85, 20.84, 21.01
[14:19:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.39, 3.88, 3.62
[14:20:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:21:32] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[14:23:19] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.05, 19.04, 20.29
[14:23:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.16, 3.75, 3.67
[14:25:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:25:35] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.089 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[14:27:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.26, 21.62, 21.07
[14:29:55] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.06, 2.83, 3.30
[14:35:19] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.24, 18.93, 20.10
[14:36:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:37:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.46, 3.46, 3.27
[14:39:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.87, 3.24, 3.19
[14:41:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:41:55] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.78, 2.93, 3.11
[14:45:41] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.78, 18.50, 22.94
[14:47:41] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 20.48, 23.10
[14:52:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:56:30] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.94, 4.18, 3.33
[14:57:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:58:24] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.06, 3.43, 3.14
[14:59:41] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:00:19] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.05, 3.22, 3.11
[15:04:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.07, 4.23, 3.50
[15:04:10] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2407:3641:2161:9774::1/cpweb
[15:04:41] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:06:05] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.05, 2.94, 3.11
[15:06:19] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[15:07:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.37, 21.50, 19.41
[15:09:34] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.54, 18.28, 14.24
[15:11:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.53, 23.01, 20.37
[15:13:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.46, 22.85, 20.63
[15:15:34] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.76, 17.71, 15.42
[15:17:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.13, 24.66, 21.81
[15:19:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.89, 22.69, 21.43
[15:22:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:25:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.30, 22.01, 21.31
[15:30:45] <icinga-miraheze>	 PROBLEM - wiki.metaiitgn.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.metaiitgn.org' expires in 15 day(s) (Mon 12 Aug 2024 03:07:07 PM GMT +0000).
[15:30:57] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d1e827aa87c3...6ae4217cfd18
[15:31:00] <botifico>	 [02ssl] 07WikiTideSSLBot 036ae4217 - Bot: Update SSL cert for wiki.metaiitgn.org
[15:39:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.63, 23.22, 23.13
[15:41:43] <icinga-miraheze>	 PROBLEM - wiki.seamly.io - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.seamly.io' expires in 15 day(s) (Mon 12 Aug 2024 03:29:10 PM GMT +0000).
[15:41:54] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6ae4217cfd18...53e8f6daae5c
[15:41:57] <botifico>	 [02ssl] 07WikiTideSSLBot 0353e8f6d - Bot: Update SSL cert for wiki.seamly.io
[15:43:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.75, 23.37, 23.05
[15:45:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.77, 22.59, 22.80
[15:55:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.68, 21.95, 21.81
[15:57:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.13, 21.88, 21.84
[15:59:30] <icinga-miraheze>	 RECOVERY - wiki.metaiitgn.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.metaiitgn.org' will expire on Fri 25 Oct 2024 02:30:51 PM GMT +0000.
[16:01:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.80, 21.55, 21.52
[16:03:51] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.43, 22.67, 18.93
[16:04:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.22, 20.47, 17.25
[16:05:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.89, 23.80, 22.65
[16:05:45] <botifico>	 [02ssl] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/53e8f6daae5c...381b8f7e6b56
[16:05:47] <botifico>	 [02ssl] 07OAuthority 03381b8f7 - remove psychonautzhwikiwiki (T12396)
[16:06:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.28, 23.72, 18.84
[16:07:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.81, 25.34, 23.34
[16:10:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.49, 23.75, 19.96
[16:12:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.41, 24.06, 20.49
[16:18:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.94, 22.03, 20.99
[16:20:55] <icinga-miraheze>	 PROBLEM - wikappedia.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikappedia.cc' expires in 15 day(s) (Mon 12 Aug 2024 03:57:45 PM GMT +0000).
[16:21:07] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/381b8f7e6b56...f05db672148d
[16:21:08] <botifico>	 [02ssl] 07WikiTideSSLBot 03f05db67 - Bot: Update SSL cert for wikappedia.cc
[16:21:41] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.60, 23.81, 22.78
[16:22:48] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.22, 19.80, 20.32
[16:23:37] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.20, 24.94, 23.30
[16:26:47] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.31, 21.17, 20.82
[16:28:46] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.77, 24.06, 21.92
[16:30:46] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.14, 23.91, 22.18
[16:36:06] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152
[16:36:21] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:38:03] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[16:38:26] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.102 second response time
[16:40:10] <icinga-miraheze>	 RECOVERY - wiki.seamly.io - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seamly.io' will expire on Fri 25 Oct 2024 02:41:49 PM GMT +0000.
[16:44:42] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.21, 22.01, 21.45
[16:46:41] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.98, 22.22, 21.60
[16:50:08] <icinga-miraheze>	 RECOVERY - wikappedia.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'wikappedia.cc' will expire on Fri 25 Oct 2024 03:21:01 PM GMT +0000.
[16:52:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.63, 22.31, 23.91
[16:54:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.78, 23.79, 24.21
[16:54:39] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.20, 19.57, 20.38
[17:00:36] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.04, 21.38, 20.92
[17:02:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:02:36] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.05, 18.22, 19.82
[17:02:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.64, 20.74, 23.14
[17:07:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:08:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.07, 21.35, 22.30
[17:10:11] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[17:10:15] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.50, 4.35, 2.24
[17:12:06] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.069 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:12:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.61, 21.62, 22.33
[17:14:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.31, 22.21, 22.40
[17:15:58] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.96, 3.87, 2.71
[17:16:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.06, 21.54, 22.11
[17:17:53] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.71, 3.29, 2.63
[17:18:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 22.59, 22.44
[17:20:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.32, 23.20, 22.74
[17:21:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 9.97, 6.14, 3.89
[17:22:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.12, 24.74, 23.33
[17:25:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.72, 3.80, 3.44
[17:26:27] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.97, 22.76, 20.48
[17:27:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:27:39] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.95, 4.91, 3.88
[17:28:26] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.59, 22.80, 20.79
[17:28:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.71, 23.08, 23.34
[17:32:25] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.99, 20.01, 20.10
[17:32:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:33:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.59, 3.16, 3.49
[17:37:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:37:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.66, 4.08, 3.74
[17:39:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.14, 3.62, 3.64
[17:41:36] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.35, 2.45, 3.21
[17:42:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:46:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 14.12, 18.11, 23.09
[17:46:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.20, 17.97, 19.92
[17:47:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:50:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.04, 20.44, 20.46
[17:50:42] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152
[17:52:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.95, 21.54, 22.76
[17:52:39] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[17:54:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.70, 20.79, 22.36
[17:54:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.61, 23.00, 21.40
[17:57:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:58:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.62, 21.49, 21.23
[18:02:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 23.00, 22.72
[18:04:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.13, 19.35, 20.34
[18:08:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.38, 23.12, 23.20
[18:08:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.24, 21.29, 20.92
[18:12:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.56, 22.46, 21.33
[18:14:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.93, 20.57, 20.79
[18:16:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.05, 24.24, 23.29
[18:20:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.82, 19.12, 20.12
[18:24:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.40, 22.25, 21.06
[18:26:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.01, 21.81, 21.04
[18:27:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:28:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.36, 23.83, 21.84
[18:30:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.14, 21.47, 21.22
[18:32:55] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152
[18:34:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.33, 19.06, 20.34
[18:34:50] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[18:36:53] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:37:13] <icinga-miraheze>	 PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 50%
[18:39:01] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.074 second response time
[18:39:13] <icinga-miraheze>	 RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 21%
[18:42:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.46, 22.94, 23.95
[18:47:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:48:33] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:54:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.74, 22.41, 22.53
[18:56:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.53, 20.66, 19.19
[18:58:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.98, 19.48, 18.93
[19:00:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.50, 23.71, 23.20
[19:04:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 22.71, 22.76
[19:08:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.60, 20.59, 22.06
[19:15:56] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:16:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.98, 18.06, 20.29
[19:22:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.76, 22.93, 19.96
[19:22:52] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.12, 20.99, 18.32
[19:23:14] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.63, 24.69, 22.33
[19:24:50] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152
[19:26:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.76, 23.33, 20.95
[19:26:50] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[19:28:50] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.13, 20.00, 18.96
[19:30:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.35, 19.12, 19.70
[19:30:51] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.35, 21.99, 22.34
[19:40:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.56, 18.45, 20.37
[19:43:13] <icinga-miraheze>	 PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CRITICAL - NGINX Error Rate is 87%
[19:45:13] <icinga-miraheze>	 PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 51%
[19:47:13] <icinga-miraheze>	 RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 5%
[19:49:37] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152
[19:53:37] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[20:06:46] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.56, 23.79, 21.08
[20:08:41] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.40, 21.98, 20.76
[20:12:29] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.90, 23.68, 21.67
[20:16:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.66, 19.07, 17.95
[20:18:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.99, 19.93, 18.45
[20:24:32] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.15, 20.41, 17.93
[20:26:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.83, 22.49, 19.87
[20:28:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.95, 20.78, 19.57
[20:30:31] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.55, 19.92, 18.62
[20:38:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.91, 19.85, 20.12
[20:52:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.84, 21.82, 23.88
[20:54:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.38, 22.81, 23.96
[20:56:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.03, 23.07, 23.95
[20:58:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.64, 23.62, 24.02
[21:02:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:03:14] <MirahezeLSBot>	 !log [void@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CountDownClock'} to all
[21:03:15] <MirahezeLSBot>	 !log [void@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'CountDownClock'} to all - SUCCESS in 1s
[21:03:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:03:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:03:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.15, 3.16, 1.30
[21:05:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.52, 2.87, 1.41
[21:07:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:09:37] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.17, 2.61, 1.60
[21:10:42] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.60, 20.30, 19.24
[21:12:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.42, 18.52, 18.72
[21:17:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:19:15] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.40, 18.63, 17.23
[21:20:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.16, 21.99, 20.09
[21:21:15] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.44, 18.51, 17.42
[21:21:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:22:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.28, 19.46, 19.43
[21:23:37] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152
[21:25:37] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[21:26:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:28:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:31:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.92, 3.42, 2.89
[21:33:36] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.03, 3.16, 2.88
[21:33:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:35:58] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:40:25] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.83, 4.26, 3.38
[21:42:19] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.52, 3.43, 3.18
[21:44:13] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.03, 4.32, 3.53
[21:45:58] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:46:08] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.44, 3.44, 3.29
[21:47:59] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:48:01] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.81, 3.39, 3.30
[21:52:11] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.176 second response time
[21:55:58] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:56:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.89, 3.68, 3.36
[21:58:37] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:00:24] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.47, 3.72, 3.48
[22:00:43] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.329 second response time
[22:00:58] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:02:18] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.68, 3.12, 3.28
[22:06:06] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 2.92, 4.24, 3.79
[22:08:00] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.75, 3.79, 3.68
[22:10:58] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:11:50] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.04, 3.90, 3.71
[22:13:44] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.29, 3.37, 3.53
[22:15:58] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:17:36] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.89, 3.00, 3.37
[22:25:16] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.37, 21.10, 19.02
[22:25:55] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.50, 19.76, 17.62
[22:27:54] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.38, 23.29, 19.18
[22:29:08] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.51, 22.96, 20.27
[22:29:54] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.09, 22.52, 19.38
[22:30:19] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.76, 3.81, 3.30
[22:32:13] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.56, 3.28, 3.15
[22:33:52] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.17, 19.52, 18.96
[22:34:07] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.23, 2.72, 2.95
[22:34:56] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.82, 20.00, 19.79
[22:44:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.70, 20.31, 23.46
[22:54:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.21, 20.26, 21.46
[22:56:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.53, 20.06, 21.26
[23:11:08] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.18, 19.64, 17.74
[23:14:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.65, 19.37, 20.33
[23:15:00] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.37, 18.26, 17.61
[23:26:57] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.19, 21.42, 20.52
[23:28:51] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.91, 21.83, 20.78
[23:30:46] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.43, 19.36, 19.98
[23:36:29] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.10, 21.00, 20.50
[23:38:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.20, 22.32, 21.02
[23:40:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.29, 21.16, 20.76
[23:45:58] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:50:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.10, 19.42, 20.26