[00:02:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.29, 23.86, 21.29
[00:04:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.07, 22.94, 21.24
[00:08:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.43, 20.06, 20.40
[00:09:28] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:11:32] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.540 second response time
[00:13:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:18:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.23, 21.75, 20.74
[00:20:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.40, 23.62, 21.52
[00:29:26] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.33, 23.28, 20.20
[00:31:22] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.18, 21.27, 19.83
[00:33:18] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.25, 19.99, 19.53
[00:53:18] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152
[00:55:15] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[01:28:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.59, 20.25, 23.42
[01:34:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.07, 21.31, 22.68
[01:36:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.36, 22.70, 23.08
[01:38:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:44:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.09, 23.22, 22.68
[01:46:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.17, 22.97, 22.65
[02:04:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.08, 22.31, 21.36
[02:06:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:06:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.67, 20.87, 20.97
[02:08:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.78, 19.03, 20.31
[02:08:48] <icinga-miraheze>	 PROBLEM - wiki.pulsus.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pulsus.cc' expires in 15 day(s) (Tue 13 Aug 2024 01:37:49 AM GMT +0000).
[02:09:01] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f05db672148d...c23a31f7f7e9
[02:09:03] <botifico>	 [02ssl] 07WikiTideSSLBot 03c23a31f - Bot: Update SSL cert for wiki.pulsus.cc
[02:14:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.23, 21.11, 20.74
[02:16:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.69, 22.05, 21.11
[02:17:44] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.37, 20.72, 17.92
[02:19:43] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.28, 19.69, 17.90
[02:20:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.47, 23.78, 22.13
[02:27:47] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.32, 21.31, 19.48
[02:29:43] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.34, 20.24, 19.27
[02:32:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.76, 23.83, 22.88
[02:38:06] <icinga-miraheze>	 RECOVERY - wiki.pulsus.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pulsus.cc' will expire on Sat 26 Oct 2024 01:08:54 AM GMT +0000.
[02:38:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.82, 23.81, 23.40
[02:48:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.74, 23.87, 23.03
[02:49:04] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.84, 19.83, 18.93
[02:50:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.10, 22.32, 22.61
[02:51:00] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.49, 18.70, 18.63
[03:02:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.00, 23.85, 22.95
[03:02:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.78, 20.17, 18.95
[03:04:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.45, 22.78, 22.72
[03:04:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.75, 18.16, 18.35
[03:06:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:09:18] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[03:12:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.06, 20.84, 21.38
[03:14:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.00, 19.91, 20.97
[03:26:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.27, 23.75, 22.05
[03:28:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.72, 22.32, 21.76
[03:32:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.36, 18.58, 20.34
[03:33:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:36:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.41, 21.92, 21.29
[03:38:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.03, 20.80, 20.94
[03:42:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.51, 20.25, 18.23
[03:44:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.48, 19.25, 20.24
[03:44:44] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.36, 19.19, 18.07
[03:54:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.87, 20.72, 20.55
[03:56:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.35, 19.66, 20.20
[04:06:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.75, 20.07, 19.78
[04:08:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.59, 19.15, 19.44
[04:09:05] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[04:41:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.99, 20.47, 18.85
[04:43:03] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.07, 19.28, 18.64
[04:46:52] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 24.00, 22.18, 19.98
[04:50:41] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.91, 20.35, 19.69
[04:53:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:58:06] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.00, 21.16, 18.44
[05:02:06] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.18, 18.31, 18.00
[05:02:25] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:07:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:08:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:10:21] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[05:10:25] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.36, 3.80, 2.10
[05:12:19] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.83, 3.08, 2.04
[05:12:22] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 6.188 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[05:13:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:17:05] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[05:22:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:36:37] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.56, 21.71, 20.26
[05:42:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.83, 20.17, 20.15
[05:45:03] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[05:47:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:52:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.54, 22.57, 20.95
[05:52:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:57:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:58:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.44, 22.11, 21.46
[05:59:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[05:59:51] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 37.91, 16.53, 6.92
[06:03:51] <icinga-miraheze>	 RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.43, 8.55, 5.92
[06:04:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.71, 23.99, 22.38
[06:04:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:07:03] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:13:50] <icinga-miraheze>	 PROBLEM - swiftac171 Current Load on swiftac171 is WARNING: LOAD WARNING - total load average: 11.26, 9.73, 5.30
[06:14:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.16, 23.57, 23.23
[06:15:46] <icinga-miraheze>	 RECOVERY - swiftac171 Current Load on swiftac171 is OK: LOAD OK - total load average: 3.76, 7.42, 4.97
[06:16:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.81, 24.05, 23.44
[06:17:03] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:17:26] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:18:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.60, 23.49, 23.31
[06:23:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:24:13] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.66, 4.10, 3.24
[06:24:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.90, 23.78, 23.16
[06:26:07] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.51, 3.65, 3.19
[06:28:01] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.15, 3.28, 3.12
[06:28:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:28:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.59, 22.22, 22.72
[06:33:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:33:44] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.45, 4.49, 3.57
[06:35:38] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.52, 3.75, 3.39
[06:36:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.01, 17.93, 20.40
[06:37:37] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.34, 4.38, 3.67
[06:38:20] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:39:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.31, 3.43, 3.40
[06:40:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.22, 20.01, 20.86
[06:41:36] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.60, 2.86, 3.20
[06:44:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.84, 23.82, 22.09
[06:45:14] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:48:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.04, 23.26, 22.35
[06:53:15] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.55, 3.50, 3.37
[06:53:20] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:55:09] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.27, 3.73, 3.45
[06:57:03] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.82, 3.13, 3.25
[07:04:39] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.23, 3.53, 3.33
[07:06:34] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.24, 3.55, 3.35
[07:08:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:08:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.95, 19.03, 20.29
[07:08:28] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.98, 3.68, 3.44
[07:10:21] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.63, 2.56, 3.06
[07:13:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:15:35] <icinga-miraheze>	 PROBLEM - wiki.moores.tech - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.moores.tech could not be found
[07:23:20] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:36:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.84, 21.29, 20.39
[07:38:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.16, 19.07, 19.67
[08:01:54] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.03, 21.09, 19.65
[08:07:38] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.20, 23.31, 21.21
[08:09:32] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.10, 23.64, 21.52
[08:11:27] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.21, 22.20, 21.25
[08:15:15] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 22.80, 21.65
[08:17:10] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.33, 22.41, 21.61
[08:18:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:19:04] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.41, 22.32, 21.62
[08:22:53] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.91, 22.97, 22.13
[08:28:36] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.92, 21.53, 21.47
[08:30:31] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.05, 21.11, 21.37
[08:44:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.34, 18.38, 19.93
[09:15:33] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[09:17:14] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.01, 19.89, 19.34
[09:19:08] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.75, 19.86, 19.42
[09:24:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.62, 20.71, 18.37
[09:24:52] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.46, 21.32, 20.14
[09:26:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.09, 20.75, 18.73
[09:28:41] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.40, 23.82, 21.30
[09:32:29] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.75, 22.01, 21.27
[09:32:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.18, 19.47, 18.98
[09:38:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 12.81, 18.48, 20.14
[09:45:04] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[10:04:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.88, 20.31, 18.58
[10:06:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.99, 19.66, 18.55
[10:17:12] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:22:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.65, 20.24, 19.00
[10:28:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.24, 20.09, 19.48
[10:36:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.03, 21.69, 20.24
[10:44:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.86, 19.60, 19.87
[10:47:09] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[11:03:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:03:09] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[11:03:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.42, 2.92, 1.23
[11:05:12] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 9.083 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[11:05:36] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.31, 2.70, 1.34
[11:07:44] <icinga-miraheze>	 PROBLEM - db171 Backups SQL on db171 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864197 seconds old and 139855 bytes
[11:08:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:18:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:25:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:25:36] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 12.88, 19.62, 23.89
[11:26:59] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.64, 4.06, 2.84
[11:27:06] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:29:05] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[11:30:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:32:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:34:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.63, 3.88, 3.46
[11:35:35] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.48, 21.46, 22.31
[11:36:37] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.46, 3.35, 3.33
[11:37:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:39:35] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.53, 22.05, 22.64
[11:43:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:45:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.14, 4.17, 3.65
[11:47:35] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.19, 21.53, 21.77
[11:48:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:49:00] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.65, 3.97, 3.70
[11:49:35] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.80, 19.10, 20.86
[11:50:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.68, 4.36, 3.85
[11:51:35] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.91, 17.04, 19.87
[11:52:49] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.96, 3.54, 3.60
[11:54:44] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.14, 3.00, 3.39
[11:56:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:57:33] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[11:58:12] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:58:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.29, 4.83, 4.06
[11:59:27] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.074 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[12:01:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:01:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:03:48] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[12:04:19] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[12:05:42] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.087 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[12:06:18] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.98, 20.55, 18.14
[12:08:18] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.94, 20.24, 18.33
[12:09:12] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.70, 21.73, 19.97
[12:11:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:12:16] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.69, 21.02, 19.18
[12:12:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:13:51] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[12:13:53] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[12:15:48] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0004437565804 secs
[12:16:15] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.89, 20.06, 19.23
[12:16:56] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.40, 23.33, 21.25
[12:20:47] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.41, 22.18, 21.28
[12:22:02] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.222 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[12:22:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:26:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.77, 19.17, 20.20
[12:27:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:32:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:34:26] <icinga-miraheze>	 PROBLEM - wiki.sukotuze.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sukotuze.com' expires in 15 day(s) (Tue 13 Aug 2024 12:17:15 PM GMT +0000).
[12:34:38] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c23a31f7f7e9...997fbd777b40
[12:34:41] <botifico>	 [02ssl] 07WikiTideSSLBot 03997fbd7 - Bot: Update SSL cert for wiki.sukotuze.com
[12:38:53] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:40:48] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[12:44:24] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[12:50:29] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.063 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[12:54:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.42, 19.92, 19.02
[12:57:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.59, 2.98, 4.00
[12:57:55] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.79, 20.34, 19.51
[13:00:06] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.20, 20.58, 19.22
[13:01:39] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.92, 3.51, 3.94
[13:02:06] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.08, 18.39, 18.58
[13:02:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:02:35] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[13:03:06] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:03:51] <icinga-miraheze>	 RECOVERY - wiki.sukotuze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sukotuze.com' will expire on Sat 26 Oct 2024 11:34:32 AM GMT +0000.
[13:04:32] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001434594393 secs
[13:05:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.85, 3.78, 3.99
[13:07:05] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[13:07:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:08:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.25, 18.02, 14.18
[13:09:10] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.98, 23.25, 18.61
[13:09:38] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.61, 3.82, 3.95
[13:09:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:10:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.49, 20.00, 15.34
[13:11:26] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:13:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.16, 22.16, 20.15
[13:13:23] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[13:13:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.26, 3.54, 3.83
[13:14:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:15:02] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.40, 20.46, 19.81
[13:15:40] <icinga-miraheze>	 PROBLEM - dlfm-wiki.top - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dlfm-wiki.top' expires in 15 day(s) (Tue 13 Aug 2024 12:58:51 PM GMT +0000).
[13:16:01] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/997fbd777b40...f09db3e6ba70
[13:16:02] <botifico>	 [02ssl] 07WikiTideSSLBot 03f09db3e - Bot: Update SSL cert for dlfm-wiki.top
[13:17:01] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.38, 18.87, 19.29
[13:17:37] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.58, 2.68, 3.39
[13:24:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.85, 22.31, 20.60
[13:26:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.14, 19.90, 19.93
[13:27:26] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:31:37] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.15, 3.97, 3.36
[13:32:26] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:33:37] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.46, 3.53, 3.30
[13:35:36] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.43, 3.03, 3.16
[13:37:25] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:39:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.36, 4.36, 3.66
[13:41:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.59, 3.56, 3.44
[13:42:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:43:36] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.00, 3.15, 3.29
[13:44:21] <icinga-miraheze>	 RECOVERY - dlfm-wiki.top - LetsEncrypt on sslhost is OK: OK - Certificate 'dlfm-wiki.top' will expire on Sat 26 Oct 2024 12:15:46 PM GMT +0000.
[13:47:06] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[13:55:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:57:18] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[13:57:39] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.66, 5.27, 3.81
[13:58:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.56, 17.60, 16.73
[13:59:13] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.076 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[14:00:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:00:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.94, 17.94, 16.99
[14:01:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.66, 3.68, 3.53
[14:07:36] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.70, 2.96, 3.28
[14:08:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.78, 18.37, 22.94
[14:14:46] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.18, 14.86, 19.83
[14:17:20] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:21:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.12, 4.08, 3.47
[14:22:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:23:37] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.87, 3.33, 3.25
[14:24:34] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:29:34] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:31:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:33:09] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[14:33:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.43, 4.48, 3.63
[14:35:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.00, 3.59, 3.39
[14:37:16] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 8.083 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[14:41:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:43:08] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:43:36] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.56, 3.27, 3.32
[14:45:17] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:48:07] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:48:08] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:50:03] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -3.844499588e-06 secs
[14:53:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:54:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.42, 3.59, 3.34
[14:56:07] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.97, 3.07, 3.16
[14:58:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:00:31] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:04:41] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.34, 3.85, 3.42
[15:06:35] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.81, 3.28, 3.28
[15:09:01] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:10:31] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:10:58] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001121193171 secs
[15:18:28] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.79, 20.82, 18.86
[15:20:27] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.83, 22.41, 19.64
[15:24:26] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.47, 23.19, 20.65
[15:30:24] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.67, 20.20, 20.32
[15:47:06] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:04:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.43, 18.64, 15.58
[16:07:07] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.92, 25.32, 21.67
[16:08:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.73, 19.99, 16.95
[16:15:04] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:15:57] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.64, 20.85, 18.93
[16:17:53] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.31, 20.80, 19.16
[16:19:49] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.09, 22.83, 20.12
[16:26:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.63, 20.42, 18.34
[16:31:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:34:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.97, 23.77, 20.68
[16:36:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.19, 22.82, 20.74
[16:38:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.46, 20.18, 20.03
[16:41:05] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.89, 22.02, 23.81
[16:50:57] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:52:41] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.70, 21.95, 22.64
[16:52:57] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001021295786 secs
[16:54:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.10, 21.33, 22.33
[16:54:46] <icinga-miraheze>	 PROBLEM - urbanshade.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'urbanshade.org' expires in 15 day(s) (Tue 13 Aug 2024 04:41:52 PM GMT +0000).
[16:54:56] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f09db3e6ba70...b4fb9eedefaa
[16:54:59] <botifico>	 [02ssl] 07WikiTideSSLBot 03b4fb9ee - Bot: Update SSL cert for urbanshade.org
[16:56:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.03, 24.05, 23.19
[16:58:26] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[17:00:28] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 397 system event log (SEL) entries present]
[17:00:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.21, 23.50, 23.32
[17:06:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:18:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.72, 18.79, 20.30
[17:19:40] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.99, 20.94, 19.41
[17:23:28] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 22.51, 20.36
[17:23:31] <icinga-miraheze>	 RECOVERY - urbanshade.org - LetsEncrypt on sslhost is OK: OK - Certificate 'urbanshade.org' will expire on Sat 26 Oct 2024 03:54:50 PM GMT +0000.
[17:25:23] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.46, 21.14, 20.08
[17:25:32] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.60, 23.95, 21.82
[17:29:12] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.03, 19.37, 19.69
[17:29:24] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.22, 22.62, 21.83
[17:44:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 21.01, 20.84
[17:46:48] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.91, 20.56, 20.69
[17:47:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.60, 20.13, 19.08
[17:48:44] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.30, 22.75, 21.45
[17:49:13] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.70, 22.25, 19.96
[17:50:40] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.40, 22.40, 21.45
[17:51:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.29, 22.38, 20.29
[17:52:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.80, 24.81, 22.46
[17:53:02] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.12, 23.67, 20.98
[17:54:57] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.40, 23.58, 21.28
[17:56:51] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.29, 25.13, 22.10
[17:58:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.49, 23.29, 22.82
[18:00:12] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:00:40] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.59, 21.77, 21.50
[18:02:14] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0002845823765 secs
[18:06:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 12.77, 17.69, 19.93
[18:06:26] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[18:08:28] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 399 system event log (SEL) entries present]
[18:08:38] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.32, 18.48, 20.37
[18:14:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.87, 20.31, 20.18
[18:18:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.59, 20.22, 20.31
[18:21:22] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.45, 21.33, 20.84
[18:23:18] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.26, 22.83, 21.42
[18:24:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.32, 22.67, 21.13
[18:29:26] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:32:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.21, 22.62, 22.07
[18:33:39] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:35:36] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0008054077625 secs
[18:36:50] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.36, 23.05, 23.55
[18:42:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.06, 18.01, 20.17
[18:45:25] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:47:22] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0006259679794 secs
[18:54:26] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[18:54:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.99, 23.22, 22.19
[18:56:28] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 401 system event log (SEL) entries present]
[18:58:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.70, 23.45, 22.62
[19:06:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.24, 22.65, 22.40
[19:18:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.51, 19.88, 18.74
[19:20:02] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.64, 21.34, 19.38
[19:21:56] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.89, 20.91, 19.47
[19:22:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:23:51] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.13, 19.00, 18.93
[19:31:29] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.02, 20.98, 19.86
[19:33:24] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.25, 19.00, 19.31
[19:42:26] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[19:44:28] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 403 system event log (SEL) entries present]
[19:47:26] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:47:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:57:20] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:59:27] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[20:01:34] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:03:30] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001214742661 secs
[20:04:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.93, 21.39, 23.97
[20:10:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.35, 21.70, 23.22
[20:12:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.43, 20.02, 22.44
[20:15:06] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[20:16:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.56, 23.97, 23.43
[20:19:14] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.38, 20.94, 19.99
[20:21:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.40, 22.43, 20.64
[20:23:03] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.47, 22.56, 20.94
[20:28:46] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.61, 23.15, 21.54
[20:30:40] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.27, 20.78, 20.85
[20:34:29] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.20, 18.09, 19.81
[20:34:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:36:34] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[20:38:32] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 405 system event log (SEL) entries present]
[20:38:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.69, 21.73, 23.69
[20:52:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.39, 22.67, 22.46
[20:54:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.45, 22.14, 22.31
[21:03:37] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.10, 2.51, 1.05
[21:04:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:04:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.50, 22.32, 21.78
[21:06:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.73, 21.92, 18.66
[21:06:59] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[21:07:09] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[21:08:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.98, 19.17, 18.07
[21:08:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.09, 23.25, 22.37
[21:09:06] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.190 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[21:09:11] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures
[21:09:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:10:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.71, 23.68, 22.60
[21:12:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.68, 22.80, 22.40
[21:13:37] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.06, 3.61, 2.69
[21:14:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:15:37] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.92, 3.01, 2.57
[21:16:06] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:17:25] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:18:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.80, 22.97, 22.48
[21:22:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:23:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.08, 3.64, 3.19
[21:24:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:25:37] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.58, 4.40, 3.52
[21:26:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.46, 22.15, 19.93
[21:27:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.00, 3.71, 3.39
[21:29:37] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.11, 4.10, 3.56
[21:29:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:30:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 7.87, 16.32, 18.25
[21:30:26] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[21:32:12] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:32:28] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 407 system event log (SEL) entries present]
[21:35:36] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.24, 2.82, 3.38
[21:37:12] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:44:24] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[21:45:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:49:14] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.38, 21.94, 20.20
[21:51:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.91, 21.32, 20.20
[21:53:03] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.87, 22.02, 20.54
[21:54:57] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.48, 23.13, 21.16
[21:58:46] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.83, 19.31, 20.08
[22:10:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:14:02] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.18, 20.39, 19.90
[22:15:56] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.42, 19.18, 19.54
[22:25:29] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.23, 19.83, 19.18
[22:27:23] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.49, 21.29, 19.77
[22:29:18] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.97, 19.47, 19.31
[22:40:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:05:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:06:25] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.93, 20.37, 18.23
[23:08:25] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 11.19, 16.73, 17.15
[23:08:26] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[23:10:28] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 411 system event log (SEL) entries present]
[23:10:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:11:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.18, 3.70, 2.07
[23:13:06] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.63, 2.56, 1.84
[23:15:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:20:57] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.67, 19.32, 18.18
[23:22:51] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.25, 17.36, 17.62
[23:25:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:35:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:40:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.39, 20.63, 23.39
[23:44:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.19, 21.78, 23.08
[23:45:30] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:48:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.81, 22.89, 23.29
[23:50:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 23.05, 23.27
[23:52:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.81, 22.25, 22.99
[23:54:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 22.36, 22.88
[23:56:26] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[23:56:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.88, 22.11, 22.71
[23:58:28] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 413 system event log (SEL) entries present]