[00:01:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.45, 3.23, 3.54
[00:03:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.67, 4.27, 3.89
[00:05:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.69, 3.54, 3.67
[00:10:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.81, 21.61, 23.76
[00:12:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.82, 22.42, 23.81
[00:13:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.32, 3.29, 3.46
[00:18:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.18, 22.38, 23.22
[00:19:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.13, 3.70, 3.66
[00:20:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.18, 22.10, 23.01
[00:25:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.87, 2.73, 3.28
[00:28:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.07, 22.46, 22.60
[00:30:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.64, 22.05, 22.38
[00:35:41] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.11, 3.99, 3.63
[00:37:40] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.04, 4.43, 3.82
[00:44:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.22, 19.20, 20.34
[00:45:35] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.52, 3.87, 3.91
[00:47:34] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.58, 4.66, 4.20
[00:48:23] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:48:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.20, 22.18, 22.08
[00:50:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.72, 22.58, 21.41
[00:51:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.51, 19.15, 17.74
[00:52:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.35, 20.88, 20.94
[00:52:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.53, 23.10, 22.52
[00:53:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.93, 17.77, 17.41
[00:56:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.71, 19.34, 20.30
[00:58:50] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[00:59:25] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.56, 3.58, 3.96
[01:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.98, 21.28, 20.92
[01:01:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:02:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.04, 23.61, 22.80
[01:03:24] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.96, 4.18, 4.10
[01:04:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.61, 22.68, 22.57
[01:05:23] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.16, 3.67, 3.94
[01:08:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.93, 18.78, 19.98
[01:09:21] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.51, 4.26, 4.05
[01:12:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.36, 22.43, 22.45
[01:13:18] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.18, 3.82, 4.00
[01:14:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.28, 21.00, 20.38
[01:14:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.61, 22.15, 22.33
[01:16:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.80, 21.98, 20.82
[01:16:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.20, 23.32, 22.74
[01:18:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.52, 23.20, 22.77
[01:19:14] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.50, 3.75
[01:23:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.68, 3.69, 3.81
[01:24:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.22, 19.78, 20.39
[01:27:09] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.32, 3.25, 3.54
[01:28:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.31, 20.22, 20.48
[01:29:08] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.19, 3.45, 3.60
[01:31:08] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.05, 3.97, 3.77
[01:36:53] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:41:01] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.70, 3.51, 3.80
[01:42:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.24, 19.03, 19.98
[01:43:01] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.121 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[01:46:58] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.45, 4.39, 4.02
[01:50:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.75, 21.36, 20.57
[01:50:54] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:51:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:52:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.74, 19.69, 20.04
[01:53:16] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[02:01:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:12:59] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.13, 20.00, 20.34
[02:16:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.83, 21.30, 20.16
[02:16:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.30, 22.46, 21.21
[02:18:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.41, 22.45, 20.75
[02:18:57] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.88, 18.26, 16.40
[02:18:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.56, 22.10, 21.24
[02:20:55] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.06, 17.65, 16.42
[02:23:05] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[02:24:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.92, 22.99, 21.74
[02:25:23] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[02:25:29] <botifico>	 [02mediawiki-repos] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/0f3d78436495...3d65b9b1d804
[02:25:32] <botifico>	 [02mediawiki-repos] 07AverageHelper 033d65b9b - T12447: Add GoogleForms (#30)
[02:25:32] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[02:25:33] <botifico>	 [02mediawiki-repos] 07OAuthority closed pull request 03#30: T12447: Add GoogleForms - 13https://github.com/miraheze/mediawiki-repos/pull/30
[02:26:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:26:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.28, 23.17, 21.98
[02:27:07] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.071 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[02:27:19] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[02:27:33] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures
[02:31:30] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:33:59] <botifico>	 [02mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8f23490b9596...b194aea19ac3
[02:34:02] <botifico>	 [02mw-config] 07anpang54 03b194aea - Remove spacewiki code in LocalWiki.php (#5637)
[02:34:03] <botifico>	 [02mw-config] 07OAuthority closed pull request 03#5637: Remove spacewiki code in LocalWiki.php - 13https://github.com/miraheze/mw-config/pull/5637
[02:35:00] <miraheze-github>	 miraheze/mw-config - OAuthority the build passed.
[02:35:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.55, 19.85, 18.01
[02:37:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.23, 19.08, 17.95
[02:37:40] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[02:37:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:37:55] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 15s
[02:38:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:42:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.01, 22.15, 21.90
[02:44:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.21, 21.23, 21.54
[02:46:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.40, 19.52, 20.25
[02:50:19] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.31, 2.83, 3.97
[02:52:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.19, 20.73, 20.52
[02:52:19] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.61, 3.90, 4.22
[02:52:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 21.46, 21.22
[02:53:19] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[02:53:19] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[02:53:30] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[02:53:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:53:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:54:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.95, 21.63, 21.32
[02:57:31] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.073 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[02:58:17] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.23, 3.29, 3.90
[03:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.03, 21.75, 20.93
[03:00:05] <icinga-miraheze>	 RECOVERY - db171 Backups SQL on db171 is OK: FILE_AGE OK: /var/log/sql-backup.log is 4 seconds old and 0 bytes
[03:02:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.44, 21.13, 20.83
[03:02:03] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[03:02:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:02:27] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 23s
[03:02:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:06:08] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[03:06:14] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.95, 4.23, 4.08
[03:06:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:08:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.23, 18.76, 19.98
[03:08:06] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 1.612 second response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[03:11:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:16:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.00, 22.50, 21.38
[03:18:07] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.84, 3.23, 3.90
[03:18:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.61, 21.90, 21.31
[03:22:05] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.16, 3.79, 3.94
[03:22:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.26, 18.88, 17.19
[03:24:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.42, 17.70, 16.95
[03:26:01] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.93, 3.54, 3.83
[03:28:01] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.62, 4.38, 4.06
[03:29:34] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.21, 21.58, 20.48
[03:31:57] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.61, 3.85, 3.97
[03:32:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.03, 19.40, 17.65
[03:32:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.08, 23.89, 22.43
[03:33:56] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.30, 3.67, 3.86
[03:34:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.12, 18.83, 17.65
[03:34:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.22, 23.49, 22.46
[03:35:54] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.03, 3.12, 3.62
[03:36:30] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:37:54] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[03:37:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.42, 4.30, 3.99
[03:39:54] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 3.603 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[03:39:54] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.25, 3.74, 3.82
[03:43:11] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.91, 19.77, 20.23
[03:49:49] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.81, 3.64, 3.67
[03:50:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.64, 22.62, 22.16
[03:52:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.53, 21.56, 21.82
[03:54:12] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[03:54:22] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[03:56:08] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[03:56:23] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures
[03:56:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:00:12] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[04:01:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:04:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.74, 22.94, 22.13
[04:06:26] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.97, 20.55, 19.84
[04:06:27] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 8.377 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[04:16:10] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.52, 23.22, 21.60
[04:16:17] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.77, 6.36, 6.79
[04:17:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.44, 2.90, 3.94
[04:18:06] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.59, 22.70, 21.59
[04:19:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.49, 4.21, 4.28
[04:23:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.46, 3.19, 3.89
[04:24:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.32, 22.66, 21.83
[04:24:17] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.84, 6.75, 6.73
[04:25:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.48, 4.04, 4.11
[04:28:17] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.63, 6.28, 6.55
[04:30:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.72, 22.55, 22.37
[04:32:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.30, 24.62, 23.15
[04:32:09] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[04:34:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.67, 23.58, 22.96
[04:34:04] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.211 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[04:45:11] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.14, 6.76, 6.54
[04:49:09] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 6.25, 6.59, 6.54
[04:50:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.03, 21.87, 21.35
[04:51:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:52:17] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:52:26] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:54:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.13, 22.00, 21.63
[04:55:08] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[04:56:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:56:38] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[05:05:58] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.23, 6.72, 6.55
[05:07:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.34, 2.46, 3.93
[05:07:57] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.97, 6.39, 6.45
[05:10:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.86, 19.08, 20.38
[05:11:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.05, 1.14, 3.05
[05:14:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.41, 23.52, 21.95
[05:18:49] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.55, 6.99, 6.73
[05:21:29] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.65, 21.25, 18.07
[05:22:26] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.34, 21.37, 18.61
[05:23:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.86, 20.58, 18.34
[05:23:27] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.36, 21.14, 18.45
[05:24:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.45, 20.88, 17.92
[05:24:46] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.91, 6.75, 6.76
[05:25:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.67, 20.31, 18.55
[05:26:17] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.92, 21.40, 19.23
[05:28:13] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.49, 20.13, 18.98
[05:28:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.22, 19.22, 18.00
[05:31:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.39, 21.46, 19.63
[05:31:22] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.49, 20.16, 19.16
[05:33:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.46, 24.02, 20.79
[05:33:28] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.43, 22.17, 18.68
[05:34:01] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.63, 23.74, 20.71
[05:34:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.57, 21.48, 19.36
[05:35:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 18.04, 20.91, 20.00
[05:35:17] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.32, 22.11, 20.32
[05:35:24] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.88, 20.85, 18.62
[05:35:57] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.79, 22.00, 20.43
[05:37:20] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.58, 18.65, 18.09
[05:40:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.77, 20.11, 19.51
[05:41:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.20, 19.40, 19.87
[05:41:13] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.40, 18.75, 19.55
[05:41:36] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.24, 6.94, 6.63
[05:41:44] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.95, 19.68, 20.02
[05:45:34] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 6.70, 6.80, 6.65
[05:47:07] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.64, 22.70, 20.87
[05:47:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.48, 22.80, 21.03
[05:47:19] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.88, 23.98, 21.16
[05:47:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.07, 23.64, 21.49
[05:49:05] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.50, 22.33, 20.99
[05:49:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.96, 20.95, 20.53
[05:49:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.93, 21.57, 20.62
[05:49:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.59, 22.15, 21.19
[05:51:07] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.12, 20.22, 20.25
[05:51:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.55, 19.12, 19.93
[05:52:00] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:53:03] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.37, 20.09, 20.38
[05:53:29] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.05, 6.94, 6.75
[05:53:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.78, 18.58, 19.93
[05:54:01] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001596301794 secs
[05:59:26] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.04, 6.55, 6.72
[06:11:19] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.87, 6.72, 6.50
[06:15:17] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.79, 6.45, 6.47
[06:17:42] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.59, 19.77, 19.05
[06:17:47] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:19:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.27, 18.87, 18.82
[06:24:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.66, 19.86, 17.37
[06:25:11] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.44, 6.71, 6.50
[06:26:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.93, 18.69, 17.25
[06:29:09] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.07, 6.36, 6.45
[06:34:12] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.89, 20.78, 19.43
[06:34:22] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.98, 20.74, 19.40
[06:34:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.04, 20.76, 18.36
[06:35:06] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.65, 21.24, 19.74
[06:36:07] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 20.17, 20.31, 19.40
[06:36:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 17.39, 19.81, 18.33
[06:37:00] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.40, 19.95, 19.48
[06:38:11] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.20, 18.85, 18.96
[06:45:05] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[06:45:47] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.26, 20.52, 19.33
[06:45:47] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.05, 21.24, 19.40
[06:46:28] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:47:06] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001714408398 secs
[06:47:41] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.01, 21.90, 19.89
[06:49:35] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.35, 20.31, 19.53
[06:51:34] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.93, 19.11, 19.24
[06:55:08] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[06:59:34] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001710087061 secs
[07:03:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.88, 3.24, 1.32
[07:04:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.44, 22.83, 23.86
[07:07:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.63, 2.55, 1.57
[07:08:04] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.87, 23.34, 23.63
[07:10:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.59, 22.96, 23.52
[07:12:43] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.16, 6.72, 6.34
[07:12:54] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw181
[07:12:54] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw181
[07:13:05] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181
[07:14:53] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[07:14:54] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy
[07:15:05] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[07:17:27] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:20:10] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:22:16] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.00180748105 secs
[07:24:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.68, 23.77, 23.47
[07:26:36] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.78, 6.76, 6.75
[07:27:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.13, 19.10, 17.89
[07:31:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.12, 21.14, 19.11
[07:32:01] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.06, 21.56, 19.33
[07:32:04] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.08, 22.23, 19.59
[07:33:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.68, 19.33, 18.56
[07:33:40] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 12.56, 18.08, 18.24
[07:33:55] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.64, 19.32, 18.77
[07:33:58] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.74, 19.87, 19.07
[07:34:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.25, 23.01, 23.88
[07:34:31] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.89, 6.87, 6.78
[07:39:07] <botifico>	 [02mw-config] 07BlankEclair opened pull request 03#5639: Add incompatibility warning for HeaderTabs - 13https://github.com/miraheze/mw-config/pull/5639
[07:40:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.60, 24.24, 24.03
[07:40:10] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[07:41:35] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.66, 21.43, 19.54
[07:43:25] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.78, 20.28, 19.19
[07:43:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.18, 21.06, 19.61
[07:44:04] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.76, 23.19, 23.81
[07:45:19] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.34, 19.52, 19.05
[07:45:24] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.73, 19.90, 19.34
[07:46:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 22.97, 23.60
[07:48:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.41, 22.63, 23.37
[07:54:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.18, 23.31, 23.39
[07:54:54] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.67, 19.44, 17.69
[07:55:06] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181
[07:56:49] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.31, 18.44, 17.54
[07:57:03] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[08:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.29, 23.60, 23.56
[08:00:38] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.86, 20.63, 19.86
[08:02:32] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.24, 19.83, 19.67
[08:08:17] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.58, 6.21, 6.70
[08:12:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.47, 22.95, 22.89
[08:15:04] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:16:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.61, 23.82, 23.35
[08:26:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.82, 24.31, 23.73
[08:33:31] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.44, 20.89, 18.57
[08:35:27] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.55, 21.79, 19.20
[08:35:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.77, 20.56, 18.70
[08:37:22] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.08, 19.30, 18.63
[08:37:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.41, 19.53, 18.53
[08:43:11] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.18, 20.48, 19.19
[08:43:55] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.39, 21.14, 19.37
[08:44:51] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.25, 22.01, 19.21
[08:45:07] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.56, 19.45, 18.99
[08:46:45] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.79, 21.50, 19.43
[08:48:39] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.37, 19.35, 18.91
[08:48:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.18, 20.44, 19.35
[08:49:37] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.81, 19.35, 19.32
[08:53:26] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query.
[08:54:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 22.77, 20.65
[08:54:47] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.71, 20.49, 19.28
[08:55:20] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.63, 22.66, 20.60
[08:56:07] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 17.86, 20.66, 19.70
[08:56:30] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:56:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 18.68, 21.40, 20.42
[08:56:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.92, 19.70, 19.10
[08:57:14] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.71, 21.86, 20.54
[08:58:02] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.73, 19.96, 19.55
[08:59:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.19, 19.39, 19.79
[09:00:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.16, 18.87, 19.74
[09:08:17] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.87, 6.21, 5.73
[09:14:17] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.68, 5.89, 5.83
[09:17:26] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[09:21:34] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.15, 3.83, 2.40
[09:23:33] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.40, 3.22, 2.33
[09:23:35] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.29, 19.57, 18.93
[09:29:30] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.02, 18.50, 18.80
[09:31:30] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.37, 4.52, 3.16
[09:32:23] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.56, 20.22, 18.87
[09:34:18] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 17.43, 19.49, 18.78
[09:35:27] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.33, 3.60, 3.12
[09:37:26] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.72, 4.05, 3.34
[09:38:02] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[09:39:31] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.57, 20.81, 19.67
[09:40:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.93, 19.96, 19.12
[09:40:55] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.54, 20.18, 18.81
[09:42:28] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.91, 20.09, 19.30
[09:43:19] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.11, 20.05, 19.69
[09:44:09] <botifico>	 [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9642611dc63a
[09:44:12] <botifico>	 [02puppet] 07Reception123 039642611 - create new deploy test user for T12486
[09:44:15] <botifico>	 [02puppet] 07Reception123 created branch 03Reception123-patch-3 - 13https://github.com/miraheze/puppet
[09:44:17] <botifico>	 [02puppet] 07Reception123 opened pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897
[09:44:18] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897#issuecomment-2295198194
[09:44:44] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.82, 19.08, 18.70
[09:44:54] <botifico>	 [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9642611dc63a...bad649d55fb3
[09:44:57] <botifico>	 [02puppet] 07Reception123 03bad649d - Update test151.yaml
[09:45:00] <botifico>	 [02puppet] 07Reception123 synchronize pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897
[09:45:20] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.64, 3.88, 3.66
[09:45:21] <botifico>	 [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bad649d55fb3...7cd7c3df7a5a
[09:45:23] <botifico>	 [02puppet] 07Reception123 037cd7c3d - Update data.yaml
[09:45:24] <botifico>	 [02puppet] 07Reception123 synchronize pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897
[09:45:38] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[09:46:15] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.060 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[09:47:19] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.37, 3.68, 3.59
[09:47:33] <botifico>	 [02puppet] 07Reception123 synchronize pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897
[09:47:34] <botifico>	 [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7cd7c3df7a5a...73edcfd2f068
[09:47:35] <botifico>	 [02puppet] 07Reception123 0373edcfd - Update data.yaml
[09:48:32] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.09, 21.68, 19.89
[09:49:18] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.57, 3.06, 3.36
[09:50:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.53, 19.87, 19.44
[09:53:26] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found
[09:57:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.58, 20.63, 19.88
[09:57:19] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.42, 21.55, 20.10
[09:58:05] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.50, 21.43, 19.80
[09:59:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.40, 18.97, 19.39
[09:59:13] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.11, 19.93, 19.64
[09:59:45] <botifico>	 [02puppet] 07Reception123 synchronize pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897
[09:59:48] <botifico>	 [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/73edcfd2f068...d1948b7cb4ab
[09:59:51] <botifico>	 [02puppet] 07Reception123 03d1948b7 - Update data.yaml
[10:01:25] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[10:01:30] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:02:02] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.05, 17.93, 18.79
[10:02:07] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.65, 3.91, 3.48
[10:04:06] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.91, 3.92, 3.51
[10:05:30] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 3.056 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[10:06:30] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:09:53] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[10:11:54] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 4.691 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[10:14:01] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.54, 3.83, 3.94
[10:16:00] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.44, 4.13, 4.06
[10:16:30] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:21:56] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.70, 3.84, 3.97
[10:25:17] <botifico>	 [02puppet] 07redbluegreenhat reviewed pull request 03#3897 commit - 13https://github.com/miraheze/puppet/pull/3897#discussion_r1720944572
[10:27:52] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.96, 3.76, 3.79
[10:27:56] <botifico>	 [02puppet] 07redbluegreenhat commented on pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897#issuecomment-2295210896
[10:29:50] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.12, 3.49, 3.71
[10:30:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.56, 21.26, 23.47
[10:34:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.29, 23.04, 23.65
[10:35:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.88, 3.49, 3.60
[10:36:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.49, 23.52, 23.79
[10:37:47] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.51, 2.89, 3.35
[10:42:42] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:42:49] <botifico>	 [02puppet] 07Reception123 synchronize pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897
[10:42:50] <botifico>	 [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d1948b7cb4ab...6b08e54ac08d
[10:42:52] <botifico>	 [02puppet] 07Reception123 036b08e54 - Update data.yaml
[10:44:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.15, 22.31, 22.93
[10:44:48] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0008567273617 secs
[10:45:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.37, 3.65, 3.43
[10:48:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:48:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.21, 23.29, 23.24
[10:49:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.85, 3.58, 3.50
[10:50:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.75, 23.72, 23.37
[10:51:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.28, 3.52, 3.46
[10:51:53] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.10, 21.51, 18.49
[10:52:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.81, 21.96, 18.39
[10:52:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.88, 21.20, 17.28
[10:52:46] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw181 mw182
[10:53:12] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw182
[10:53:21] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.98, 23.17, 19.29
[10:53:47] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.00, 22.30, 19.18
[10:53:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.66, 3.19, 3.33
[10:54:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.10, 23.04, 23.30
[10:54:03] <botifico>	 [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6b08e54ac08d...2ac3e0288360
[10:54:04] <botifico>	 [02puppet] 07Reception123 032ac3e02 - nano is dangerous
[10:54:06] <botifico>	 [02puppet] 07Reception123 synchronize pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897
[10:54:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.59, 21.42, 18.67
[10:54:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.19, 19.63, 17.19
[10:54:45] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[10:55:11] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[10:55:19] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 18.46, 21.27, 19.07
[10:55:41] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.95, 19.87, 18.64
[10:55:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.04, 3.92, 3.58
[10:56:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.94, 19.86, 18.46
[10:57:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.96, 3.42, 3.46
[10:58:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.47, 25.26, 24.02
[10:59:37] <botifico>	 [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2ac3e0288360...119e38e4f693
[10:59:40] <botifico>	 [02puppet] 07Reception123 03119e38e - Update data.yaml
[10:59:41] <botifico>	 [02puppet] 07Reception123 synchronize pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897
[10:59:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.51, 3.06, 3.34
[11:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.91, 23.24, 23.43
[11:01:15] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.35, 19.40, 19.07
[11:07:08] <botifico>	 [02puppet] 07Reception123 synchronize pull request 03#3897: create new deploy test user for T12486 - 13https://github.com/miraheze/puppet/pull/3897
[11:07:10] <botifico>	 [02puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/119e38e4f693...e45a214f76d6
[11:07:12] <botifico>	 [02puppet] 07Reception123 03e45a214 - Update data.yaml
[11:10:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.32, 24.61, 23.53
[11:10:07] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.41, 20.02, 18.02
[11:10:39] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[11:12:03] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.49, 19.80, 18.18
[11:12:40] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0002812743187 secs
[11:13:45] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.52, 20.73, 19.20
[11:14:03] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.85, 20.78, 19.34
[11:15:39] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.88, 20.53, 19.30
[11:17:51] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.38, 22.21, 19.59
[11:19:11] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:19:27] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.64, 19.86, 19.37
[11:19:59] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.11, 22.88, 20.69
[11:21:12] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0006095468998 secs
[11:21:42] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 17.30, 20.91, 19.74
[11:21:57] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.06, 21.89, 20.58
[11:22:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.00, 20.46, 19.05
[11:23:16] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.43, 21.64, 20.28
[11:26:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.80, 22.55, 20.15
[11:27:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.23, 20.21, 20.05
[11:28:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.88, 21.86, 20.18
[11:29:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.27, 22.61, 21.01
[11:29:52] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.20, 22.34, 21.19
[11:30:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.44, 23.01, 20.79
[11:31:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.81, 21.57, 20.84
[11:31:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 18.45, 20.48, 20.28
[11:32:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.29, 21.74, 20.59
[11:33:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.49, 21.54, 20.66
[11:33:49] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.81, 23.56, 21.99
[11:35:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.47, 20.34, 20.32
[11:36:54] <botifico>	 [02puppet] 07BlankEclair reviewed pull request 03#3897 commit - 13https://github.com/miraheze/puppet/pull/3897#discussion_r1720956000
[11:36:56] <botifico>	 [02puppet] 07BlankEclair reviewed pull request 03#3897 commit - 13https://github.com/miraheze/puppet/pull/3897#discussion_r1720956037
[11:37:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.31, 18.47, 19.84
[11:45:40] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.44, 18.74, 20.24
[11:46:48] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/sysctl.d]
[11:49:36] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.14, 20.85, 20.85
[11:57:04] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.54, 6.68, 6.07
[11:57:30] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.85, 19.79, 20.39
[11:58:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.53, 19.38, 20.34
[11:59:03] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 6.23, 6.56, 6.11
[12:09:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.05, 21.54, 19.44
[12:09:13] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.08, 22.37, 20.65
[12:09:18] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.03, 23.21, 21.37
[12:10:07] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.19, 24.08, 20.86
[12:10:46] <icinga-miraheze>	 PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received
[12:10:58] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.46, 21.50, 18.31
[12:11:05] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181
[12:11:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.73, 21.47, 16.35
[12:12:46] <icinga-miraheze>	 RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3820 bytes in 1.203 second response time
[12:12:54] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.51, 22.14, 18.94
[12:13:01] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.32, 23.15, 21.56
[12:13:05] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[12:13:08] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 12.79, 18.43, 15.90
[12:13:15] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.34, 23.27, 21.85
[12:14:21] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[12:15:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.49, 23.61, 21.34
[12:15:14] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 23.68, 24.12, 22.38
[12:15:54] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 17.01, 21.65, 21.04
[12:16:21] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001378059387 secs
[12:16:45] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.19, 18.32, 18.14
[12:19:11] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 17.37, 21.40, 21.72
[12:19:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.68, 18.57, 19.77
[12:20:38] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.20, 18.94, 20.25
[12:21:41] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.75, 19.23, 20.16
[12:24:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.49, 20.21, 20.57
[12:26:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.26, 22.32, 21.29
[12:27:05] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.23, 21.82, 21.37
[12:27:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.47, 20.17, 19.85
[12:28:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.84, 21.32, 21.11
[12:29:04] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 18.65, 20.43, 20.92
[12:29:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.78, 19.08, 19.48
[12:31:02] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.16, 22.94, 21.80
[12:32:25] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.77, 21.96, 21.03
[12:33:01] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 15.71, 20.36, 21.01
[12:36:16] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.28, 19.68, 20.28
[12:36:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.04, 18.09, 19.82
[12:36:58] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.77, 18.63, 20.09
[12:39:02] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[12:41:02] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.000256717205 secs
[12:45:40] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[12:46:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.98, 19.69, 19.18
[12:48:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.80, 20.74, 19.58
[12:49:44] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.18, 21.77, 20.84
[12:50:00] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.50, 20.03, 19.51
[12:50:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.09, 19.35, 19.22
[12:53:48] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.93, 22.22, 18.78
[12:53:49] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.33, 23.44, 20.88
[12:54:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 16.50, 20.48, 19.91
[12:54:38] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.81, 22.89, 21.20
[12:55:40] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.01, 21.62, 21.24
[12:55:43] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.34, 21.54, 20.49
[12:55:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.33, 20.61, 18.64
[12:56:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 20.33, 20.15, 19.84
[12:57:37] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.68, 23.10, 21.14
[12:57:39] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.88, 22.73, 21.64
[12:58:11] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:58:29] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.19, 23.49, 21.66
[12:59:35] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.47, 23.45, 20.11
[12:59:56] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw182
[13:00:11] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001214742661 secs
[13:00:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.60, 21.86, 20.80
[13:01:31] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.61, 22.08, 20.02
[13:01:41] <botifico>	 [02CreateWiki] 07RhinosF1 commented on pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546#issuecomment-2295254753
[13:01:45] <botifico>	 [02CreateWiki] 07coderabbitai[bot] commented on pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546#issuecomment-2295254770
[13:01:47] <botifico>	 [02CreateWiki] 07coderabbitai[bot] edited a comment on pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546#issuecomment-2294957983
[13:01:51] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[13:02:20] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.14, 22.99, 22.00
[13:03:19] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 16.10, 21.81, 21.58
[13:03:26] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.48, 19.48, 19.30
[13:03:34] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 16.37, 21.60, 21.86
[13:03:39] <botifico>	 [02CreateWiki] 07coderabbitai[bot] edited pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546
[13:04:28] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.20, 6.32, 5.57
[13:05:16] <botifico>	 [02CreateWiki] 07coderabbitai[bot] reviewed pull request 03#546 commit - 13https://github.com/miraheze/CreateWiki/pull/546#discussion_r1720972989
[13:05:18] <botifico>	 [02CreateWiki] 07coderabbitai[bot] reviewed pull request 03#546 commit - 13https://github.com/miraheze/CreateWiki/pull/546#discussion_r1720972992
[13:05:20] <botifico>	 [02CreateWiki] 07coderabbitai[bot] edited a comment on pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546#issuecomment-2294957983
[13:05:24] <miraheze-github>	 miraheze/CreateWiki - pixDeVl the build has errored.
[13:05:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.66, 2.59, 1.11
[13:06:12] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.25, 18.59, 20.38
[13:06:27] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 6.39, 6.27, 5.65
[13:06:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.04, 18.52, 19.81
[13:07:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.89, 17.81, 19.93
[13:07:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.53, 1.74, 0.98
[13:08:47] <icinga-miraheze>	 PROBLEM - www.streamerlore.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.streamerlore.com' expires in 15 day(s) (Tue 03 Sep 2024 12:50:20 PM GMT +0000).
[13:08:58] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a3f26741b8e0...8c7418a41145
[13:09:00] <botifico>	 [02ssl] 07WikiTideSSLBot 038c7418a - Bot: Update SSL cert for www.streamerlore.com
[13:11:01] <icinga-miraheze>	 PROBLEM - wiki.routedoodle.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.routedoodle.com' expires in 15 day(s) (Tue 03 Sep 2024 12:54:01 PM GMT +0000).
[13:11:11] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8c7418a41145...c494b67b5779
[13:11:14] <botifico>	 [02ssl] 07WikiTideSSLBot 03c494b67 - Bot: Update SSL cert for wiki.routedoodle.com
[13:11:28] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.95, 17.71, 19.84
[13:16:19] <botifico>	 [02CreateWiki] 07pixDeVl synchronize pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546
[13:16:24] <botifico>	 [02CreateWiki] 07coderabbitai[bot] edited a comment on pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546#issuecomment-2294957983
[13:17:11] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[13:17:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.76, 20.49, 20.04
[13:17:23] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.16, 22.92, 21.16
[13:18:35] <botifico>	 [02CreateWiki] 07RhinosF1 commented on pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546#issuecomment-2295260210
[13:18:47] <botifico>	 [02CreateWiki] 07coderabbitai[bot] commented on pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546#issuecomment-2295260286
[13:18:48] <botifico>	 [02CreateWiki] 07coderabbitai[bot] edited a comment on pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546#issuecomment-2294957983
[13:19:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.75, 19.53, 19.78
[13:19:21] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.46, 19.85, 20.26
[13:20:24] <botifico>	 [02CreateWiki] 07coderabbitai[bot] edited pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546
[13:20:26] <botifico>	 [02CreateWiki] 07coderabbitai[bot] edited a comment on pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546#issuecomment-2294957983
[13:22:15] <icinga-miraheze>	 PROBLEM - cpmusicwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cpmusicwiki.com' expires in 15 day(s) (Tue 03 Sep 2024 12:51:15 PM GMT +0000).
[13:22:18] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.00, 6.21, 5.70
[13:22:29] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c494b67b5779...b401ada8cf31
[13:22:31] <botifico>	 [02ssl] 07WikiTideSSLBot 03b401ada - Bot: Update SSL cert for cpmusicwiki.com
[13:23:26] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query.
[13:24:17] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.00, 5.51, 5.52
[13:35:35] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:36:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.49, 20.31, 23.37
[13:37:35] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001057893038 secs
[13:38:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:38:05] <icinga-miraheze>	 RECOVERY - www.streamerlore.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.streamerlore.com' will expire on Sat 16 Nov 2024 12:10:22 PM GMT +0000.
[13:44:11] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[13:51:29] <icinga-miraheze>	 RECOVERY - cpmusicwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'cpmusicwiki.com' will expire on Sat 16 Nov 2024 12:23:53 PM GMT +0000.
[13:53:26] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found
[13:54:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.35, 19.51, 20.33
[13:58:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.63, 21.35, 20.88
[14:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.97, 22.79, 21.46
[14:02:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 14.48, 19.49, 20.41
[14:04:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.87, 17.67, 19.63
[14:09:01] <icinga-miraheze>	 RECOVERY - wiki.routedoodle.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.routedoodle.com' will expire on Sat 16 Nov 2024 12:12:36 PM GMT +0000.
[14:10:17] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.65, 6.59, 5.82
[14:24:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.33, 20.31, 16.28
[14:24:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.47, 19.33, 15.36
[14:24:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.07, 22.86, 23.92
[14:25:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.69, 18.64, 15.31
[14:26:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.40, 22.15, 19.52
[14:26:17] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.05, 6.26, 6.40
[14:26:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.67, 20.67, 16.92
[14:26:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 20.07, 19.57, 15.96
[14:26:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.72, 24.62, 24.41
[14:27:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.37, 17.26, 15.21
[14:28:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.73, 22.43, 19.98
[14:28:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.74, 18.55, 16.58
[14:30:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.70, 22.37, 23.59
[14:32:03] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.73, 19.73, 19.42
[14:32:17] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.29, 6.70, 6.51
[14:50:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.66, 23.48, 22.81
[14:52:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.07, 22.35, 22.50
[14:54:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 23.60, 22.96
[14:56:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.77, 23.16, 22.85
[15:02:22] <botifico>	 [02CreateWiki] 07RhinosF1 reviewed pull request 03#546 commit - 13https://github.com/miraheze/CreateWiki/pull/546#discussion_r1720995872
[15:02:34] <botifico>	 [02CreateWiki] 07coderabbitai[bot] reviewed pull request 03#546 commit - 13https://github.com/miraheze/CreateWiki/pull/546#discussion_r1720972992
[15:02:42] <botifico>	 [02CreateWiki] 07coderabbitai[bot] reviewed pull request 03#546 commit - 13https://github.com/miraheze/CreateWiki/pull/546#discussion_r1720996013
[15:02:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 23.57, 23.01
[15:03:00] <botifico>	 [02CreateWiki] 07coderabbitai[bot] edited a comment on pull request 03#546: Add Request Lock to RequestWiki - 13https://github.com/miraheze/CreateWiki/pull/546#issuecomment-2294957983
[15:04:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.71, 23.55, 23.10
[15:05:58] <miraheze-github>	 miraheze/CreateWiki - pixDeVl the build has errored.
[15:06:17] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.99, 5.85, 6.59
[15:06:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.18, 24.48, 23.45
[15:08:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.33, 23.11, 23.11
[15:12:11] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[15:12:42] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.46, 3.62, 1.99
[15:12:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.63, 23.73, 23.20
[15:14:40] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.15, 3.25, 2.04
[15:14:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.09, 23.45, 23.18
[15:16:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.39, 25.65, 24.01
[15:17:08] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[15:18:22] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 3.848 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:18:53] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.20, 22.47, 17.94
[15:19:33] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.18, 19.74, 15.31
[15:19:35] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.46, 19.04, 15.19
[15:19:43] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:20:50] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.42, 22.24, 18.39
[15:21:28] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.57, 18.50, 15.41
[15:21:33] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.80, 17.87, 15.21
[15:22:47] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.88, 19.49, 17.84
[15:24:17] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.16, 6.59, 6.39
[15:27:28] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.50, 5.39, 3.47
[15:28:17] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.36, 6.37, 6.38
[15:28:37] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:28:38] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[15:28:58] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[15:28:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.72, 22.48, 23.71
[15:29:08] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[15:29:43] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:30:34] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.075 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:30:34] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:30:55] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[15:30:59] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.21, 23.66, 23.97
[15:34:07] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures
[15:34:43] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:43:37] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[15:56:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.86, 18.77, 15.15
[15:57:19] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.08, 3.06, 3.92
[15:58:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.55, 16.51, 14.75
[16:00:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.02, 20.63, 18.21
[16:02:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.63, 20.64, 16.45
[16:03:00] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.65, 22.18, 17.61
[16:03:49] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.99, 20.34, 16.93
[16:04:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.73, 23.53, 19.93
[16:04:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.14, 20.00, 16.75
[16:04:58] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.01, 21.21, 17.82
[16:05:15] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.62, 4.26, 4.04
[16:06:57] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.23, 22.62, 18.72
[16:07:14] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.96, 3.47, 3.77
[16:08:29] <icinga-miraheze>	 PROBLEM - wiki.maribelhearn.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.maribelhearn.com All nameservers failed to answer the query.
[16:08:55] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.31, 21.31, 18.69
[16:09:31] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 13.77, 18.17, 17.26
[16:09:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.36, 20.41, 17.85
[16:10:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.87, 23.53, 21.06
[16:10:54] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.69, 19.62, 18.36
[16:11:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.31, 4.85, 4.18
[16:11:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.56, 19.54, 17.88
[16:13:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.64, 3.90, 3.91
[16:14:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.69, 23.66, 21.62
[16:16:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.98, 23.00, 21.64
[16:18:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.00, 24.48, 22.33
[16:20:45] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.67, 20.50, 19.30
[16:22:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.09, 22.91, 22.22
[16:22:43] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.14, 19.90, 19.24
[16:23:02] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.98, 2.63, 3.32
[16:23:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.34, 21.72, 19.49
[16:23:39] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[16:24:03] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.22, 24.66, 22.91
[16:24:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.04, 21.65, 19.11
[16:25:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.33, 18.91, 17.66
[16:25:40] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001228451729 secs
[16:27:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.22, 21.61, 18.78
[16:28:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.82, 22.43, 20.43
[16:29:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.82, 22.90, 19.64
[16:29:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:30:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.20, 21.14, 20.18
[16:30:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.19, 23.11, 20.80
[16:31:55] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.12, 3.79, 3.50
[16:32:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.97, 19.45, 19.72
[16:33:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.61, 20.05, 19.25
[16:33:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 18.29, 21.87, 21.25
[16:35:53] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.99, 3.53, 3.44
[16:37:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.54, 22.15, 20.36
[16:37:26] <icinga-miraheze>	 RECOVERY - wiki.maribelhearn.com - reverse DNS on sslhost is OK: SSL OK - wiki.maribelhearn.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[16:37:44] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:37:52] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.12, 4.81, 3.92
[16:42:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 17.94, 19.89, 20.22
[16:43:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.21, 19.34, 19.73
[16:44:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:46:12] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[16:46:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 21.74, 20.88
[16:47:11] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:47:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.35, 24.18, 21.63
[16:47:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 17.83, 19.15, 20.11
[16:48:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.25, 20.68, 20.60
[16:51:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 18.82, 22.48, 21.59
[16:51:58] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[16:52:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.30, 22.80, 21.38
[16:54:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.56, 21.52, 21.13
[16:55:59] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.081 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[16:56:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.17, 22.66, 21.57
[16:58:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.94, 21.78, 21.37
[17:00:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.67, 22.83, 21.78
[17:00:47] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.67, 22.08, 19.84
[17:01:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.14, 23.82, 22.27
[17:01:27] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.56, 22.26, 19.70
[17:02:16] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[17:02:20] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.87, 21.87, 20.52
[17:02:46] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181
[17:03:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.37, 20.43, 16.46
[17:03:26] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.34, 22.56, 20.14
[17:04:16] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.00, 24.42, 21.61
[17:04:28] <icinga-miraheze>	 PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query.
[17:04:45] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[17:04:49] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[17:05:25] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.86, 24.85, 21.25
[17:07:05] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query.
[17:07:08] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.46, 20.09, 17.35
[17:08:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.85, 23.66, 21.96
[17:08:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.10, 23.19, 22.70
[17:09:22] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 18.04, 21.91, 21.02
[17:10:03] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 17.81, 22.99, 22.18
[17:13:19] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.90, 19.58, 20.32
[17:13:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.84, 3.47, 3.97
[17:14:06] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[17:14:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 21.66, 21.44
[17:15:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.90, 23.99, 23.82
[17:16:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.04, 24.21, 22.92
[17:17:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.04, 23.83, 23.77
[17:17:14] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.03, 21.06, 20.73
[17:18:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.29, 21.45, 21.53
[17:18:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.07, 23.69, 22.95
[17:19:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.04, 23.09, 23.54
[17:21:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.99, 24.66, 24.08
[17:21:54] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.41, 4.16, 3.90
[17:22:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.20, 22.88, 22.08
[17:23:53] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.40, 3.48, 3.67
[17:28:27] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:28:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.43, 23.51, 22.71
[17:29:49] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.50, 3.83, 3.73
[17:30:28] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0009778439999 secs
[17:31:04] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.38, 21.81, 21.30
[17:31:46] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[17:32:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.13, 24.98, 23.40
[17:33:03] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.43, 22.51, 21.65
[17:33:42] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.075 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:33:57] <icinga-miraheze>	 RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK
[17:34:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.01, 21.59, 21.92
[17:36:52] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'}
[17:37:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.68, 22.73, 21.93
[17:39:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 17.94, 20.87, 21.35
[17:40:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.37, 23.37, 22.62
[17:42:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.08, 22.81, 23.64
[17:43:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.23, 18.18, 20.12
[17:43:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.12, 3.49, 3.91
[17:45:00] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:45:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.46, 22.89, 23.93
[17:46:51] <icinga-miraheze>	 PROBLEM - wiki.recaptime.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:52:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.41, 22.66, 21.86
[17:53:07] <icinga-miraheze>	 PROBLEM - wiki.recaptime.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.recaptime.eu.org reverse DNS resolves to 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.b.0.b.0.1.0.0.4.0.0.9.1.0.0.6.2.bc.googleusercontent.com
[17:53:43] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[17:53:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.75, 3.83, 3.80
[17:55:44] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0007990598679 secs
[17:56:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.80, 22.37, 22.12
[17:56:45] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.59, 18.62, 19.86
[17:57:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 23.06, 23.21
[17:58:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.06, 23.25, 22.94
[17:58:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.18, 23.93, 22.68
[17:59:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.39, 23.62, 21.35
[17:59:43] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:59:49] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.23, 3.45, 3.79
[18:00:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.04, 23.68, 21.55
[18:01:04] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw152 mw162
[18:01:05] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw162
[18:01:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.02, 20.82, 16.78
[18:01:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.67, 4.48, 4.13
[18:03:03] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[18:03:05] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[18:03:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.85, 21.60, 17.60
[18:03:18] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[18:04:55] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:05:42] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:06:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 15.86, 21.37, 21.55
[18:07:08] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.89, 18.68, 17.43
[18:07:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.36, 23.87, 22.75
[18:08:17] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[18:08:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 16.51, 22.52, 23.55
[18:08:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.03, 22.99, 23.47
[18:09:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 17.12, 22.24, 23.66
[18:09:25] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.076 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[18:10:37] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 23 minutes ago with 0 failures
[18:14:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.22, 19.25, 20.39
[18:16:10] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:17:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 22.19, 22.16
[18:18:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.43, 24.08, 23.40
[18:18:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.83, 21.06, 20.88
[18:18:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.61, 23.13, 22.78
[18:19:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.60, 23.89, 23.29
[18:19:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.80, 22.35, 22.24
[18:21:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.21, 23.35, 22.59
[18:23:26] <botifico>	 [02mediawiki-repos] 07coderabbitai[bot] edited a comment on pull request 03#31: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mediawiki-repos/pull/31#issuecomment-2286735922
[18:23:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.57, 22.34, 22.29
[18:24:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.69, 23.18, 23.55
[18:26:13] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[18:26:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.16, 24.85, 24.11
[18:26:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.96, 22.98, 21.61
[18:27:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.65, 23.75, 22.80
[18:28:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.78, 23.15, 21.89
[18:30:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.86, 22.75, 23.72
[18:31:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 18.35, 22.19, 22.51
[18:32:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.34, 23.18, 23.72
[18:33:14] <botifico>	 [02mw-config] 07songnguxyz synchronize pull request 03#5636: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mw-config/pull/5636
[18:33:29] <botifico>	 [02mediawiki-repos] 07songnguxyz synchronize pull request 03#31: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mediawiki-repos/pull/31
[18:34:17] <miraheze-github>	 miraheze/mw-config - songnguxyz the build passed.
[18:34:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.97, 23.56, 23.96
[18:34:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:35:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.24, 22.93, 22.68
[18:36:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.86, 25.63, 24.67
[18:36:37] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 5.456 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[18:36:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.19, 22.72, 21.81
[18:38:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.55, 23.17, 22.11
[18:38:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.33, 23.61, 23.76
[18:41:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.86, 23.80, 23.49
[18:42:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.79, 23.39, 23.97
[18:44:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.43, 24.64, 24.32
[18:46:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.19, 22.75, 21.97
[18:46:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.45, 23.49, 23.30
[18:48:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.59, 23.26, 22.30
[18:48:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.83, 23.04, 23.21
[18:49:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.94, 22.34, 23.73
[18:50:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.54, 22.86, 23.06
[18:52:42] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:54:43] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.00113633275 secs
[18:54:43] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:59:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.30, 22.40, 22.29
[18:59:43] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:00:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.16, 23.61, 23.67
[19:01:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.01, 21.59, 22.02
[19:02:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.78, 24.28, 23.86
[19:03:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.80, 23.34, 22.97
[19:04:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.66, 22.15, 23.40
[19:04:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 22.49, 21.87
[19:04:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.85, 22.06, 23.12
[19:05:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.47, 23.63, 23.13
[19:06:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.74, 23.31, 23.66
[19:06:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.97, 21.52, 21.59
[19:08:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.16, 23.47, 23.72
[19:09:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.42, 23.86, 23.22
[19:11:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.75, 22.79, 22.88
[19:12:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.90, 24.16, 23.91
[19:13:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.87, 3.58, 4.00
[19:14:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.25, 23.76, 23.83
[19:18:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.98, 25.22, 24.34
[19:18:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.54, 23.07, 22.19
[19:18:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.33, 24.38, 23.25
[19:19:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.68, 24.52, 23.38
[19:19:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.64, 23.72, 21.95
[19:21:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.95, 23.54, 22.14
[19:23:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.38, 24.48, 22.68
[19:23:47] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.38, 2.41, 3.15
[19:24:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.42, 22.77, 22.50
[19:26:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.81, 22.84, 22.51
[19:28:46] <botifico>	 [02mw-config] 07redbluegreenhat created branch 03T12484-RightFunctions - 13https://github.com/miraheze/mw-config
[19:28:48] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03T12484-RightFunctions [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/cac823cdf88f
[19:28:49] <botifico>	 [02mw-config] 07redbluegreenhat 03cac823c - T12484: Re-enable RightFunctions
[19:28:57] <botifico>	 [02mw-config] 07redbluegreenhat opened pull request 03#5640: T12484: Re-enable RightFunctions - 13https://github.com/miraheze/mw-config/pull/5640
[19:29:31] <botifico>	 [02mw-config] 07github-actions[bot] pushed 031 commit to 03T12484-RightFunctions [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/cac823cdf88f...6995013eb866
[19:29:33] <botifico>	 [02mw-config] 07github-actions 036995013 - CI: lint code to MediaWiki standards
[19:29:36] <botifico>	 [02mw-config] 07github-actions[bot] synchronize pull request 03#5640: T12484: Re-enable RightFunctions - 13https://github.com/miraheze/mw-config/pull/5640
[19:29:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:29:45] <botifico>	 [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/b194aea19ac3...a800f1591a52
[19:29:48] <botifico>	 [02mw-config] 07redbluegreenhat 03a800f15 - T12484: Re-enable RightFunctions (#5640)
[19:29:51] <botifico>	 [02mw-config] 07redbluegreenhat closed pull request 03#5640: T12484: Re-enable RightFunctions - 13https://github.com/miraheze/mw-config/pull/5640
[19:29:53] <botifico>	 [02mw-config] 07redbluegreenhat deleted branch 03T12484-RightFunctions
[19:29:54] <botifico>	 [02mw-config] 07redbluegreenhat deleted branch 03T12484-RightFunctions - 13https://github.com/miraheze/mw-config
[19:30:03] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[19:30:06] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'versions': '1.42', 'upgrade_extensions': 'RightFunctions'} to all
[19:30:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:30:35] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[19:31:08] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.08, 19.96, 17.08
[19:31:27] <MirahezeLSBot>	 !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'versions': '1.42', 'upgrade_extensions': 'RightFunctions'} to all - SUCCESS in 81s
[19:31:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:33:08] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.59, 18.14, 16.75
[19:34:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 17.78, 21.93, 22.63
[19:34:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.19, 22.33, 23.89
[19:35:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.09, 23.20, 23.98
[19:35:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.47, 4.22, 3.65
[19:36:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.46, 22.66, 22.79
[19:36:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.43, 22.79, 23.83
[19:37:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.95, 25.50, 24.73
[19:37:39] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[19:38:19] <MirahezeLSBot>	 !log [@mwtask171] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw182.wikitide.net
[19:38:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:38:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.89, 23.42, 23.97
[19:38:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:39:21] <MirahezeRelay>	 <songngu.xyz> !?
[19:39:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.02, 3.76, 3.58
[19:40:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.43, 25.16, 24.53
[19:40:50] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync]
[19:41:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.24, 4.54, 3.88
[19:41:52] <botifico>	 [02mw-config] 07Reception123 closed pull request 03#5638: Reenable Evelution for MediaWiki 1.42 - 13https://github.com/miraheze/mw-config/pull/5638
[19:41:53] <botifico>	 [02mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a800f1591a52...6381dfa4b407
[19:41:55] <botifico>	 [02mw-config] 07BlankEclair 036381dfa - Reenable Evelution for MediaWiki 1.42 (#5638)
[19:42:06] <MirahezeLSBot>	 !log [reception@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_skins': 'Evelution'} to all
[19:42:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:42:22] <botifico>	 [02mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6381dfa4b407...9e04c07f3e99
[19:42:24] <botifico>	 [02mw-config] 07BlankEclair 039e04c07 - Add incompatibility warning for HeaderTabs (#5639)
[19:42:27] <botifico>	 [02mw-config] 07Reception123 closed pull request 03#5639: Add incompatibility warning for HeaderTabs - 13https://github.com/miraheze/mw-config/pull/5639
[19:42:37] <MirahezeLSBot>	 !log [reception@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_skins': 'Evelution'} to all - SUCCESS in 31s
[19:42:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.76, 23.77, 23.50
[19:42:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:42:47] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[19:42:58] <MirahezeLSBot>	 !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[19:43:14] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[19:43:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:43:22] <MirahezeLSBot>	 !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s
[19:43:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:44:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.23, 24.14, 23.63
[19:47:00] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:47:19] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw161
[19:47:42] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:47:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.84, 3.36, 3.67
[19:49:19] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[19:49:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.22, 3.76, 3.75
[19:49:52] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[19:51:36] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:51:53] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0004848241806 secs
[19:52:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 17.26, 21.65, 23.14
[19:53:09] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[19:53:10] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[19:53:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:53:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:53:52] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[19:54:02] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[19:55:40] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[19:59:22] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[19:59:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:02:44] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures
[20:03:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.72, 22.10, 23.90
[20:04:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:04:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.35, 21.05, 23.65
[20:05:02] <icinga-miraheze>	 PROBLEM - wiki.ouro.one - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.ouro.one All nameservers failed to answer the query.
[20:05:56] <icinga-miraheze>	 PROBLEM - threedomwiki.pcast.site - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - threedomwiki.pcast.site All nameservers failed to answer the query.
[20:06:13] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:06:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.90, 18.31, 20.30
[20:07:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.73, 23.31, 23.87
[20:07:25] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[20:07:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.44, 21.23, 23.38
[20:07:35] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.062 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[20:07:37] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[20:07:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:07:59] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 22s
[20:08:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:08:20] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:08:29] <icinga-miraheze>	 PROBLEM - wiki.maribelhearn.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.maribelhearn.com All nameservers failed to answer the query.
[20:08:50] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[20:09:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.50, 23.22, 23.80
[20:09:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.73, 22.61, 23.63
[20:09:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:10:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.25, 23.23, 23.76
[20:11:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.43, 22.03, 23.35
[20:11:58] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[20:12:43] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:12:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.42, 22.99, 23.59
[20:13:44] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[20:13:54] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.074 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[20:14:40] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:15:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.93, 23.96, 23.88
[20:15:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 22.84, 23.31
[20:17:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.22, 23.73, 23.82
[20:17:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.41, 23.30, 23.47
[20:17:35] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.48, 23.05, 21.32
[20:18:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.84, 24.23, 23.84
[20:19:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.41, 25.54, 24.47
[20:19:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.23, 24.71, 23.99
[20:19:33] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.84, 22.27, 21.28
[20:23:26] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query.
[20:25:29] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.37, 24.11, 22.16
[20:26:47] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[20:27:27] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.71, 23.92, 22.37
[20:28:47] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.000955581665 secs
[20:31:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.71, 22.91, 23.73
[20:31:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.39, 3.11, 3.97
[20:33:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.91, 24.63, 24.27
[20:33:34] <icinga-miraheze>	 RECOVERY - wiki.ouro.one - reverse DNS on sslhost is OK: SSL OK - wiki.ouro.one reverse DNS resolves to cp36.wikitide.net - CNAME OK
[20:35:22] <icinga-miraheze>	 RECOVERY - threedomwiki.pcast.site - reverse DNS on sslhost is OK: SSL OK - threedomwiki.pcast.site reverse DNS resolves to cp36.wikitide.net - CNAME OK
[20:35:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 16.74, 21.23, 23.04
[20:37:20] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.14, 22.76, 22.07
[20:37:26] <icinga-miraheze>	 RECOVERY - wiki.maribelhearn.com - reverse DNS on sslhost is OK: SSL OK - wiki.maribelhearn.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[20:39:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.51, 22.16, 22.85
[20:39:46] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.63, 2.41, 3.31
[20:43:16] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.64, 23.00, 22.42
[20:43:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.78, 23.13, 23.19
[20:45:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.62, 23.54, 23.30
[20:45:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.39, 3.60, 3.52
[20:46:40] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[20:47:13] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 23.56, 22.70
[20:47:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.91, 23.41, 23.27
[20:47:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.12, 4.20, 3.75
[20:48:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.41, 22.38, 23.95
[20:49:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.34, 22.74, 24.00
[20:49:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.49, 23.68, 23.39
[20:49:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.62, 3.55, 3.57
[20:50:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.86, 21.49, 23.64
[20:51:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.11, 23.26, 23.26
[20:52:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.45, 22.96, 23.87
[20:53:09] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.54, 22.70, 22.81
[20:53:26] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found
[20:53:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.55, 4.07, 3.75
[20:54:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.49, 20.57, 22.87
[21:02:10] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[21:03:46] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[21:04:08] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[21:05:25] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query.
[21:07:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.69, 22.94, 22.38
[21:09:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.01, 23.21, 22.73
[21:09:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.35, 23.09, 22.51
[21:09:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:10:03] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 9.760 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[21:10:55] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.81, 19.08, 20.35
[21:11:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 18.47, 21.51, 22.17
[21:12:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.22, 18.50, 20.37
[21:14:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:15:26] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:18:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.77, 18.87, 20.02
[21:21:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.75, 23.40, 22.53
[21:21:19] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.50, 21.85, 20.88
[21:21:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.75, 22.44, 21.66
[21:22:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.83, 22.99, 21.29
[21:22:45] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.87, 22.04, 20.84
[21:25:32] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.21, 23.19, 22.24
[21:26:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.72, 23.78, 22.08
[21:27:01] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.44, 23.88, 22.18
[21:27:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 18.41, 23.73, 23.34
[21:28:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 21.61, 20.97
[21:28:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.61, 23.31, 22.05
[21:29:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.71, 24.58, 23.68
[21:30:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.05, 21.05, 20.86
[21:30:49] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.88, 23.42, 22.26
[21:31:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.28, 2.93, 3.79
[21:32:44] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.41, 21.68, 21.76
[21:32:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.42, 23.77, 22.60
[21:33:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.46, 23.15, 23.38
[21:35:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.16, 23.07, 23.25
[21:35:13] <icinga-miraheze>	 PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'}
[21:36:32] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.43, 22.65, 22.05
[21:36:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.46, 22.38, 21.26
[21:36:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.80, 24.73, 23.11
[21:37:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.89, 3.28, 3.61
[21:38:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 16.97, 20.83, 21.49
[21:38:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.70, 21.43, 21.01
[21:38:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.82, 23.13, 22.70
[21:39:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.07, 23.03, 23.34
[21:39:38] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[21:40:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.93, 19.45, 20.38
[21:41:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.49, 3.75, 3.78
[21:42:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.77, 21.92, 22.15
[21:43:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 21.58, 22.52
[21:44:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.13, 23.81, 22.32
[21:44:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.52, 21.17, 20.88
[21:44:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:44:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.53, 22.85, 22.50
[21:45:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.41, 21.70, 22.50
[21:45:32] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.65, 18.31, 20.07
[21:47:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.40, 4.56, 4.01
[21:48:24] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[21:48:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.45, 21.52, 21.89
[21:48:41] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.45, 18.89, 20.08
[21:52:26] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:54:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.11, 18.61, 20.39
[21:54:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:56:13] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.089 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[21:57:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.15, 21.77, 21.69
[21:59:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 17.85, 20.44, 21.24
[22:01:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 21.29, 21.40
[22:02:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.85, 19.31, 20.30
[22:02:55] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[22:03:12] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 17.77, 19.73, 20.81
[22:05:12] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.20, 17.82, 19.98
[22:11:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.94, 3.21, 3.79
[22:14:28] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:14:43] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:15:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.20, 4.24, 4.03
[22:17:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.66, 3.78, 3.90
[22:22:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.15, 20.13, 19.27
[22:22:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.89, 20.15, 19.19
[22:23:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.74, 3.71, 3.80
[22:24:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.07, 19.24, 19.06
[22:26:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.25, 22.51, 20.34
[22:27:35] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.37, 20.49, 19.30
[22:28:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.53, 20.97, 20.03
[22:28:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.39, 20.59, 19.91
[22:29:33] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.35, 19.00, 18.88
[22:29:51] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.65, 3.38, 3.70
[22:30:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.32, 22.15, 20.53
[22:30:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.69, 21.94, 20.46
[22:31:31] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.93, 20.88, 20.28
[22:32:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.16, 21.28, 20.44
[22:32:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 17.58, 20.23, 20.02
[22:33:25] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 19.03, 20.10, 20.05
[22:33:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.40, 3.85, 3.83
[22:36:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.78, 20.06, 20.14
[22:37:49] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.44, 3.48, 3.66
[22:39:43] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:41:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.45, 4.59, 3.98
[22:42:21] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[22:44:19] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 1.947 second response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[22:45:02] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:47:07] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 21.38, 20.38
[22:49:17] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.17, 20.61, 18.78
[22:49:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.72, 3.66, 3.86
[22:50:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.12, 20.41, 19.67
[22:50:55] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.43, 20.45, 20.27
[22:51:16] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.09, 18.65, 18.27
[22:53:48] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.24, 4.22, 4.06
[22:54:36] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[22:54:44] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.38, 19.70, 20.05
[22:54:59] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:56:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.72, 20.18, 19.89
[22:59:08] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[23:00:02] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:00:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.60, 20.25, 20.17
[23:00:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.40, 20.50, 20.12
[23:02:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 20.06, 19.96, 20.07
[23:04:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.95, 20.26, 20.14
[23:09:07] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.074 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[23:10:02] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:15:02] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:16:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.51, 19.56, 18.97
[23:18:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.95, 22.13, 19.95
[23:18:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.90, 20.53, 19.49
[23:20:38] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.25, 22.46, 20.66
[23:20:41] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.93, 24.04, 20.95
[23:20:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.02, 22.81, 20.44
[23:22:30] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.62, 21.69, 20.37
[23:22:33] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.01, 24.25, 21.50
[23:22:36] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 18.34, 21.94, 20.57
[23:22:44] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.19, 21.89, 20.36
[23:22:51] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.43, 20.94, 18.91
[23:24:30] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.38, 20.14, 19.97
[23:24:49] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.71, 20.03, 18.82
[23:26:21] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.01, 22.26, 21.37
[23:26:28] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.54, 19.22, 19.77
[23:30:44] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.44, 19.33, 19.92
[23:33:57] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.29, 18.73, 20.23
[23:35:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.86, 3.30, 3.83
[23:37:49] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.16, 3.46, 3.79
[23:39:47] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.37, 3.38, 3.75
[23:41:46] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.23, 4.22, 4.00
[23:49:24] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:50:02] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:50:42] <icinga-miraheze>	 PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[23:51:52] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 36 minutes ago with 0 failures
[23:52:44] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[23:53:26] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query.
[23:55:02] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1