[00:03:36] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.21, 20.75, 20.66
[00:05:36] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.16, 19.94, 20.37
[02:16:06] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.94, 19.47, 17.85
[02:18:01] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.88, 21.31, 18.67
[02:19:57] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.36, 21.55, 19.09
[02:25:57] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.44, 18.58, 16.01
[02:27:44] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.86, 24.09, 21.18
[02:27:57] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.01, 18.71, 16.35
[02:31:44] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.26, 23.29, 21.56
[02:32:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:47:36] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.72, 18.89, 16.18
[02:49:36] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.11, 17.55, 16.03
[02:53:44] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.33, 18.53, 20.05
[04:32:20] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:34:29] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:36:28] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.00119048357 secs
[04:46:49] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[05:15:08] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[05:46:13] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:15:02] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:32:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:35:19] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[06:39:47] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:41:47] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 5.856156349e-05 secs
[07:01:46] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:03:44] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.00012627244 secs
[08:03:44] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.46, 19.79, 17.48
[08:05:44] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.77, 19.42, 17.60
[08:05:57] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:08:07] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[08:12:15] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0004019141197 secs
[08:16:17] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query.
[08:16:37] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 21.36, 19.28
[08:18:32] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.83, 20.63, 19.25
[08:20:28] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.41, 20.27, 19.32
[08:25:00] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:26:59] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0003427863121 secs
[08:45:34] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:55:44] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:57:43] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -6.499886513e-05 secs
[09:35:44] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.74, 20.92, 18.37
[09:43:44] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.91, 20.01, 19.08
[10:32:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:46:51] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:48:54] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[11:53:03] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0007266998291 secs
[12:12:21] <botifico>	 [02CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±13] 13https://github.com/miraheze/CreateWiki/compare/c304efb08f82...7207b0b97564
[12:12:22] <botifico>	 [02CreateWiki] 07translatewiki 037207b0b - Localisation updates from https://translatewiki.net.
[12:12:23] <botifico>	 [02ErrorPages] 07translatewiki pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/ErrorPages/compare/d32ed98c2ae2...f4ed225d67fc
[12:12:24] <botifico>	 [02ErrorPages] 07translatewiki 03f4ed225 - Localisation updates from https://translatewiki.net.
[12:12:25] <botifico>	 [02GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/5d7133450ae3...ab8f05b4b752
[12:12:26] <botifico>	 [02GlobalNewFiles] 07translatewiki 03ab8f05b - Localisation updates from https://translatewiki.net.
[12:12:28] <botifico>	 [02ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/a4451c043e53...505e85f05678
[12:12:29] <botifico>	 [02ImportDump] 07translatewiki 03505e85f - Localisation updates from https://translatewiki.net.
[12:12:30] <botifico>	 [02landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/landing/compare/aa8ca2879c76...5d54c9c18c55
[12:12:32] <botifico>	 [02landing] 07translatewiki 035d54c9c - Localisation updates from https://translatewiki.net.
[12:12:33] <botifico>	 [02DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/3347ac687aab...38687d9f633a
[12:12:36] <botifico>	 [02DataDump] 07translatewiki 0338687d9 - Localisation updates from https://translatewiki.net.
[12:12:39] <botifico>	 [02RottenLinks] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/6d9ba07a7789...b2468bb65bce
[12:12:42] <botifico>	 [02RottenLinks] 07translatewiki 03b2468bb - Localisation updates from https://translatewiki.net.
[12:12:45] <botifico>	 [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±8] 13https://github.com/miraheze/MirahezeMagic/compare/6f350919aa08...64597fddb1df
[12:12:48] <botifico>	 [02MirahezeMagic] 07translatewiki 0364597fd - Localisation updates from https://translatewiki.net.
[12:12:49] <botifico>	 [02WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/7cc6b158d76c...f4496b1ed733
[12:12:50] <botifico>	 [02WikiDiscover] 07translatewiki 03f4496b1 - Localisation updates from https://translatewiki.net.
[12:13:07] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:13:10] <miraheze-github_>	 miraheze/ErrorPages - translatewiki the build passed.
[12:15:47] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build has errored.
[12:16:57] <miraheze-github>	 miraheze/RottenLinks - translatewiki the build passed.
[12:17:04] <miraheze-github_>	 miraheze/GlobalNewFiles - translatewiki the build passed.
[12:17:41] <miraheze-github>	 miraheze/WikiDiscover - translatewiki the build passed.
[12:17:56] <miraheze-github>	 miraheze/DataDump - translatewiki the build passed.
[12:21:36] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build passed.
[12:22:25] <miraheze-github>	 miraheze/ImportDump - translatewiki the build passed.
[12:23:43] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[12:23:44] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:23:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:55] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[12:23:55] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:08] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151
[12:24:09] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:32:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:32:46] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:32:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:32:56] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s
[12:33:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:07] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[12:33:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:18] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 11s
[12:33:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:50] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:37:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:59] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[12:38:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:38:10] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[12:38:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:38:19] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[12:38:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:15:09] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[13:53:44] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.57, 20.43, 17.36
[13:55:44] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.33, 23.27, 18.77
[14:16:54] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[14:23:40] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.23, 16.87, 13.45
[14:25:40] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.41, 15.93, 13.55
[14:34:29] <icinga-miraheze>	 PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:34:37] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw162
[14:34:39] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw162
[14:34:58] <icinga-miraheze>	 PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[14:35:08] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw162
[14:35:28] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw162
[14:35:37] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw162
[14:36:07] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw162
[14:41:09] <icinga-miraheze>	 RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.063 second response time
[14:41:28] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[14:41:37] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[14:42:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:42:07] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy
[14:42:37] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy
[14:42:39] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[14:42:50] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.296 second response time
[14:43:08] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[14:43:30] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[14:57:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:41:44] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.85, 22.70, 23.76
[15:55:44] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.05, 18.59, 20.25
[17:00:24] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/af8ed7833a28...0c4298fd850c
[17:00:26] <botifico>	 [02ssl] 07WikiTideSSLBot 030c4298f - Bot: Add SSL cert for wiki.enkro.co
[17:03:15] <botifico>	 [02dns] 07MacFan4000 opened pull request 03#531: Add beyondtheveilrp.com zone - 13https://github.com/miraheze/dns/pull/531
[17:27:01] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.66, 20.77, 17.74
[17:31:01] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.64, 20.60, 18.52
[17:33:01] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 12.29, 17.86, 17.78
[17:49:40] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 22.74, 19.71
[17:51:34] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.70, 21.68, 19.67
[17:54:20] <icinga-miraheze>	 PROBLEM - www.gengbaike.top - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.gengbaike.top' expires in 15 day(s) (Sat 13 Jul 2024 05:37:44 PM GMT +0000).
[17:54:54] <MacFan4000>	 Could somebody please merge https://github.com/miraheze/dns/pull/531?
[17:57:18] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.35, 22.16, 20.40
[18:11:02] <icinga-miraheze>	 PROBLEM - fid.koymi.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fid.koymi.net' expires in 15 day(s) (Sat 13 Jul 2024 06:06:25 PM GMT +0000).
[18:11:17] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0c4298fd850c...a247c72d5155
[18:11:19] <botifico>	 [02ssl] 07WikiTideSSLBot 03a247c72 - Bot: Update SSL cert for fid.koymi.net
[18:17:28] <icinga-miraheze>	 PROBLEM - progressbar95.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'progressbar95.wiki' expires in 15 day(s) (Sat 13 Jul 2024 05:55:43 PM GMT +0000).
[18:17:44] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a247c72d5155...ca04fe8f3d25
[18:17:45] <botifico>	 [02ssl] 07WikiTideSSLBot 03ca04fe8 - Bot: Update SSL cert for progressbar95.wiki
[18:22:40] <icinga-miraheze>	 PROBLEM - wiki.widwwa.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.widwwa.co.uk' expires in 15 day(s) (Sat 13 Jul 2024 06:04:01 PM GMT +0000).
[18:22:51] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ca04fe8f3d25...bd10ae91484a
[18:22:53] <botifico>	 [02ssl] 07WikiTideSSLBot 03bd10ae9 - Bot: Update SSL cert for wiki.widwwa.co.uk
[18:28:40] <icinga-miraheze>	 PROBLEM - wiki.rlab.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rlab.org.uk' expires in 15 day(s) (Sat 13 Jul 2024 06:01:16 PM GMT +0000).
[18:28:58] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bd10ae91484a...e1d6e627ddf2
[18:28:59] <botifico>	 [02ssl] 07WikiTideSSLBot 03e1d6e62 - Bot: Update SSL cert for wiki.rlab.org.uk
[18:37:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:44:39] <icinga-miraheze>	 PROBLEM - de.berlinwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'de.berlinwiki.org' expires in 15 day(s) (Sat 13 Jul 2024 06:13:10 PM GMT +0000).
[18:44:52] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e1d6e627ddf2...41fbd3eb90ed
[18:44:52] <botifico>	 [02ssl] 07WikiTideSSLBot 0341fbd3e - Bot: Update SSL cert for de.berlinwiki.org
[18:45:01] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 14.77, 18.28, 22.91
[18:51:01] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.99, 15.75, 20.31
[18:51:28] <icinga-miraheze>	 RECOVERY - wiki.widwwa.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.widwwa.co.uk' will expire on Wed 25 Sep 2024 05:22:46 PM GMT +0000.
[18:58:19] <icinga-miraheze>	 RECOVERY - wiki.rlab.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rlab.org.uk' will expire on Wed 25 Sep 2024 05:28:46 PM GMT +0000.
[19:00:03] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:01:59] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0003095269203 secs
[19:08:59] <icinga-miraheze>	 RECOVERY - fid.koymi.net - LetsEncrypt on sslhost is OK: OK - Certificate 'fid.koymi.net' will expire on Wed 25 Sep 2024 05:11:11 PM GMT +0000.
[19:14:38] <icinga-miraheze>	 RECOVERY - progressbar95.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'progressbar95.wiki' will expire on Wed 25 Sep 2024 05:17:36 PM GMT +0000.
[19:16:59] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:44:38] <icinga-miraheze>	 RECOVERY - de.berlinwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'de.berlinwiki.org' will expire on Wed 25 Sep 2024 05:44:45 PM GMT +0000.
[19:44:55] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:52:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:57:51] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.04, 19.54, 16.44
[19:59:45] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.69, 17.89, 16.19
[20:38:40] <botifico>	 [02dns] 07MacFan4000 synchronize pull request 03#531: Add beyondtheveilrp.com zone - 13https://github.com/miraheze/dns/pull/531
[20:39:01] <botifico>	 [02dns] 07MacFan4000 edited pull request 03#531: Add beyondtheveilrp.com and terraforming.wiki zones - 13https://github.com/miraheze/dns/pull/531
[20:44:46] <icinga-miraheze>	 PROBLEM - pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pyramidgames.wiki' expires in 7 day(s) (Fri 05 Jul 2024 08:41:07 PM GMT +0000).
[20:55:48] <icinga-miraheze>	 PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pyramidgames.wiki' expires in 7 day(s) (Fri 05 Jul 2024 08:41:07 PM GMT +0000).
[20:57:02] <icinga-miraheze>	 PROBLEM - partopedia.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'partopedia.ru' expires in 15 day(s) (Sat 13 Jul 2024 08:50:58 PM GMT +0000).
[20:57:17] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/41fbd3eb90ed...3f3d7367132e
[20:57:19] <botifico>	 [02ssl] 07WikiTideSSLBot 033f3d736 - Bot: Update SSL cert for partopedia.ru
[20:59:13] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.16, 20.93, 18.54
[21:01:07] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.98, 20.03, 18.48
[21:01:31] <botifico>	 [02dns] 07The-Voidwalker closed pull request 03#531: Add beyondtheveilrp.com and terraforming.wiki zones - 13https://github.com/miraheze/dns/pull/531
[21:01:33] <botifico>	 [02dns] 07The-Voidwalker pushed 031 commit to 03master [+2/-0/±0] 13https://github.com/miraheze/dns/compare/5706d4ce14b5...24a847c9e223
[21:01:34] <botifico>	 [02dns] 07MacFan4000 0324a847c - Add beyondtheveilrp.com and terraforming.wiki zones (#531)
[21:01:57] <MirahezeRelay>	 <orduin> MacFan4000, done
[21:02:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:15:01] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.32, 21.93, 19.36
[21:25:01] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.32, 22.68, 21.59
[21:26:23] <icinga-miraheze>	 RECOVERY - partopedia.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'partopedia.ru' will expire on Wed 25 Sep 2024 07:57:11 PM GMT +0000.
[21:30:50] <icinga-miraheze>	 PROBLEM - www.widedata.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.widedata.org' expires in 15 day(s) (Sat 13 Jul 2024 09:18:16 PM GMT +0000).
[21:31:02] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3f3d7367132e...94add32add28
[21:31:04] <botifico>	 [02ssl] 07WikiTideSSLBot 0394add32 - Bot: Update SSL cert for www.widedata.org
[21:31:59] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/94add32add28...c6099eca558c
[21:32:02] <botifico>	 [02ssl] 07WikiTideSSLBot 03c6099ec - Bot: Add SSL cert for terraforming.wiki
[21:32:18] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/c6099eca558c...0d62e3cbafaf
[21:32:19] <botifico>	 [02ssl] 07WikiTideSSLBot 030d62e3c - Bot: Add SSL cert for beyondtheveilrp.com
[21:43:01] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.11, 18.65, 20.21
[21:44:11] <icinga-miraheze>	 PROBLEM - osfirsttimer.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'osfirsttimer.wiki' expires in 15 day(s) (Sat 13 Jul 2024 09:16:39 PM GMT +0000).
[21:44:22] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0d62e3cbafaf...a8f96a785026
[21:44:23] <botifico>	 [02ssl] 07WikiTideSSLBot 03a8f96a7 - Bot: Update SSL cert for osfirsttimer.wiki
[21:59:50] <icinga-miraheze>	 RECOVERY - www.widedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.widedata.org' will expire on Wed 25 Sep 2024 08:30:55 PM GMT +0000.
[22:12:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:12:51] <icinga-miraheze>	 RECOVERY - osfirsttimer.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'osfirsttimer.wiki' will expire on Wed 25 Sep 2024 08:44:16 PM GMT +0000.
[23:51:20] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.51, 19.64, 17.47
[23:55:14] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.97, 22.89, 19.24