[00:01:04] <icinga-miraheze>	 PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.autocountsoft.com' expires in 15 day(s) (Mon 13 May 2024 11:56:21 PM GMT +0000).
[00:04:11] <icinga-miraheze>	 PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Mon 13 May 2024 11:43:32 PM GMT +0000).
[00:19:11] <icinga-miraheze>	 PROBLEM - speleo.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'speleo.wiki' expires in 15 day(s) (Mon 13 May 2024 11:56:37 PM GMT +0000).
[00:19:23] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/585f74ae3a61...6f6553dd5940
[00:19:26] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 036f6553d - Bot: Update SSL cert for speleo.wiki
[00:20:30] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.05 ms
[00:22:33] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.99 ms
[00:26:06] <icinga-miraheze>	 PROBLEM - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worlduniversityandschool.org' expires in 15 day(s) (Tue 14 May 2024 12:09:12 AM GMT +0000).
[00:28:01] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[00:30:18] <icinga-miraheze>	 PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thelonsdalebattalion.co.uk' expires in 15 day(s) (Tue 14 May 2024 12:06:56 AM GMT +0000).
[00:30:36] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6f6553dd5940...36fa81836b74
[00:30:38] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0336fa818 - Bot: Update SSL cert for thelonsdalebattalion.co.uk
[00:34:41] <icinga-miraheze>	 PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gesamtschule-nordkirchen.de' expires in 15 day(s) (Tue 14 May 2024 12:06:33 AM GMT +0000).
[00:34:52] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/36fa81836b74...6b8281fa8efe
[00:34:53] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 036b8281f - Bot: Update SSL cert for wiki.gesamtschule-nordkirchen.de
[00:37:27] <icinga-miraheze>	 RECOVERY - www.allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Fri 26 Jul 2024 10:49:12 PM GMT +0000.
[00:41:33] <icinga-miraheze>	 PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Tue 14 May 2024 12:15:30 AM GMT +0000).
[00:41:45] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6b8281fa8efe...bbd71d5b4d87
[00:41:48] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03bbd71d5 - Bot: Update SSL cert for marinebiodiversitymatrix.org
[00:45:51] <icinga-miraheze>	 PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.make717.org' expires in 15 day(s) (Tue 14 May 2024 12:35:28 AM GMT +0000).
[00:47:06] <icinga-miraheze>	 RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Fri 26 Jul 2024 10:49:12 PM GMT +0000.
[00:47:57] <icinga-miraheze>	 PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'adadevelopersacademy.wiki' expires in 15 day(s) (Tue 14 May 2024 12:38:05 AM GMT +0000).
[00:53:47] <icinga-miraheze>	 PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tulpa.info' expires in 15 day(s) (Tue 14 May 2024 12:32:31 AM GMT +0000).
[00:56:57] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[00:59:37] <icinga-miraheze>	 RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Fri 26 Jul 2024 11:30:30 PM GMT +0000.
[01:01:04] <icinga-miraheze>	 PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[01:01:04] <icinga-miraheze>	 PROBLEM - changeprop151 NTP time on changeprop151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[01:03:07] <icinga-miraheze>	 PROBLEM - changeprop151 NTP time on changeprop151 is CRITICAL: NTP CRITICAL: Offset 89.94933629 secs
[01:03:09] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[01:03:37] <icinga-miraheze>	 PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grottocenter.org' expires in 15 day(s) (Tue 14 May 2024 12:36:39 AM GMT +0000).
[01:03:50] <icinga-miraheze>	 RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Fri 26 Jul 2024 11:34:47 PM GMT +0000.
[01:12:20] <icinga-miraheze>	 PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Tue 14 May 2024 01:02:34 AM GMT +0000).
[01:13:50] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Tue 14 May 2024 01:02:34 AM GMT +0000).
[01:14:05] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bbd71d5b4d87...6f6a37db1f8d
[01:14:08] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 036f6a37d - Bot: Update SSL cert for permanentfuturelab.wiki
[01:17:49] <icinga-miraheze>	 RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Fri 26 Jul 2024 11:19:17 PM GMT +0000.
[01:30:46] <icinga-miraheze>	 PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.autocountsoft.com' expires in 15 day(s) (Mon 13 May 2024 11:56:21 PM GMT +0000).
[01:33:09] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:36:50] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.54 ms
[01:38:43] <icinga-miraheze>	 RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Fri 26 Jul 2024 11:41:38 PM GMT +0000.
[01:38:53] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.65 ms
[02:10:59] <icinga-miraheze>	 RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sat 27 Jul 2024 12:13:59 AM GMT +0000.
[02:11:30] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sat 27 Jul 2024 12:13:59 AM GMT +0000.
[02:20:58] <MirahezeRelay>	 <azve> it worked... then i updated ONE file and the cache broke again...
[02:25:55] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[02:26:11] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ac99c896596f...07650d31d452
[02:26:13] <botifico>	 [02miraheze/puppet] 07Universal-Omega 0307650d3 - check_reverse_dns: force mw-lb again
[02:26:17] <MirahezeRelay>	 <azve> Is there like, any way to fix the cache?
[02:26:40] <MirahezeRelay>	 <elliethepwincess> what exactly is the cache problem you're having?
[02:27:11] <MirahezeRelay>	 <bryceeetaylorrrd> have you purge the page
[02:27:25] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:27:29] <MirahezeRelay>	 <azve> when i update a file the old version still shows as the thumbnail, and yes i have purged... many times...
[02:27:35] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[02:27:51] <MirahezeRelay>	 <bryceeetaylorrrd> i don't know then
[02:28:02] <MirahezeRelay>	 <elliethepwincess> I had this same issue a while ago and I thought it was resolved
[02:28:20] <MirahezeRelay>	 <elliethepwincess> but I guess it's a problem again and as far as I know there is no way to easily fix it...
[02:28:21] <MirahezeRelay>	 <azve> it came back for me
[02:28:29] <MirahezeRelay>	 <bryceeetaylorrrd> is it cf
[02:28:50] <MirahezeRelay>	 <azve> i guess i'll have to upload each individual update as its own file... even if it wastes space...
[02:29:11] <MirahezeRelay>	 <m3w> seems to be a cache issue still
[02:29:59] <MirahezeRelay>	 <azve> is there any fix?
[02:30:11] <MirahezeRelay>	 <bryceeetaylorrrd> maybe no
[02:30:15] <MirahezeRelay>	 <azve> i don't feel like i need to be uploading every new version as its own file
[02:30:29] <MirahezeRelay>	 <elliethepwincess> as I've said before it's not easily fixable so SRE is going to have to look into it
[02:31:08] <MirahezeRelay>	 <bryceeetaylorrrd> i see the images for me
[02:31:09] <icinga-miraheze>	 PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[02:33:01] <MirahezeRelay>	 <azve> it's not that they're not loading, it's that they are loading the older version in place of the new one
[02:36:54] <MirahezeRelay>	 <azve> and with Cosmic heading for a trip... yeah may not be fixed for a while
[02:37:05] <MirahezeRelay>	 <cosmicalpha> What is it now?
[02:37:36] <MirahezeRelay>	 <azve> still the same cache issues
[02:37:55] <MirahezeRelay>	 <cosmicalpha> Yesh but I've always seen most recent version afaik I have an idea one sec
[02:38:08] <MirahezeRelay>	 <azve> on my end is shows the old version
[02:38:22] <MirahezeRelay>	 <azve> it worked for a bit, but as soon as i updated ONE file... it went back to the old
[02:39:08] <MirahezeRelay>	 <cosmicalpha> Give it a few minutes and see if it works again
[02:39:31] <MirahezeRelay>	 <azve> ok
[02:42:25] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:42:37] <icinga-miraheze>	 RECOVERY - taswin.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000.
[02:46:16] <botifico>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1677150ffbd2...6488764550c7
[02:46:19] <botifico>	 [02miraheze/mw-config] 07MacFan4000 036488764 - Add maintenance notice
[02:47:12] <miraheze-github>	 miraheze/mw-config - MacFan4000 the build passed.
[02:47:34] <MirahezeLSBot>	 !log [macfan@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[02:47:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:47:44] <MirahezeLSBot>	 !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s
[02:47:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:53:27] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[02:53:28] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[02:53:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:53:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:55:57] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[02:59:15] <icinga-miraheze>	 PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.polandballwiki.com could not be found
[02:59:36] <icinga-miraheze>	 PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.autocountsoft.com' expires in 15 day(s) (Mon 13 May 2024 11:56:21 PM GMT +0000).
[03:01:01] <icinga-miraheze>	 PROBLEM - polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for polandballwiki.com could not be found
[03:01:33] <icinga-miraheze>	 PROBLEM - pwsc.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[03:05:06] <icinga-miraheze>	 PROBLEM - fanon.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fanon.polandballwiki.com could not be found
[03:07:41] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[03:07:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:07:50] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s
[03:07:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:17:51] <icinga-miraheze>	 PROBLEM - pwsc.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwsc.polandballwiki.com could not be found
[03:33:47] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.25 ms
[03:35:48] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.82 ms
[03:37:25] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:39:56] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.93 ms
[03:44:00] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 177.09 ms
[04:07:48] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6f6a37db1f8d...627c9f562bb2
[04:07:49] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03627c9f5 - Bot: Update SSL cert for www.durawiki.com
[04:08:23] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.25 ms
[04:08:36] <MirahezeRelay>	 <ltdk> this is a genuine question but, short of the bot logging to this channel, what is the difference between here and #tech supposed to be?
[04:08:45] <MirahezeRelay>	 <ltdk> mostly because I see a lot of people mixing stuff back and forth between the two channels
[04:09:56] <MirahezeRelay>	 <cosmicalpha, replying to ltdk> This is more for SRE also where #tech is for more generic technical questions or discussions.
[04:10:24] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.05 ms
[04:10:27] <MirahezeRelay>	 <ltdk> so like, general questions rather than tasks at hand?
[04:10:35] <MirahezeRelay>	 <cosmicalpha> Yep.
[04:10:46] <MirahezeRelay>	 <ltdk> that makes sense
[04:12:21] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/627c9f562bb2...b8d4a09e4fe7
[04:12:22] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03b8d4a09 - Bot: Update SSL cert for wiki.autocountsoft.com
[04:13:52] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b8d4a09e4fe7...20b48b58694d
[04:13:55] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0320b48b5 - Bot: Update SSL cert for adadevelopersacademy.wiki
[04:16:34] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/20b48b58694d...f96af1a3722b
[04:16:36] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03f96af1a - Bot: Update SSL cert for publictestwiki.com
[04:17:48] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f96af1a3722b...03cc90aab8e3
[04:17:49] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0303cc90a - Bot: Update SSL cert for wiki.make717.org
[04:18:29] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/03cc90aab8e3...d1d42fc1abd9
[04:18:30] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03d1d42fc - Bot: Update SSL cert for wiki.tulpa.info
[04:20:51] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d1d42fc1abd9...42453b9385ee
[04:20:52] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0342453b9 - Bot: Update SSL cert for www.zenbuddhism.info
[04:38:53] <MirahezeRelay>	 <m3w, replying to cosmicalpha> if this is for sre, perhaps restrict it to only sre to avoid confusion?
[04:40:05] <MirahezeRelay>	 <elliethepwincess> > Public channel for technically-focused discussions and reporting problems related to Miraheze's technical infrastructure
[04:40:35] <MirahezeRelay>	 <elliethepwincess> i do think "technically-focused discussions" would also mean #tech
[04:46:50] <icinga-miraheze>	 RECOVERY - www.durawiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.durawiki.com' will expire on Sat 27 Jul 2024 03:07:43 AM GMT +0000.
[04:47:04] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.68 ms
[04:49:57] <MirahezeRelay>	 <cosmicalpha, replying to m3w> Its not only for SRE but rather for things that require SRE attention. #tech is more for the wider technical community.
[04:51:31] <MirahezeRelay>	 <cosmicalpha> I will update channel topics sometime to make this a more clear distinction.
[04:53:03] <MirahezeRelay>	 <m3w> [1/4] hmm what if we split this channel into 2
[04:53:04] <MirahezeRelay>	 <m3w> [2/4] - turn this into private to only SRE
[04:53:04] <MirahezeRelay>	 <m3w> [3/4] - set up a forum channel named `support-sre` where users can ping sre for attention
[04:53:04] <MirahezeRelay>	 <m3w> [4/4] this way purpose is clear and things are better focused. sre have their stuff to discuss or track separately, users can ping in a focused manner where sre can easily locate the request later if necessary
[04:53:12] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.66 ms
[04:53:47] <MirahezeRelay>	 <m3w> in any case, focus on the maintenance
[04:54:05] <icinga-miraheze>	 RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Sat 27 Jul 2024 03:12:15 AM GMT +0000.
[04:54:35] <MirahezeRelay>	 <m3w> wouldn't it additionally make sense to move tech channel under support category?
[04:57:25] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:58:08] <icinga-miraheze>	 PROBLEM - ping on jobchron171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.120)
[04:58:18] <icinga-miraheze>	 PROBLEM - ldap171 LDAP on ldap171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:58:19] <icinga-miraheze>	 PROBLEM - jobchron171 ferm_active on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[04:58:19] <icinga-miraheze>	 PROBLEM - jobchron171 PowerDNS Recursor on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[04:58:19] <icinga-miraheze>	 PROBLEM - jobchron171 Redis Process on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[04:58:20] <icinga-miraheze>	 PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: connect to address 10.0.17.124 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket
[04:58:35] <icinga-miraheze>	 PROBLEM - ping on phorge171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.124)
[04:58:36] <icinga-miraheze>	 PROBLEM - jobchron171 conntrack_table_size on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[04:58:36] <icinga-miraheze>	 PROBLEM - jobchron171 Puppet on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[04:58:39] <icinga-miraheze>	 PROBLEM - jobchron171 JobChron Service on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[04:58:40] <icinga-miraheze>	 PROBLEM - ldap171 Disk Space on ldap171 is CRITICAL: connect to address 10.0.17.121 port 5666: No route to hostconnect to host 10.0.17.121 port 5666: No route to host
[04:58:41] <icinga-miraheze>	 PROBLEM - ping on ldap171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.121)
[04:58:42] <icinga-miraheze>	 PROBLEM - reports171 SSH on reports171 is CRITICAL: connect to address 10.0.17.139 and port 22: No route to host
[04:58:46] <icinga-miraheze>	 PROBLEM - Host jobchron171 is DOWN: CRITICAL - Host Unreachable (10.0.17.120)
[04:58:46] <icinga-miraheze>	 PROBLEM - Host phorge171 is DOWN: CRITICAL - Host Unreachable (10.0.17.124)
[04:58:48] <icinga-miraheze>	 PROBLEM - mwtask171 ferm_active on mwtask171 is CRITICAL: connect to address 10.0.17.144 port 5666: No route to hostconnect to host 10.0.17.144 port 5666: No route to host
[04:58:56] <icinga-miraheze>	 PROBLEM - ldap171 conntrack_table_size on ldap171 is CRITICAL: connect to address 10.0.17.121 port 5666: No route to hostconnect to host 10.0.17.121 port 5666: No route to host
[04:58:59] <icinga-miraheze>	 PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: connect to address 10.0.17.121 port 5666: No route to hostconnect to host 10.0.17.121 port 5666: No route to host
[04:59:00] <icinga-miraheze>	 RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sat 27 Jul 2024 03:16:29 AM GMT +0000.
[04:59:02] <icinga-miraheze>	 PROBLEM - ldap171 PowerDNS Recursor on ldap171 is CRITICAL: connect to address 10.0.17.121 port 5666: No route to hostconnect to host 10.0.17.121 port 5666: No route to host
[04:59:11] <icinga-miraheze>	 PROBLEM - mwtask171 SSH on mwtask171 is CRITICAL: connect to address 10.0.17.144 and port 22: No route to host
[04:59:15] <icinga-miraheze>	 PROBLEM - ldap171 SSH on ldap171 is CRITICAL: connect to address 10.0.17.121 and port 22: No route to host
[04:59:15] <icinga-miraheze>	 PROBLEM - ldap171 NTP time on ldap171 is CRITICAL: connect to address 10.0.17.121 port 5666: No route to hostconnect to host 10.0.17.121 port 5666: No route to host
[04:59:20] <icinga-miraheze>	 PROBLEM - Host mwtask171 is DOWN: CRITICAL - Host Unreachable (10.0.17.144)
[04:59:21] <icinga-miraheze>	 PROBLEM - reports171 Current Load on reports171 is CRITICAL: connect to address 10.0.17.139 port 5666: No route to hostconnect to host 10.0.17.139 port 5666: No route to host
[04:59:21] <icinga-miraheze>	 PROBLEM - reports171 php-fpm on reports171 is CRITICAL: connect to address 10.0.17.139 port 5666: No route to hostconnect to host 10.0.17.139 port 5666: No route to host
[04:59:28] <icinga-miraheze>	 PROBLEM - reports171 NTP time on reports171 is CRITICAL: connect to address 10.0.17.139 port 5666: No route to hostconnect to host 10.0.17.139 port 5666: No route to host
[04:59:28] <icinga-miraheze>	 PROBLEM - reports171 conntrack_table_size on reports171 is CRITICAL: connect to address 10.0.17.139 port 5666: No route to hostconnect to host 10.0.17.139 port 5666: No route to host
[04:59:28] <icinga-miraheze>	 PROBLEM - reports171 Puppet on reports171 is CRITICAL: connect to address 10.0.17.139 port 5666: No route to hostconnect to host 10.0.17.139 port 5666: No route to host
[04:59:28] <icinga-miraheze>	 PROBLEM - ping on reports171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.139)
[04:59:31] <icinga-miraheze>	 PROBLEM - Host reports171 is DOWN: CRITICAL - Host Unreachable (10.0.17.139)
[04:59:34] <icinga-miraheze>	 PROBLEM - Host ldap171 is DOWN: CRITICAL - Host Unreachable (10.0.17.121)
[04:59:34] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/07650d31d452...17c267dc0b90
[04:59:35] <botifico>	 [02miraheze/puppet] 07Universal-Omega 0317c267d - Temp allow cloud ssh without bastion
[04:59:54] <icinga-miraheze>	 RECOVERY - publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Sat 27 Jul 2024 03:16:29 AM GMT +0000.
[05:03:28] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.62 ms
[05:03:31] <icinga-miraheze>	 PROBLEM - puppet181 puppetdb on puppet181 is CRITICAL: connect to address 10.0.18.100 and port 8081: No route to host
[05:03:35] <icinga-miraheze>	 PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[05:03:45] <icinga-miraheze>	 PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[05:03:47] <icinga-miraheze>	 PROBLEM - puppet181 puppetserver on puppet181 is CRITICAL: connect to address 10.0.18.100 and port 8140: No route to host
[05:03:59] <icinga-miraheze>	 PROBLEM - bast181 PowerDNS Recursor on bast181 is CRITICAL: connect to address 10.0.18.101 port 5666: No route to hostconnect to host 10.0.18.101 port 5666: No route to host
[05:04:06] <icinga-miraheze>	 PROBLEM - db171 PowerDNS Recursor on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host
[05:04:15] <icinga-miraheze>	 PROBLEM - ping on puppet181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.100)
[05:04:15] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[05:04:19] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2934 bytes in 3.256 second response time
[05:04:21] <icinga-miraheze>	 PROBLEM - bast181 conntrack_table_size on bast181 is CRITICAL: connect to address 10.0.18.101 port 5666: No route to hostconnect to host 10.0.18.101 port 5666: No route to host
[05:04:21] <icinga-miraheze>	 PROBLEM - bast181 Disk Space on bast181 is CRITICAL: connect to address 10.0.18.101 port 5666: No route to hostconnect to host 10.0.18.101 port 5666: No route to host
[05:04:27] <icinga-miraheze>	 PROBLEM - ping on swiftproxy171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.108)
[05:04:27] <icinga-miraheze>	 PROBLEM - swiftproxy171 memcached on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 and port 11211: No route to host
[05:04:29] <icinga-miraheze>	 PROBLEM - db171 SSH on db171 is CRITICAL: connect to address 10.0.17.119 and port 22: No route to host
[05:04:31] <icinga-miraheze>	 PROBLEM - bast181 SSH on bast181 is CRITICAL: connect to address 10.0.18.101 and port 22: No route to host
[05:04:31] <icinga-miraheze>	 PROBLEM - puppet181 WikiTideRenewSSL on puppet181 is CRITICAL: connect to address 10.0.18.100 and port 5000: No route to host
[05:04:31] <icinga-miraheze>	 PROBLEM - puppet181 Disk Space on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[05:04:31] <icinga-miraheze>	 PROBLEM - puppet181 Backups Private on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[05:04:32] <icinga-miraheze>	 PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:04:34] <icinga-miraheze>	 PROBLEM - puppet181 Current Load on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[05:04:38] <icinga-miraheze>	 PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 port 5666: No route to hostconnect to host 10.0.17.108 port 5666: No route to host
[05:04:39] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:04:39] <icinga-miraheze>	 PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw172.wikitide.net port 443 after 3078 ms: Couldn't connect to server
[05:04:41] <icinga-miraheze>	 PROBLEM - swiftproxy171 PowerDNS Recursor on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 port 5666: No route to hostconnect to host 10.0.17.108 port 5666: No route to host
[05:04:43] <icinga-miraheze>	 PROBLEM - Host bast181 is DOWN: CRITICAL - Host Unreachable (10.0.18.101)
[05:04:46] <icinga-miraheze>	 PROBLEM - db171 Disk Space on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host
[05:04:47] <icinga-miraheze>	 PROBLEM - Host db171 is DOWN: CRITICAL - Host Unreachable (10.0.17.119)
[05:04:47] <icinga-miraheze>	 PROBLEM - puppet181 SSH on puppet181 is CRITICAL: connect to address 10.0.18.100 and port 22: No route to host
[05:04:47] <icinga-miraheze>	 PROBLEM - puppet181 NTP time on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[05:04:58] <icinga-miraheze>	 PROBLEM - swiftproxy171 Swift Proxy on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 and port 80: No route to host
[05:05:08] <icinga-miraheze>	 PROBLEM - puppet181 conntrack_table_size on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[05:05:11] <icinga-miraheze>	 PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: connect to address 10.0.17.122 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket
[05:05:13] <icinga-miraheze>	 PROBLEM - swiftproxy171 ferm_active on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 port 5666: No route to hostconnect to host 10.0.17.108 port 5666: No route to host
[05:05:13] <icinga-miraheze>	 PROBLEM - swiftproxy171 conntrack_table_size on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 port 5666: No route to hostconnect to host 10.0.17.108 port 5666: No route to host
[05:05:14] <icinga-miraheze>	 PROBLEM - mw171 Puppet on mw171 is CRITICAL: connect to address 10.0.17.122 port 5666: No route to hostconnect to host 10.0.17.122 port 5666: No route to host
[05:05:15] <icinga-miraheze>	 PROBLEM - mw172 PowerDNS Recursor on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host
[05:05:15] <icinga-miraheze>	 PROBLEM - swiftproxy161 HTTP on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized
[05:05:15] <icinga-miraheze>	 PROBLEM - puppet181 PowerDNS Recursor on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[05:05:15] <icinga-miraheze>	 PROBLEM - puppet181 ferm_active on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[05:05:16] <icinga-miraheze>	 PROBLEM - Host swiftproxy171 is DOWN: CRITICAL - Host Unreachable (10.0.17.108)
[05:05:17] <icinga-miraheze>	 PROBLEM - mw171 ferm_active on mw171 is CRITICAL: connect to address 10.0.17.122 port 5666: No route to hostconnect to host 10.0.17.122 port 5666: No route to host
[05:05:17] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: connect to address 10.0.17.122 port 5666: No route to hostconnect to host 10.0.17.122 port 5666: No route to host
[05:05:20] <icinga-miraheze>	 PROBLEM - ping on swiftobject171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.126)
[05:05:21] <icinga-miraheze>	 PROBLEM - mw172 Disk Space on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host
[05:05:22] <icinga-miraheze>	 PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[05:05:23] <icinga-miraheze>	 PROBLEM - Host puppet181 is DOWN: CRITICAL - Host Unreachable (10.0.18.100)
[05:05:23] <icinga-miraheze>	 PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: No response from DNS 10.0.17.136
[05:05:26] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for matomo151 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:05:29] <icinga-miraheze>	 PROBLEM - swiftac171 Swift Account Service on swiftac171 is CRITICAL: connect to address 10.0.17.125 and port 6002: No route to host
[05:05:31] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.46 ms
[05:05:32] <icinga-miraheze>	 PROBLEM - mw172 NTP time on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host
[05:05:32] <icinga-miraheze>	 PROBLEM - mw172 SSH on mw172 is CRITICAL: connect to address 10.0.17.123 and port 22: No route to host
[05:05:32] <icinga-miraheze>	 PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:05:34] <icinga-miraheze>	 PROBLEM - mw171 PowerDNS Recursor on mw171 is CRITICAL: connect to address 10.0.17.122 port 5666: No route to hostconnect to host 10.0.17.122 port 5666: No route to host
[05:05:34] <icinga-miraheze>	 PROBLEM - mw171 Disk Space on mw171 is CRITICAL: connect to address 10.0.17.122 port 5666: No route to hostconnect to host 10.0.17.122 port 5666: No route to host
[05:05:37] <icinga-miraheze>	 PROBLEM - mw171 conntrack_table_size on mw171 is CRITICAL: connect to address 10.0.17.122 port 5666: No route to hostconnect to host 10.0.17.122 port 5666: No route to host
[05:05:38] <icinga-miraheze>	 PROBLEM - swiftac171 Puppet on swiftac171 is CRITICAL: connect to address 10.0.17.125 port 5666: No route to hostconnect to host 10.0.17.125 port 5666: No route to host
[05:05:38] <icinga-miraheze>	 PROBLEM - Host swiftac171 is DOWN: CRITICAL - Host Unreachable (10.0.17.125)
[05:05:38] <icinga-miraheze>	 PROBLEM - swiftac171 Disk Space on swiftac171 is CRITICAL: connect to address 10.0.17.125 port 5666: No route to hostconnect to host 10.0.17.125 port 5666: No route to host
[05:05:38] <icinga-miraheze>	 PROBLEM - ping on swiftac171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.125)
[05:05:41] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host
[05:05:41] <icinga-miraheze>	 PROBLEM - mw172 Puppet on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host
[05:05:43] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 43%
[05:05:44] <icinga-miraheze>	 PROBLEM - mw172 conntrack_table_size on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host
[05:05:46] <icinga-miraheze>	 PROBLEM - cp37 SSH on cp37 is CRITICAL: connect to address 10.0.17.138 and port 22: No route to host
[05:05:46] <icinga-miraheze>	 PROBLEM - ping on cp37 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.138)
[05:05:47] <icinga-miraheze>	 PROBLEM - mw171 NTP time on mw171 is CRITICAL: connect to address 10.0.17.122 port 5666: No route to hostconnect to host 10.0.17.122 port 5666: No route to host
[05:05:51] <icinga-miraheze>	 PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[05:05:53] <icinga-miraheze>	 PROBLEM - ping on mw172 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.123)
[05:05:56] <icinga-miraheze>	 PROBLEM - cp27 HTTPS on cp27 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[05:05:56] <icinga-miraheze>	 PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: connect to address 10.0.17.123 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket
[05:05:56] <icinga-miraheze>	 PROBLEM - mw172 php-fpm on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host
[05:05:56] <icinga-miraheze>	 PROBLEM - swiftobject171 Disk Space on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host
[05:05:58] <icinga-miraheze>	 PROBLEM - Host ns1 is DOWN: CRITICAL - Host Unreachable (10.0.17.136)
[05:05:59] <icinga-miraheze>	 PROBLEM - Host swiftobject171 is DOWN: CRITICAL - Host Unreachable (10.0.17.126)
[05:06:00] <icinga-miraheze>	 PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp37.wikitide.net port 443 after 3073 ms: Couldn't connect to server
[05:06:01] <icinga-miraheze>	 PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw171.wikitide.net port 443 after 3067 ms: Couldn't connect to server
[05:06:05] <icinga-miraheze>	 PROBLEM - bots171 SSH on bots171 is CRITICAL: connect to address 10.0.17.143 and port 22: No route to host
[05:06:07] <icinga-miraheze>	 PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:06:08] <icinga-miraheze>	 PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 977 bytes in 8.609 second response time
[05:06:08] <icinga-miraheze>	 PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[05:06:08] <icinga-miraheze>	 PROBLEM - db181 Backups SQL on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused
[05:06:10] <icinga-miraheze>	 PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw182.wikitide.net port 443 after 2711 ms: Couldn't connect to server
[05:06:13] <icinga-miraheze>	 PROBLEM - mw171 SSH on mw171 is CRITICAL: connect to address 10.0.17.122 and port 22: No route to host
[05:06:13] <icinga-miraheze>	 PROBLEM - ping on mw171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.122)
[05:06:13] <icinga-miraheze>	 PROBLEM - db181 Backups SQL mhglobal on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused
[05:06:13] <icinga-miraheze>	 PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: connect to address 10.0.18.105 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket
[05:06:15] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.206/cpweb, 2602:294:0:b23::112/cpweb
[05:06:17] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw151 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:17] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw182 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:17] <icinga-miraheze>	 PROBLEM - cp37 Puppet on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:17] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw161 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:20] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for test151 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:21] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki
[05:06:23] <icinga-miraheze>	 PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized
[05:06:24] <icinga-miraheze>	 PROBLEM - mw172 ferm_active on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host
[05:06:24] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:24] <icinga-miraheze>	 PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:24] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw152 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:24] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for phorge171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:27] <icinga-miraheze>	 PROBLEM - ping on bots171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.143)
[05:06:27] <icinga-miraheze>	 PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[05:06:27] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for swiftproxy171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:27] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for reports171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:29] <icinga-miraheze>	 PROBLEM - mw171 php-fpm on mw171 is CRITICAL: connect to address 10.0.17.122 port 5666: No route to hostconnect to host 10.0.17.122 port 5666: No route to host
[05:06:30] <icinga-miraheze>	 PROBLEM - bots171 ferm_active on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[05:06:32] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki
[05:06:33] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki
[05:06:34] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mon181 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:35] <icinga-miraheze>	 PROBLEM - cp37 NTP time on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[05:06:36] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki
[05:06:40] <icinga-miraheze>	 PROBLEM - bots171 IRC RC Bot on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[05:06:40] <icinga-miraheze>	 PROBLEM - db182 Current Load on db182 is CRITICAL: connect to address 10.0.18.103 port 5666: No route to hostconnect to host 10.0.18.103 port 5666: No route to host
[05:06:46] <icinga-miraheze>	 PROBLEM - Host cp37 is DOWN: CRITICAL - Host Unreachable (10.0.17.138)
[05:06:46] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki
[05:06:50] <icinga-miraheze>	 PROBLEM - bots171 IRC Log Bot on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[05:06:50] <icinga-miraheze>	 PROBLEM - bots171 conntrack_table_size on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[05:06:50] <icinga-miraheze>	 PROBLEM - mem151 Puppet on mem151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[05:06:53] <icinga-miraheze>	 PROBLEM - bots171 PowerDNS Recursor on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[05:06:55] <icinga-miraheze>	 PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[05:06:56] <icinga-miraheze>	 PROBLEM - Host mw171 is DOWN: CRITICAL - Host Unreachable (10.0.17.122)
[05:06:58] <icinga-miraheze>	 PROBLEM - mw181 conntrack_table_size on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: No route to hostconnect to host 10.0.18.104 port 5666: No route to host
[05:07:04] <icinga-miraheze>	 PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection
[05:07:04] <icinga-miraheze>	  on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66
[05:07:05] <icinga-miraheze>	 PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host
[05:07:06] <icinga-miraheze>	 PROBLEM - Host bots171 is DOWN: CRITICAL - Host Unreachable (10.0.17.143)
[05:07:07] <icinga-miraheze>	 PROBLEM - ping on db182 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.103)
[05:07:07] <icinga-miraheze>	 PROBLEM - Host mw172 is DOWN: CRITICAL - Host Unreachable (10.0.17.123)
[05:07:09] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host
[05:07:12] <icinga-miraheze>	 PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[05:07:12] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: No route to hostconnect to host 10.0.18.106 port 5666: No route to host
[05:07:19] <icinga-miraheze>	 PROBLEM - db181 ferm_active on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host
[05:07:20] <icinga-miraheze>	 PROBLEM - db181 Disk Space on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:07:31] <icinga-miraheze>	 PROBLEM - db181 SSH on db181 is CRITICAL: connect to address 10.0.18.102 and port 22: No route to host
[05:07:33] <icinga-miraheze>	 PROBLEM - db182 Puppet on db182 is CRITICAL: connect to address 10.0.18.103 port 5666: No route to hostconnect to host 10.0.18.103 port 5666: No route to host
[05:07:33] <icinga-miraheze>	 PROBLEM - db182 Backups SQL icingaweb2 on db182 is CRITICAL: connect to address 10.0.18.103 port 5666: No route to hostconnect to host 10.0.18.103 port 5666: No route to host
[05:07:38] <icinga-miraheze>	 PROBLEM - mwtask181 JobRunner Service on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: No route to hostconnect to host 10.0.18.106 port 5666: No route to host
[05:07:38] <icinga-miraheze>	 PROBLEM - mw181 Disk Space on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: No route to hostconnect to host 10.0.18.104 port 5666: No route to host
[05:07:38] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: No route to hostconnect to host 10.0.18.104 port 5666: No route to host
[05:07:41] <icinga-miraheze>	 PROBLEM - ping on mw181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.104)
[05:07:41] <icinga-miraheze>	 PROBLEM - mw181 PowerDNS Recursor on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: No route to hostconnect to host 10.0.18.104 port 5666: No route to host
[05:07:41] <icinga-miraheze>	 PROBLEM - mw181 NTP time on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: No route to hostconnect to host 10.0.18.104 port 5666: No route to host
[05:07:41] <icinga-miraheze>	 PROBLEM - db182 Backups SQL icinga on db182 is CRITICAL: connect to address 10.0.18.103 port 5666: No route to hostconnect to host 10.0.18.103 port 5666: No route to host
[05:07:41] <icinga-miraheze>	 PROBLEM - db181 MariaDB on db181 is UNKNOWN: <Timeout exceeded.><Terminated by signal 15 (Terminated).>
[05:07:43] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 64%
[05:07:44] <icinga-miraheze>	 PROBLEM - Host db182 is DOWN: CRITICAL - Host Unreachable (10.0.18.103)
[05:07:46] <icinga-miraheze>	 PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 65%
[05:07:48] <icinga-miraheze>	 PROBLEM - mwtask181 mathoid on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 10044: No route to host
[05:07:48] <icinga-miraheze>	 PROBLEM - mwtask181 conntrack_table_size on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: No route to hostconnect to host 10.0.18.106 port 5666: No route to host
[05:07:48] <icinga-miraheze>	 PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: No route to hostconnect to host 10.0.18.106 port 5666: No route to host
[05:07:50] <icinga-miraheze>	 PROBLEM - mw181 SSH on mw181 is CRITICAL: connect to address 10.0.18.104 and port 22: No route to host
[05:07:51] <icinga-miraheze>	 PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[05:07:57] <icinga-miraheze>	 PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 22: No route to host
[05:07:57] <icinga-miraheze>	 PROBLEM - mwtask181 NTP time on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: No route to hostconnect to host 10.0.18.106 port 5666: No route to host
[05:07:57] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: No route to hostconnect to host 10.0.18.106 port 5666: No route to host
[05:08:10] <icinga-miraheze>	 PROBLEM - Host mw182 is DOWN: CRITICAL - Host Unreachable (10.0.18.105)
[05:08:10] <icinga-miraheze>	 PROBLEM - ping on mw182 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.105)
[05:08:14] <icinga-miraheze>	 PROBLEM - mw181 ferm_active on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: No route to hostconnect to host 10.0.18.104 port 5666: No route to host
[05:08:14] <icinga-miraheze>	 PROBLEM - ping on swiftobject181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.107)
[05:08:17] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: No route to hostconnect to host 10.0.18.104 port 5666: No route to host
[05:08:17] <icinga-miraheze>	 PROBLEM - db181 NTP time on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host
[05:08:18] <icinga-miraheze>	 PROBLEM - mwtask181 ferm_active on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: No route to hostconnect to host 10.0.18.106 port 5666: No route to host
[05:08:23] <icinga-miraheze>	 PROBLEM - Host swiftobject181 is DOWN: CRITICAL - Host Unreachable (10.0.18.107)
[05:08:28] <icinga-miraheze>	 PROBLEM - db181 Puppet on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host
[05:08:28] <icinga-miraheze>	 PROBLEM - db181 conntrack_table_size on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: No route to hostconnect to host 10.0.18.102 port 5666: No route to host
[05:08:28] <icinga-miraheze>	 PROBLEM - Host mwtask181 is DOWN: CRITICAL - Host Unreachable (10.0.18.106)
[05:08:29] <icinga-miraheze>	 PROBLEM - ping on eventgate181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.147)
[05:08:29] <icinga-miraheze>	 PROBLEM - kafka181 Current Load on kafka181 is CRITICAL: connect to address 10.0.18.146 port 5666: No route to hostconnect to host 10.0.18.146 port 5666: No route to host
[05:08:32] <icinga-miraheze>	 PROBLEM - Host mw181 is DOWN: CRITICAL - Host Unreachable (10.0.18.104)
[05:08:32] <icinga-miraheze>	 PROBLEM - mw181 php-fpm on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: No route to hostconnect to host 10.0.18.104 port 5666: No route to host
[05:08:38] <icinga-miraheze>	 PROBLEM - Host kafka181 is DOWN: CRITICAL - Host Unreachable (10.0.18.146)
[05:08:41] <icinga-miraheze>	 RECOVERY - www.zenbuddhism.info - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zenbuddhism.info' will expire on Sat 27 Jul 2024 03:20:46 AM GMT +0000.
[05:08:42] <icinga-miraheze>	 PROBLEM - Host eventgate181 is DOWN: CRITICAL - Host Unreachable (10.0.18.147)
[05:08:51] <icinga-miraheze>	 PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[05:09:03] <icinga-miraheze>	 PROBLEM - Host db181 is DOWN: CRITICAL - Host Unreachable (10.0.18.102)
[06:04:38] <icinga-miraheze>	 RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Sat 27 Jul 2024 03:13:46 AM GMT +0000.
[06:04:50] <icinga-miraheze>	 PROBLEM - os151 Puppet on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host
[06:05:06] <icinga-miraheze>	 PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp36.wikitide.net port 443 after 3068 ms: Couldn't connect to server
[06:05:20] <icinga-miraheze>	 RECOVERY - wiki.tulpa.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Sat 27 Jul 2024 03:18:22 AM GMT +0000.
[06:05:41] <icinga-miraheze>	 PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:05:51] <icinga-miraheze>	 RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Sat 27 Jul 2024 03:17:42 AM GMT +0000.
[06:06:10] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:06:21] <icinga-miraheze>	 RECOVERY - Host db182 is UP: PING OK - Packet loss = 0%, RTA = 1.79 ms
[06:06:36] <icinga-miraheze>	 RECOVERY - Host mw181 is UP: PING OK - Packet loss = 0%, RTA = 1.68 ms
[06:06:52] <icinga-miraheze>	 RECOVERY - Host bast181 is UP: PING OK - Packet loss = 0%, RTA = 3.86 ms
[06:06:53] <icinga-miraheze>	 RECOVERY - db182 Backups SQL icinga on db182 is OK: FILE_AGE OK: /var/log/sql-icinga-backup-fortnightly.log is 1127131 seconds old and 120 bytes
[06:06:54] <icinga-miraheze>	 RECOVERY - mw181 Disk Space on mw181 is OK: DISK OK - free space: / 27388MiB (51% inode=86%);
[06:06:57] <icinga-miraheze>	 PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10008 milliseconds with 0 bytes received
[06:07:00] <icinga-miraheze>	 RECOVERY - mw181 PowerDNS Recursor on mw181 is OK: DNS OK: 0.347 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[06:07:16] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 1.69, 0.59, 0.21
[06:07:23] <icinga-miraheze>	 RECOVERY - db182 Backups SQL icingaweb2 on db182 is OK: FILE_AGE OK: /var/log/sql-icingaweb2-backup-fortnightly.log is 1127232 seconds old and 124 bytes
[06:07:27] <icinga-miraheze>	 RECOVERY - bast181 Disk Space on bast181 is OK: DISK OK - free space: / 5596MiB (63% inode=86%);
[06:07:30] <icinga-miraheze>	 RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:07:33] <icinga-miraheze>	 RECOVERY - Host puppet181 is UP: PING OK - Packet loss = 0%, RTA = 3.29 ms
[06:07:36] <icinga-miraheze>	 RECOVERY - Host mwtask181 is UP: PING OK - Packet loss = 0%, RTA = 3.22 ms
[06:07:39] <icinga-miraheze>	 RECOVERY - mwtask181 ferm_active on mwtask181 is OK: OK ferm input default policy is set
[06:07:46] <icinga-miraheze>	 RECOVERY - mwtask181 conntrack_table_size on mwtask181 is OK: OK: nf_conntrack is 0 % full
[06:07:48] <icinga-miraheze>	 RECOVERY - Host swiftobject181 is UP: PING OK - Packet loss = 0%, RTA = 2.34 ms
[06:07:49] <icinga-miraheze>	 RECOVERY - bast181 PowerDNS Recursor on bast181 is OK: DNS OK: 2.202 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[06:07:49] <icinga-miraheze>	 RECOVERY - puppet181 NTP time on puppet181 is OK: NTP OK: Offset 0.004798620939 secs
[06:07:53] <icinga-miraheze>	 RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 21410MiB (70% inode=91%);
[06:07:58] <icinga-miraheze>	 RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[06:08:04] <icinga-miraheze>	 RECOVERY - mwtask181 NTP time on mwtask181 is OK: NTP OK: Offset 0.003122001886 secs
[06:08:05] <icinga-miraheze>	 RECOVERY - bast181 SSH on bast181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:08:05] <icinga-miraheze>	 RECOVERY - Host eventgate181 is UP: PING OK - Packet loss = 0%, RTA = 3.27 ms
[06:08:05] <icinga-miraheze>	 RECOVERY - mw181 NTP time on mw181 is OK: NTP OK: Offset 0.002048075199 secs
[06:08:11] <icinga-miraheze>	 RECOVERY - Host db181 is UP: PING OK - Packet loss = 0%, RTA = 2.63 ms
[06:08:12] <icinga-miraheze>	 RECOVERY - Host mw182 is UP: PING OK - Packet loss = 0%, RTA = 3.35 ms
[06:08:12] <icinga-miraheze>	 RECOVERY - ping on swiftobject181 is OK: PING OK - Packet loss = 0%, RTA = 3.11 ms
[06:08:17] <icinga-miraheze>	 RECOVERY - ping on eventgate181 is OK: PING OK - Packet loss = 0%, RTA = 3.55 ms
[06:08:20] <icinga-miraheze>	 RECOVERY - Host kafka181 is UP: PING OK - Packet loss = 0%, RTA = 5.12 ms
[06:08:21] <icinga-miraheze>	 PROBLEM - os162 ferm_active on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[06:08:21] <icinga-miraheze>	 RECOVERY - mw181 ferm_active on mw181 is OK: OK ferm input default policy is set
[06:08:21] <icinga-miraheze>	 RECOVERY - db181 conntrack_table_size on db181 is OK: OK: nf_conntrack is 0 % full
[06:08:22] <icinga-miraheze>	 RECOVERY - puppet181 conntrack_table_size on puppet181 is OK: OK: nf_conntrack is 0 % full
[06:08:23] <icinga-miraheze>	 PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw161.wikitide.net port 443 after 2281 ms: Couldn't connect to server
[06:08:27] <icinga-miraheze>	 RECOVERY - db181 MariaDB on db181 is OK: Uptime: 221 Threads: 5 Questions: 1036 Slow queries: 0 Opens: 505 Open tables: 499 Queries per second avg: 4.687
[06:08:27] <icinga-miraheze>	 RECOVERY - puppet181 Current Load on puppet181 is OK: LOAD OK - total load average: 0.86, 1.34, 0.66
[06:08:27] <icinga-miraheze>	 RECOVERY - db181 Disk Space on db181 is OK: DISK OK - free space: / 664155MiB (74% inode=98%);
[06:08:27] <icinga-miraheze>	 RECOVERY - puppet181 PowerDNS Recursor on puppet181 is OK: DNS OK: 2.192 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[06:08:27] <icinga-miraheze>	 RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 788471 seconds old and 6541 bytes
[06:08:27] <icinga-miraheze>	 PROBLEM - kafka181 NTP time on kafka181 is WARNING: NTP WARNING: Offset 0.1888979673 secs
[06:08:29] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 0.23, 0.23, 0.10
[06:08:32] <icinga-miraheze>	 RECOVERY - mw181 php-fpm on mw181 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2'
[06:08:32] <icinga-miraheze>	 RECOVERY - db182 Current Load on db182 is OK: LOAD OK - total load average: 0.04, 0.08, 0.03
[06:08:32] <icinga-miraheze>	 RECOVERY - ping on db182 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms
[06:08:33] <icinga-miraheze>	 RECOVERY - kafka181 Current Load on kafka181 is OK: LOAD OK - total load average: 0.19, 0.16, 0.06
[06:08:34] <icinga-miraheze>	 PROBLEM - os162 Puppet on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[06:08:37] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is WARNING: WARNING: Puppet last ran 1 hour ago
[06:08:37] <icinga-miraheze>	 RECOVERY - puppet181 puppetdb on puppet181 is OK: TCP OK - 0.006 second response time on 10.0.18.100 port 8081
[06:08:37] <icinga-miraheze>	 RECOVERY - bast181 conntrack_table_size on bast181 is OK: OK: nf_conntrack is 0 % full
[06:08:37] <icinga-miraheze>	 RECOVERY - puppet181 WikiTideRenewSSL on puppet181 is OK: TCP OK - 0.000 second response time on 10.0.18.100 port 5000
[06:08:37] <icinga-miraheze>	 RECOVERY - ping on mw181 is OK: PING OK - Packet loss = 0%, RTA = 0.82 ms
[06:08:37] <icinga-miraheze>	 RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 140904MiB (63% inode=94%);
[06:08:37] <icinga-miraheze>	 RECOVERY - mw181 SSH on mw181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:08:38] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet last ran 1 hour ago
[06:08:52] <icinga-miraheze>	 RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:08:52] <icinga-miraheze>	 RECOVERY - puppet181 puppetserver on puppet181 is OK: TCP OK - 0.000 second response time on 10.0.18.100 port 8140
[06:08:52] <icinga-miraheze>	 RECOVERY - puppet181 Backups Private on puppet181 is OK: FILE_AGE OK: /var/log/private-backup.log is 11040 seconds old and 92 bytes
[06:08:52] <icinga-miraheze>	 RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 0.035 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[06:08:57] <icinga-miraheze>	 RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.02, 0.09, 0.05
[06:08:57] <icinga-miraheze>	 RECOVERY - ping on puppet181 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms
[06:09:06] <icinga-miraheze>	 PROBLEM - Host os162 is DOWN: CRITICAL - Host Unreachable (10.0.16.141)
[06:09:07] <icinga-miraheze>	 RECOVERY - db181 ferm_active on db181 is OK: OK ferm input default policy is set
[06:09:07] <icinga-miraheze>	 RECOVERY - puppet181 ferm_active on puppet181 is OK: OK ferm input default policy is set
[06:09:16] <icinga-miraheze>	 PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:09:18] <icinga-miraheze>	 PROBLEM - mem151 NTP time on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host
[06:09:18] <icinga-miraheze>	 PROBLEM - mem151 ferm_active on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host
[06:09:19] <icinga-miraheze>	 PROBLEM - mw162 Puppet on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[06:09:19] <icinga-miraheze>	 PROBLEM - mw162 PowerDNS Recursor on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[06:09:19] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[06:09:19] <icinga-miraheze>	 PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host
[06:09:20] <icinga-miraheze>	 PROBLEM - mw151 NTP time on mw151 is CRITICAL: connect to address 10.0.15.114 port 5666: No route to hostconnect to host 10.0.15.114 port 5666: No route to host
[06:09:20] <icinga-miraheze>	 PROBLEM - bast161 NTP time on bast161 is CRITICAL: connect to address 10.0.16.127 port 5666: No route to hostconnect to host 10.0.16.127 port 5666: No route to host
[06:09:20] <icinga-miraheze>	 PROBLEM - bast161 ferm_active on bast161 is CRITICAL: connect to address 10.0.16.127 port 5666: No route to hostconnect to host 10.0.16.127 port 5666: No route to host
[06:09:20] <icinga-miraheze>	 PROBLEM - bast161 SSH on bast161 is CRITICAL: connect to address 10.0.16.127 and port 22: No route to host
[06:09:20] <icinga-miraheze>	 PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:09:21] <icinga-miraheze>	 PROBLEM - mem161 SSH on mem161 is CRITICAL: connect to address 10.0.16.131 and port 22: No route to host
[06:11:19] <icinga-miraheze>	 RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[06:12:10] <icinga-miraheze>	 RECOVERY - mon181 Current Load on mon181 is OK: LOAD OK - total load average: 0.72, 6.36, 4.37
[06:12:29] <icinga-miraheze>	 RECOVERY - Host cloud15 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms
[06:13:23] <icinga-miraheze>	 RECOVERY - Host os151 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms
[06:13:31] <icinga-miraheze>	 RECOVERY - Host mem151 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[06:13:35] <icinga-miraheze>	 RECOVERY - Host db151 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms
[06:13:37] <icinga-miraheze>	 RECOVERY - cloud15 SSH on cloud15 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:13:41] <icinga-miraheze>	 RECOVERY - Host mw152 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms
[06:13:43] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[06:13:46] <icinga-miraheze>	 RECOVERY - Host swiftobject151 is UP: PING OK - Packet loss = 0%, RTA = 127.37 ms
[06:13:47] <icinga-miraheze>	 PROBLEM - os151 NTP time on os151 is CRITICAL: NTP CRITICAL: Offset 1.126735121 secs
[06:13:47] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms
[06:13:57] <icinga-miraheze>	 PROBLEM - db151 NTP time on db151 is WARNING: NTP WARNING: Offset 0.4539371133 secs
[06:14:08] <icinga-miraheze>	 RECOVERY - Host test151 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[06:14:26] <icinga-miraheze>	 PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 55%
[06:15:01] <icinga-miraheze>	 PROBLEM - Host cloud17 is DOWN: PING CRITICAL - Packet loss = 100%
[06:15:05] <icinga-miraheze>	 RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 0.59, 0.34, 0.13
[06:15:11] <icinga-miraheze>	 RECOVERY - swiftobject151 Swift Object Service on swiftobject151 is OK: TCP OK - 0.000 second response time on 10.0.15.117 port 6000
[06:15:12] <icinga-miraheze>	 RECOVERY - mem151 ferm_active on mem151 is OK: OK ferm input default policy is set
[06:15:12] <icinga-miraheze>	 RECOVERY - db151 conntrack_table_size on db151 is OK: OK: nf_conntrack is 0 % full
[06:15:12] <icinga-miraheze>	 RECOVERY - mw152 Disk Space on mw152 is OK: DISK OK - free space: / 27369MiB (51% inode=86%);
[06:15:13] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet last ran 1 hour ago
[06:15:13] <icinga-miraheze>	 RECOVERY - ping on swiftobject151 is OK: PING OK - Packet loss = 0%, RTA = 0.21 ms
[06:15:13] <icinga-miraheze>	 RECOVERY - mem151 SSH on mem151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:15:13] <icinga-miraheze>	 RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[06:15:14] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 0.64, 0.25, 0.09
[06:15:16] <icinga-miraheze>	 RECOVERY - Host matomo151 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms
[06:15:16] <icinga-miraheze>	 RECOVERY - ping on mem151 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms
[06:15:16] <icinga-miraheze>	 RECOVERY - test151 ferm_active on test151 is OK: OK ferm input default policy is set
[06:15:17] <icinga-miraheze>	 RECOVERY - Host mw151 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms
[06:15:17] <icinga-miraheze>	 RECOVERY - db151 Backups SQL mhglobal on db151 is OK: FILE_AGE OK: /var/log/sql-mhglobal-backup-weekly.log is 4505 seconds old and 208 bytes
[06:15:17] <icinga-miraheze>	 RECOVERY - db151 Disk Space on db151 is OK: DISK OK - free space: / 521209MiB (58% inode=98%);
[06:15:29] <icinga-miraheze>	 RECOVERY - Host graphite151 is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms
[06:15:30] <icinga-miraheze>	 RECOVERY - Host changeprop151 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[06:15:31] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet last ran 1 hour ago
[06:15:33] <icinga-miraheze>	 RECOVERY - swiftobject151 NTP time on swiftobject151 is OK: NTP OK: Offset 0.000169724226 secs
[06:15:34] <icinga-miraheze>	 RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.079 second response time
[06:15:37] <icinga-miraheze>	 RECOVERY - swiftobject151 Disk Space on swiftobject151 is OK: DISK OK - free space: / 479885MiB (35% inode=88%);
[06:15:40] <icinga-miraheze>	 RECOVERY - test151 conntrack_table_size on test151 is OK: OK: nf_conntrack is 0 % full
[06:15:41] <icinga-miraheze>	 RECOVERY - swiftobject151 PowerDNS Recursor on swiftobject151 is OK: DNS OK: 0.355 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[06:15:45] <icinga-miraheze>	 RECOVERY - os151 NTP time on os151 is OK: NTP OK: Offset 0.002775460482 secs
[06:15:47] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset 0.0008540451527 secs
[06:15:47] <icinga-miraheze>	 RECOVERY - swiftobject151 conntrack_table_size on swiftobject151 is OK: OK: nf_conntrack is 0 % full
[06:15:52] <icinga-miraheze>	 RECOVERY - test151 Current Load on test151 is OK: LOAD OK - total load average: 0.42, 0.18, 0.07
[06:15:57] <icinga-miraheze>	 RECOVERY - db151 NTP time on db151 is OK: NTP OK: Offset 0.002764850855 secs
[06:15:57] <icinga-miraheze>	 RECOVERY - test151 Disk Space on test151 is OK: DISK OK - free space: / 64407MiB (72% inode=84%);
[06:16:10] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:16:27] <icinga-miraheze>	 RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 552 bytes in 0.143 second response time
[06:16:29] <icinga-miraheze>	 RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:16:31] <icinga-miraheze>	 RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:17:12] <icinga-miraheze>	 RECOVERY - graphite151 Puppet on graphite151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:17:17] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:17:17] <icinga-miraheze>	 RECOVERY - changeprop151 ferm_active on changeprop151 is OK: OK ferm input default policy is set
[06:17:17] <icinga-miraheze>	 RECOVERY - changeprop151 PowerDNS Recursor on changeprop151 is OK: DNS OK: 0.027 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[06:17:17] <icinga-miraheze>	 RECOVERY - ping on prometheus151 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms
[06:17:27] <icinga-miraheze>	 RECOVERY - changeprop151 Redis Process on changeprop151 is OK: PROCS OK: 1 process with args 'redis-server'
[06:17:52] <icinga-miraheze>	 PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[06:18:18] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 5 seconds ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet],Exec[git_pull_ssl],Exec[git_pull_mediawiki-repos]
[06:19:25] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 55 seconds ago with 10 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_landing],Exec[git_pull_ErrorPages],Exec[git_pull_femiwiki-deploy-1.41]
[06:20:39] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 43 seconds ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.41],Exec[git_pull_femiwiki-deploy-1.42],Exec[git_pull_3d2png]
[06:20:39] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 31 seconds ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.41],Exec[git_pull_femiwiki-deploy-1.42],Exec[git_pull_3d2png]
[06:20:43] <icinga-miraheze>	 RECOVERY - Host cp37 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[06:20:51] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for matomo151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8203
[06:20:55] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw182 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8120
[06:21:24] <icinga-miraheze>	 RECOVERY - Host mwtask171 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms
[06:21:28] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw152 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8114
[06:21:30] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for test151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8181
[06:21:33] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for swiftproxy171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8207
[06:21:49] <icinga-miraheze>	 RECOVERY - mwtask171 SSH on mwtask171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:21:56] <icinga-miraheze>	 RECOVERY - Host cloud17 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms
[06:22:11] <icinga-miraheze>	 RECOVERY - ping on cp37 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms
[06:22:14] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8115
[06:22:17] <icinga-miraheze>	 RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[06:22:21] <icinga-miraheze>	 PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 65%
[06:22:21] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8113
[06:22:27] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mon181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8201
[06:22:27] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet last ran 1 hour ago
[06:22:27] <icinga-miraheze>	 RECOVERY - cp37 NTP time on cp37 is OK: NTP OK: Offset -0.0005082786083 secs
[06:22:31] <icinga-miraheze>	 RECOVERY - mwtask171 ferm_active on mwtask171 is OK: OK ferm input default policy is set
[06:22:32] <icinga-miraheze>	 PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 7424 bytes in 4.772 second response time
[06:22:35] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for phorge171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8202
[06:22:37] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for reports171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8205
[06:22:37] <icinga-miraheze>	 RECOVERY - cp37 SSH on cp37 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:22:47] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.95 ms
[06:22:49] <icinga-miraheze>	 RECOVERY - Host jobchron171 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[06:22:59] <icinga-miraheze>	 RECOVERY - eventgate181 eventgate on eventgate181 is OK: TCP OK - 0.000 second response time on 10.0.18.147 port 8192
[06:23:02] <icinga-miraheze>	 RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:23:11] <icinga-miraheze>	 RECOVERY - Host swiftproxy171 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms
[06:23:21] <icinga-miraheze>	 RECOVERY - swiftproxy171 memcached on swiftproxy171 is OK: TCP OK - 0.000 second response time on 10.0.17.108 port 11211
[06:23:30] <icinga-miraheze>	 RECOVERY - Host mw172 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[06:23:32] <icinga-miraheze>	 RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[06:23:33] <icinga-miraheze>	 RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:23:42] <icinga-miraheze>	 RECOVERY - Host phorge171 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms
[06:23:49] <icinga-miraheze>	 RECOVERY - Host swiftobject171 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms
[06:23:54] <icinga-miraheze>	 RECOVERY - ping on phorge171 is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms
[06:23:55] <icinga-miraheze>	 RECOVERY - swiftproxy171 ferm_active on swiftproxy171 is OK: OK ferm input default policy is set
[06:23:57] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable
[06:24:02] <icinga-miraheze>	 RECOVERY - swiftproxy171 PowerDNS Recursor on swiftproxy171 is OK: DNS OK: 0.348 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[06:24:04] <icinga-miraheze>	 RECOVERY - mw172 SSH on mw172 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:24:05] <icinga-miraheze>	 RECOVERY - mw172 conntrack_table_size on mw172 is OK: OK: nf_conntrack is 1 % full
[06:24:06] <icinga-miraheze>	 RECOVERY - mw172 php-fpm on mw172 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2'
[06:24:08] <icinga-miraheze>	 RECOVERY - Host mw171 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms
[06:24:17] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTP on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable
[06:24:18] <icinga-miraheze>	 RECOVERY - mw171 ferm_active on mw171 is OK: OK ferm input default policy is set
[06:24:18] <icinga-miraheze>	 RECOVERY - ping on swiftproxy171 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms
[06:24:19] <icinga-miraheze>	 PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 54%
[06:24:23] <icinga-miraheze>	 RECOVERY - Host bots171 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms
[06:24:24] <icinga-miraheze>	 PROBLEM - cloud16 PowerDNS Recursor on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:24:25] <icinga-miraheze>	 RECOVERY - mw172 Disk Space on mw172 is OK: DISK OK - free space: / 30665MiB (57% inode=86%);
[06:24:26] <icinga-miraheze>	 RECOVERY - bots171 conntrack_table_size on bots171 is OK: OK: nf_conntrack is 0 % full
[06:24:27] <icinga-miraheze>	 RECOVERY - ping on jobchron171 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms
[06:24:29] <icinga-miraheze>	 RECOVERY - Host db171 is UP: PING OK - Packet loss = 0%, RTA = 0.58 ms
[06:24:30] <icinga-miraheze>	 RECOVERY - Host ns1 is UP: PING OK - Packet loss = 0%, RTA = 1.04 ms
[06:24:30] <icinga-miraheze>	 RECOVERY - ping on mw171 is OK: PING OK - Packet loss = 0%, RTA = 0.20 ms
[06:24:31] <icinga-miraheze>	 RECOVERY - jobchron171 Redis Process on jobchron171 is OK: PROCS OK: 1 process with args 'redis-server'
[06:24:32] <icinga-miraheze>	 RECOVERY - swiftproxy171 conntrack_table_size on swiftproxy171 is OK: OK: nf_conntrack is 1 % full
[06:24:32] <icinga-miraheze>	 RECOVERY - jobchron171 conntrack_table_size on jobchron171 is OK: OK: nf_conntrack is 0 % full
[06:24:33] <icinga-miraheze>	 RECOVERY - jobchron171 ferm_active on jobchron171 is OK: OK ferm input default policy is set
[06:24:36] <icinga-miraheze>	 RECOVERY - mw171 NTP time on mw171 is OK: NTP OK: Offset 0.01086261868 secs
[06:24:36] <icinga-miraheze>	 RECOVERY - ping on swiftobject171 is OK: PING OK - Packet loss = 0%, RTA = 0.21 ms
[06:24:37] <icinga-miraheze>	 RECOVERY - db171 Disk Space on db171 is OK: DISK OK - free space: / 660749MiB (74% inode=98%);
[06:24:37] <icinga-miraheze>	 RECOVERY - jobchron171 JobChron Service on jobchron171 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[06:24:38] <icinga-miraheze>	 RECOVERY - Host ldap171 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms
[06:24:39] <icinga-miraheze>	 RECOVERY - Host reports171 is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms
[06:24:41] <icinga-miraheze>	 RECOVERY - mw171 Disk Space on mw171 is OK: DISK OK - free space: / 33659MiB (63% inode=87%);
[06:24:42] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 0.61, 0.24, 0.09
[06:24:42] <icinga-miraheze>	 PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet last ran 1 hour ago
[06:24:42] <icinga-miraheze>	 RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:24:42] <icinga-miraheze>	 RECOVERY - jobchron171 PowerDNS Recursor on jobchron171 is OK: DNS OK: 0.031 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[06:24:43] <icinga-miraheze>	 PROBLEM - cloud16 SSH on cloud16 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:24:44] <icinga-miraheze>	 RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19090 bytes in 0.058 second response time
[06:24:45] <icinga-miraheze>	 RECOVERY - mw172 ferm_active on mw172 is OK: OK ferm input default policy is set
[06:24:45] <icinga-miraheze>	 RECOVERY - bots171 PowerDNS Recursor on bots171 is OK: DNS OK: 0.050 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[06:24:47] <icinga-miraheze>	 RECOVERY - ping on mw172 is OK: PING OK - Packet loss = 0%, RTA = 0.19 ms
[06:24:47] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.90 ms
[06:24:47] <icinga-miraheze>	 RECOVERY - mw171 PowerDNS Recursor on mw171 is OK: DNS OK: 0.025 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[06:24:48] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 100%
[06:24:52] <icinga-miraheze>	 RECOVERY - swiftproxy171 Swift Proxy on swiftproxy171 is OK: TCP OK - 0.000 second response time on 10.0.17.108 port 80
[06:24:52] <icinga-miraheze>	 PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet last ran 1 hour ago
[06:24:56] <icinga-miraheze>	 RECOVERY - swiftobject171 Disk Space on swiftobject171 is OK: DISK OK - free space: / 480689MiB (35% inode=88%);
[06:24:57] <icinga-miraheze>	 RECOVERY - db171 PowerDNS Recursor on db171 is OK: DNS OK: 0.035 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[06:24:59] <icinga-miraheze>	 RECOVERY - Host swiftac171 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms
[06:25:00] <icinga-miraheze>	 RECOVERY - bots171 IRC RC Bot on bots171 is OK: PROCS OK: 1 process with args 'ircrcbot.py'
[06:25:00] <icinga-miraheze>	 RECOVERY - bots171 IRC Log Bot on bots171 is OK: PROCS OK: 1 process with args 'adminlogbot.py'
[06:25:02] <icinga-miraheze>	 RECOVERY - ping on ldap171 is OK: PING OK - Packet loss = 0%, RTA = 0.27 ms
[06:25:07] <icinga-miraheze>	 RECOVERY - bots171 ferm_active on bots171 is OK: OK ferm input default policy is set
[06:25:07] <icinga-miraheze>	 RECOVERY - mw172 PowerDNS Recursor on mw172 is OK: DNS OK: 0.035 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[06:25:07] <icinga-miraheze>	 RECOVERY - mw171 conntrack_table_size on mw171 is OK: OK: nf_conntrack is 1 % full
[06:25:12] <icinga-miraheze>	 RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:25:12] <icinga-miraheze>	 RECOVERY - ldap171 PowerDNS Recursor on ldap171 is OK: DNS OK: 0.031 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[06:25:12] <icinga-miraheze>	 RECOVERY - db171 SSH on db171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:25:13] <icinga-miraheze>	 RECOVERY - reports171 Current Load on reports171 is OK: LOAD OK - total load average: 0.22, 0.12, 0.05
[06:25:19] <icinga-miraheze>	 RECOVERY - ping on bots171 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms
[06:25:22] <icinga-miraheze>	 RECOVERY - swiftac171 Disk Space on swiftac171 is OK: DISK OK - free space: / 74491MiB (83% inode=91%);
[06:25:27] <icinga-miraheze>	 RECOVERY - mw171 SSH on mw171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:25:27] <icinga-miraheze>	 RECOVERY - ldap171 NTP time on ldap171 is OK: NTP OK: Offset -0.004688590765 secs
[06:25:32] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 0.33, 0.21, 0.09
[06:25:47] <icinga-miraheze>	 RECOVERY - mw171 php-fpm on mw171 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2'
[06:25:47] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb
[06:25:52] <icinga-miraheze>	 RECOVERY - ldap171 SSH on ldap171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:25:57] <icinga-miraheze>	 RECOVERY - reports171 NTP time on reports171 is OK: NTP OK: Offset -0.01364150643 secs
[06:25:57] <icinga-miraheze>	 RECOVERY - reports171 php-fpm on reports171 is OK: PROCS OK: 13 processes with command name 'php-fpm8.2'
[06:25:58] <icinga-miraheze>	 PROBLEM - cloud16 Current Load on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:26:02] <icinga-miraheze>	 RECOVERY - bots171 SSH on bots171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:26:02] <icinga-miraheze>	 RECOVERY - reports171 conntrack_table_size on reports171 is OK: OK: nf_conntrack is 0 % full
[06:26:02] <icinga-miraheze>	 RECOVERY - ldap171 Disk Space on ldap171 is OK: DISK OK - free space: / 5808MiB (66% inode=86%);
[06:26:02] <icinga-miraheze>	 RECOVERY - ldap171 conntrack_table_size on ldap171 is OK: OK: nf_conntrack is 0 % full
[06:26:07] <icinga-miraheze>	 RECOVERY - ping on swiftac171 is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms
[06:26:07] <icinga-miraheze>	 PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet last ran 1 hour ago
[06:26:07] <icinga-miraheze>	 RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:26:09] <icinga-miraheze>	 PROBLEM - Host cloud16 is DOWN: PING CRITICAL - Packet loss = 100%
[06:26:12] <icinga-miraheze>	 RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot'
[06:26:15] <icinga-miraheze>	 RECOVERY - mw172 NTP time on mw172 is OK: NTP OK: Offset -0.001404821873 secs
[06:26:17] <icinga-miraheze>	 RECOVERY - reports171 SSH on reports171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:26:22] <icinga-miraheze>	 RECOVERY - ping on reports171 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms
[06:26:22] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is WARNING: WARNING: Puppet last ran 1 hour ago
[06:26:22] <icinga-miraheze>	 RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.041 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[06:26:32] <icinga-miraheze>	 RECOVERY - swiftac171 Swift Account Service on swiftac171 is OK: TCP OK - 0.000 second response time on 10.0.17.125 port 6002
[06:26:32] <icinga-miraheze>	 RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[06:26:32] <icinga-miraheze>	 RECOVERY - ldap171 LDAP on ldap171 is OK: LDAP OK - 0.006 seconds response time
[06:28:35] <icinga-miraheze>	 PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 26 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_phorge-extensions]
[06:28:39] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 27 seconds ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.41],Exec[git_pull_femiwiki-deploy-1.42],Exec[git_pull_3d2png]
[06:31:49] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:33:08] <icinga-miraheze>	 RECOVERY - Host cloud16 is UP: PING OK - Packet loss = 0%, RTA = 0.17 ms
[06:33:10] <icinga-miraheze>	 RECOVERY - Host os162 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[06:33:29] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[06:33:35] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[06:33:40] <icinga-miraheze>	 RECOVERY - os162 ferm_active on os162 is OK: OK ferm input default policy is set
[06:33:42] <icinga-miraheze>	 RECOVERY - Host cp36 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms
[06:33:42] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw182 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8120
[06:33:45] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mwtask181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8160
[06:33:47] <icinga-miraheze>	 RECOVERY - cloud16 Current Load on cloud16 is OK: LOAD OK - total load average: 3.54, 1.15, 0.41
[06:34:07] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_IRC-Discord-Relay-relaybot],Exec[git_pull_IRC-Discord-Relay-relaybot2]
[06:34:13] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[06:34:27] <icinga-miraheze>	 RECOVERY - cloud16 PowerDNS Recursor on cloud16 is OK: DNS OK: 0.027 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[06:34:29] <icinga-miraheze>	 PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 40 seconds ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.41],Exec[git_pull_femiwiki-deploy-1.42]
[06:34:30] <icinga-miraheze>	 RECOVERY - os162 NTP time on os162 is OK: NTP OK: Offset -0.001365929842 secs
[06:34:31] <icinga-miraheze>	 RECOVERY - Host os161 is UP: PING OK - Packet loss = 0%, RTA = 3.73 ms
[06:34:31] <icinga-miraheze>	 RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[06:34:34] <icinga-miraheze>	 RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 7.617 second response time
[06:34:35] <icinga-miraheze>	 RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.565 second response time
[06:34:35] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.350 second response time
[06:34:43] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for swiftproxy161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8206
[06:34:43] <icinga-miraheze>	 RECOVERY - cp36 PowerDNS Recursor on cp36 is OK: DNS OK: 0.051 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:34:43] <icinga-miraheze>	 RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.361 second response time
[06:34:45] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8119
[06:34:45] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8113
[06:34:46] <icinga-miraheze>	 RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.333 second response time
[06:34:47] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for matomo151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8203
[06:34:47] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 59%
[06:34:47] <icinga-miraheze>	 RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.104 second response time
[06:34:47] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mon181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8201
[06:34:49] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw152 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8114
[06:34:52] <icinga-miraheze>	 RECOVERY - Host bast161 is UP: PING OK - Packet loss = 0%, RTA = 4.52 ms
[06:34:52] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for puppet181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8204
[06:34:55] <icinga-miraheze>	 RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.393 second response time
[06:34:57] <icinga-miraheze>	 RECOVERY - cloud16 SSH on cloud16 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:35:00] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:35:00] <icinga-miraheze>	 RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -0.0004627108574 secs
[06:35:02] <icinga-miraheze>	 RECOVERY - os161 ferm_active on os161 is OK: OK ferm input default policy is set
[06:35:02] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 4.57 ms
[06:35:03] <icinga-miraheze>	 RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.103 second response time
[06:35:03] <icinga-miraheze>	 RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3758 bytes in 0.100 second response time
[06:35:03] <icinga-miraheze>	 RECOVERY - cp36 NTP time on cp36 is OK: NTP OK: Offset 0.001919865608 secs
[06:35:04] <icinga-miraheze>	 RECOVERY - os161 SSH on os161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:35:08] <icinga-miraheze>	 RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 1.01, 0.39, 0.14
[06:35:11] <icinga-miraheze>	 RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.087 second response time
[06:35:12] <icinga-miraheze>	 RECOVERY - cp36 SSH on cp36 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:35:17] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for reports171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8205
[06:35:21] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.520 second response time
[06:35:22] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw162 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8116
[06:35:22] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8115
[06:35:23] <icinga-miraheze>	 RECOVERY - cp36 Current Load on cp36 is OK: LOAD OK - total load average: 0.19, 0.11, 0.04
[06:35:23] <icinga-miraheze>	 RECOVERY - Host mw162 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms
[06:35:23] <icinga-miraheze>	 RECOVERY - bast161 SSH on bast161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:35:26] <icinga-miraheze>	 RECOVERY - Host mw161 is UP: PING OK - Packet loss = 0%, RTA = 3.47 ms
[06:35:27] <icinga-miraheze>	 RECOVERY - Host mem161 is UP: PING OK - Packet loss = 0%, RTA = 1.47 ms
[06:35:27] <icinga-miraheze>	 RECOVERY - ping on bast161 is OK: PING OK - Packet loss = 0%, RTA = 1.56 ms
[06:35:27] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw172 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8118
[06:35:28] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mwtask171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8161
[06:35:28] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for swiftproxy171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8207
[06:35:28] <icinga-miraheze>	 RECOVERY - os161 conntrack_table_size on os161 is OK: OK: nf_conntrack is 0 % full
[06:35:29] <icinga-miraheze>	 RECOVERY - ping on mem161 is OK: PING OK - Packet loss = 0%, RTA = 1.47 ms
[06:35:32] <icinga-miraheze>	 RECOVERY - Host swiftproxy161 is UP: PING OK - Packet loss = 0%, RTA = 3.74 ms
[06:35:32] <icinga-miraheze>	 RECOVERY - os161 PowerDNS Recursor on os161 is OK: DNS OK: 0.387 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:35:33] <icinga-miraheze>	 RECOVERY - cp36 conntrack_table_size on cp36 is OK: OK: nf_conntrack is 1 % full
[06:35:33] <icinga-miraheze>	 RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 22663MiB (25% inode=98%);
[06:35:33] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for phorge171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8202
[06:35:33] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for test151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8181
[06:35:33] <icinga-miraheze>	 RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3762 bytes in 1.655 second response time
[06:35:34] <icinga-miraheze>	 RECOVERY - os161 NTP time on os161 is OK: NTP OK: Offset 0.0004554092884 secs
[06:35:34] <icinga-miraheze>	 RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 154061MiB (69% inode=99%);
[06:35:48] <icinga-miraheze>	 RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.107 second response time
[06:35:53] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.480 second response time
[06:35:54] <icinga-miraheze>	 RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.101 second response time
[06:35:54] <icinga-miraheze>	 RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.333 second response time
[06:35:55] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.083 second response time
[06:36:02] <icinga-miraheze>	 RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3758 bytes in 0.083 second response time
[06:36:03] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 1 minute ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_landing],Exec[ErrorPages Sync]
[06:36:08] <icinga-miraheze>	 PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.41],Exec[git_pull_femiwiki-deploy-1.42]
[06:36:10] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.357 second response time
[06:36:10] <icinga-miraheze>	 RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 7%
[06:36:17] <icinga-miraheze>	 RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.091 second response time
[06:36:18] <icinga-miraheze>	 RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[06:36:20] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[06:36:22] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.301 second response time
[06:36:26] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[06:36:28] <icinga-miraheze>	 RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3759 bytes in 0.917 second response time
[06:36:31] <icinga-miraheze>	 RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3784 bytes in 0.950 second response time
[06:36:32] <icinga-miraheze>	 RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.086 second response time
[06:36:36] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy
[06:36:37] <icinga-miraheze>	 RECOVERY - bast161 ferm_active on bast161 is OK: OK ferm input default policy is set
[06:36:38] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[06:36:42] <icinga-miraheze>	 RECOVERY - mw161 php-fpm on mw161 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2' RECOVERY - ping on graylog161 is OK: PING OK - Packet loss = 0%, RTA = 2.73 ms
[06:36:45] <icinga-miraheze>	 RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 5%
[06:36:47] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 5.23, 2.49, 0.96
[06:36:50] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[06:36:52] <icinga-miraheze>	 RECOVERY - swiftobject161 Swift Object Service on swiftobject161 is OK: TCP OK - 0.005 second response time on 10.0.16.134 port 6000
[06:36:53] <icinga-miraheze>	 RECOVERY - swiftobject161 ferm_active on swiftobject161 is OK: OK ferm input default policy is set
[06:36:55] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy
[06:36:57] <icinga-miraheze>	 RECOVERY - mw161 NTP time on mw161 is OK: NTP OK: Offset -0.003739982843 secs
[06:36:58] <icinga-miraheze>	 RECOVERY - db161 ferm_active on db161 is OK: OK ferm input default policy is set
[06:36:58] <icinga-miraheze>	 RECOVERY - ping on swiftobject161 is OK: PING OK - Packet loss = 0%, RTA = 4.74 ms
[06:36:58] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.018 second response time
[06:37:01] <icinga-miraheze>	 RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 5%
[06:37:02] <icinga-miraheze>	 RECOVERY - mw162 PowerDNS Recursor on mw162 is OK: DNS OK: 0.033 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:37:06] <icinga-miraheze>	 RECOVERY - swiftobject161 NTP time on swiftobject161 is OK: NTP OK: Offset -0.0007122457027 secs
[06:37:06] <icinga-miraheze>	 PROBLEM - mon181 Current Load on mon181 is CRITICAL: LOAD CRITICAL - total load average: 15.61, 7.59, 3.90
[06:37:07] <icinga-miraheze>	 RECOVERY - db161 SSH on db161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:37:12] <icinga-miraheze>	 RECOVERY - ping on mw161 is OK: PING OK - Packet loss = 0%, RTA = 4.58 ms
[06:37:13] <icinga-miraheze>	 RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 3.6%Current connections: 18
[06:37:13] <icinga-miraheze>	 RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:37:13] <icinga-miraheze>	 RECOVERY - swiftobject161 PowerDNS Recursor on swiftobject161 is OK: DNS OK: 0.040 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:37:13] <icinga-miraheze>	 RECOVERY - db161 NTP time on db161 is OK: NTP OK: Offset -0.0005651712418 secs
[06:37:17] <icinga-miraheze>	 RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.265 second response time
[06:37:18] <icinga-miraheze>	 RECOVERY - swiftobject161 SSH on swiftobject161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:37:18] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[06:37:18] <icinga-miraheze>	 RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:37:22] <icinga-miraheze>	 RECOVERY - mem161 SSH on mem161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:37:27] <icinga-miraheze>	 RECOVERY - mem161 memcached on mem161 is OK: TCP OK - 0.000 second response time on 10.0.16.131 port 11211
[06:37:28] <icinga-miraheze>	 RECOVERY - swiftproxy161 PowerDNS Recursor on swiftproxy161 is OK: DNS OK: 0.054 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:37:37] <icinga-miraheze>	 RECOVERY - graylog161 HTTPS on graylog161 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 0.073 second response time
[06:37:38] <icinga-miraheze>	 RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 0.033 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[06:37:42] <icinga-miraheze>	 RECOVERY - swiftobject161 Disk Space on swiftobject161 is OK: DISK OK - free space: / 468306MiB (35% inode=88%);
[06:37:43] <icinga-miraheze>	 RECOVERY - mw162 NTP time on mw162 is OK: NTP OK: Offset -0.00828436017 secs
[06:38:00] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[06:38:20] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[06:38:32] <icinga-miraheze>	 RECOVERY - graylog161 Current Load on graylog161 is OK: LOAD OK - total load average: 3.27, 3.28, 1.45
[06:39:06] <icinga-miraheze>	 RECOVERY - mon181 Current Load on mon181 is OK: LOAD OK - total load average: 2.96, 5.65, 3.65
[06:39:39] <icinga-miraheze>	 RECOVERY - isocasg.org - LetsEncrypt on sslhost is OK: OK - Certificate 'isocasg.org' will expire on Sun 21 Jul 2024 03:43:22 PM GMT +0000.
[06:39:39] <icinga-miraheze>	 RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 16 Jul 2024 12:27:30 AM GMT +0000.
[06:41:10] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[06:42:07] <icinga-miraheze>	 PROBLEM - eventgate181 NTP time on eventgate181 is CRITICAL: NTP CRITICAL: Offset 0.508680433 secs
[06:46:12] <icinga-miraheze>	 RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[06:47:53] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.28 ms
[06:48:02] <icinga-miraheze>	 RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:49:28] <icinga-miraheze>	 PROBLEM - graylog161 SSH on graylog161 is CRITICAL: connect to address 10.0.16.130 and port 22: No route to host
[06:49:31] <icinga-miraheze>	 PROBLEM - graylog161 HTTPS on graylog161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to graylog161.wikitide.net port 443 after 819 ms: Couldn't connect to server
[06:49:31] <icinga-miraheze>	 PROBLEM - cp36 Puppet on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[06:49:34] <icinga-miraheze>	 PROBLEM - bast161 Puppet on bast161 is CRITICAL: connect to address 10.0.16.127 port 5666: No route to hostconnect to host 10.0.16.127 port 5666: No route to host
[06:49:37] <icinga-miraheze>	 PROBLEM - bast161 Current Load on bast161 is CRITICAL: connect to address 10.0.16.127 port 5666: No route to hostconnect to host 10.0.16.127 port 5666: No route to host
[06:49:41] <icinga-miraheze>	 PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[06:49:44] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTP on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable
[06:49:44] <icinga-miraheze>	 PROBLEM - graylog161 PowerDNS Recursor on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: No route to hostconnect to host 10.0.16.130 port 5666: No route to host
[06:49:45] <icinga-miraheze>	 PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket
[06:49:47] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.012 second response time
[06:49:48] <icinga-miraheze>	 PROBLEM - cp27 HTTPS on cp27 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[06:49:48] <icinga-miraheze>	 PROBLEM - swiftproxy161 Swift Proxy on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 and port 80: No route to host
[06:49:55] <icinga-miraheze>	 PROBLEM - ping on os162 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.141)
[06:49:56] <icinga-miraheze>	 PROBLEM - ping on mem161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.131)
[06:49:57] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable
[06:49:57] <icinga-miraheze>	 PROBLEM - ping on os161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.129)
[06:49:58] <icinga-miraheze>	 PROBLEM - ping on mw162 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.133)
[06:49:58] <icinga-miraheze>	 PROBLEM - mw162 php-fpm on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[06:49:58] <icinga-miraheze>	 PROBLEM - Host mw162 is DOWN: CRITICAL - Host Unreachable (10.0.16.133)
[06:49:58] <icinga-miraheze>	 PROBLEM - mw162 Disk Space on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[06:49:59] <icinga-miraheze>	 PROBLEM - Host mem161 is DOWN: CRITICAL - Host Unreachable (10.0.16.131)
[06:50:01] <icinga-miraheze>	 PROBLEM - Host swiftproxy161 is DOWN: CRITICAL - Host Unreachable (10.0.16.135)
[06:50:03] <icinga-miraheze>	 PROBLEM - Host swiftobject161 is DOWN: CRITICAL - Host Unreachable (10.0.16.134)
[06:50:05] <icinga-miraheze>	 PROBLEM - os162 SSH on os162 is CRITICAL: connect to address 10.0.16.141 and port 22: No route to host
[06:50:06] <icinga-miraheze>	 PROBLEM - os162 conntrack_table_size on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[06:50:06] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for matomo151 on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[06:50:06] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for swiftproxy161 on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[06:50:06] <icinga-miraheze>	 PROBLEM - cp36 PowerDNS Recursor on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[06:50:07] <icinga-miraheze>	 PROBLEM - mw161 Disk Space on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[06:50:07] <icinga-miraheze>	 PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw161.wikitide.net port 443 after 2865 ms: Couldn't connect to server
[06:50:07] <icinga-miraheze>	 PROBLEM - mw161 php-fpm on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[06:50:07] <icinga-miraheze>	 PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:50:08] <icinga-miraheze>	 PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[06:50:08] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:50:09] <icinga-miraheze>	 PROBLEM - Host cp36 is DOWN: CRITICAL - Host Unreachable (10.0.16.137)
[06:50:10] <icinga-miraheze>	 PROBLEM - mw161 PowerDNS Recursor on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[06:50:11] <icinga-miraheze>	 PROBLEM - bast161 ferm_active on bast161 is CRITICAL: connect to address 10.0.16.127 port 5666: No route to hostconnect to host 10.0.16.127 port 5666: No route to host
[06:50:12] <icinga-miraheze>	 PROBLEM - Host db161 is DOWN: CRITICAL - Host Unreachable (10.0.16.128)
[06:50:26] <icinga-miraheze>	 PROBLEM - os161 ferm_active on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host
[06:50:26] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki
[06:50:26] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki
[06:50:27] <icinga-miraheze>	 PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[06:50:28] <icinga-miraheze>	 PROBLEM - ping on graylog161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.130)
[06:50:28] <icinga-miraheze>	 PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:50:29] <icinga-miraheze>	 PROBLEM - os161 Disk Space on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host
[06:50:29] <icinga-miraheze>	 PROBLEM - os162 Puppet on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[06:50:29] <icinga-miraheze>	 PROBLEM - mw161 conntrack_table_size on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[06:50:30] <icinga-miraheze>	 PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.013 second response time
[06:50:32] <icinga-miraheze>	 PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[06:50:33] <icinga-miraheze>	 PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[06:50:34] <icinga-miraheze>	 PROBLEM - bast161 PowerDNS Recursor on bast161 is CRITICAL: connect to address 10.0.16.127 port 5666: No route to hostconnect to host 10.0.16.127 port 5666: No route to host
[06:50:35] <icinga-miraheze>	 PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[06:50:38] <icinga-miraheze>	 PROBLEM - os161 NTP time on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host
[06:50:43] <icinga-miraheze>	 PROBLEM - os162 NTP time on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[06:50:43] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[06:50:44] <icinga-miraheze>	 PROBLEM - graylog161 Puppet on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: No route to hostconnect to host 10.0.16.130 port 5666: No route to host
[06:50:47] <icinga-miraheze>	 PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: connect to address 10.0.16.132 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket
[06:50:48] <icinga-miraheze>	 PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:50:48] <icinga-miraheze>	 PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[06:50:50] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki
[06:50:51] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki
[06:50:51] <icinga-miraheze>	 PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.011 second response time
[06:50:54] <icinga-miraheze>	 PROBLEM - Host os161 is DOWN: CRITICAL - Host Unreachable (10.0.16.129)
[06:50:55] <icinga-miraheze>	 PROBLEM - os162 ferm_active on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[06:50:57] <icinga-miraheze>	 PROBLEM - Host bast161 is DOWN: CRITICAL - Host Unreachable (10.0.16.127)
[06:50:57] <icinga-miraheze>	 PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CRITICAL - NGINX Error Rate is 67%
[06:50:59] <icinga-miraheze>	 PROBLEM - cloud16 SSH on cloud16 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:51:00] <icinga-miraheze>	 PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.012 second response time
[06:51:01] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb
[06:51:01] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb
[06:51:04] <icinga-miraheze>	 PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[06:51:07] <icinga-miraheze>	 PROBLEM - graylog161 ferm_active on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: No route to hostconnect to host 10.0.16.130 port 5666: No route to host
[06:51:07] <icinga-miraheze>	 PROBLEM - os162 PowerDNS Recursor on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[06:51:09] <icinga-miraheze>	 PROBLEM - ping on mw161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.132)
[06:51:09] <icinga-miraheze>	 PROBLEM - mw161 NTP time on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[06:51:09] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 100%
[06:51:17] <icinga-miraheze>	 PROBLEM - graylog161 Disk Space on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: No route to hostconnect to host 10.0.16.130 port 5666: No route to host
[06:51:17] <icinga-miraheze>	 PROBLEM - os162 Disk Space on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[06:51:23] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 7959 bytes in 3.188 second response time
[06:51:25] <icinga-miraheze>	 PROBLEM - mw161 SSH on mw161 is CRITICAL: connect to address 10.0.16.132 and port 22: No route to host
[06:51:25] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[06:51:35] <icinga-miraheze>	 PROBLEM - Host graylog161 is DOWN: CRITICAL - Host Unreachable (10.0.16.130)
[06:51:43] <icinga-miraheze>	 PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 63%
[06:51:44] <icinga-miraheze>	 PROBLEM - cloud16 Current Load on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:51:54] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 200.47 ms
[06:51:54] <icinga-miraheze>	 PROBLEM - Host mw161 is DOWN: CRITICAL - Host Unreachable (10.0.16.132)
[06:51:56] <icinga-miraheze>	 PROBLEM - Host os162 is DOWN: CRITICAL - Host Unreachable (10.0.16.141)
[06:52:03] <icinga-miraheze>	 PROBLEM - cloud16 PowerDNS Recursor on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:52:05] <icinga-miraheze>	 PROBLEM - cloud16 NTP time on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:52:07] <icinga-miraheze>	 PROBLEM - cloud16 ferm_active on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:52:49] <icinga-miraheze>	 PROBLEM - Host cloud16 is DOWN: PING CRITICAL - Packet loss = 100%
[06:54:51] <icinga-miraheze>	 RECOVERY - Host cloud16 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms
[06:54:57] <icinga-miraheze>	 RECOVERY - Host bast161 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms
[06:54:58] <icinga-miraheze>	 RECOVERY - Host os161 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms
[06:55:01] <icinga-miraheze>	 RECOVERY - cloud16 SSH on cloud16 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[06:55:03] <icinga-miraheze>	 PROBLEM - bast161 NTP time on bast161 is WARNING: NTP WARNING: Offset 0.3746124208 secs
[06:55:12] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.21 ms
[06:55:19] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.414 second response time
[06:55:26] <icinga-miraheze>	 PROBLEM - phorge171 NTP time on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host
[06:55:32] <icinga-miraheze>	 PROBLEM - mwtask171 PowerDNS Recursor on mwtask171 is CRITICAL: connect to address 10.0.17.144 port 5666: No route to hostconnect to host 10.0.17.144 port 5666: No route to host
[06:55:33] <icinga-miraheze>	 PROBLEM - mw172 conntrack_table_size on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host
[06:55:34] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for phorge171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[06:55:35] <icinga-miraheze>	 RECOVERY - Host graylog161 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms
[06:55:38] <icinga-miraheze>	 RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures
[06:55:38] <icinga-miraheze>	 PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: No response from DNS 10.0.17.136
[06:55:39] <icinga-miraheze>	 PROBLEM - swiftobject171 SSH on swiftobject171 is CRITICAL: connect to address 10.0.17.126 and port 22: No route to host
[06:55:39] <icinga-miraheze>	 RECOVERY - bast161 Current Load on bast161 is OK: LOAD OK - total load average: 0.59, 0.25, 0.09
[06:55:42] <icinga-miraheze>	 PROBLEM - bots171 SSH on bots171 is CRITICAL: connect to address 10.0.17.143 and port 22: No route to host
[06:55:44] <icinga-miraheze>	 PROBLEM - mwtask171 mathoid on mwtask171 is CRITICAL: connect to address 10.0.17.144 and port 10044: No route to host
[06:55:47] <icinga-miraheze>	 PROBLEM - swiftobject171 NTP time on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host
[06:55:49] <icinga-miraheze>	 RECOVERY - graylog161 PowerDNS Recursor on graylog161 is OK: DNS OK: 0.344 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[06:55:49] <icinga-miraheze>	 PROBLEM - phorge171 PowerDNS Recursor on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host
[06:55:50] <icinga-miraheze>	 PROBLEM - Host swiftobject171 is DOWN: CRITICAL - Host Unreachable (10.0.17.126)
[06:55:50] <icinga-miraheze>	 PROBLEM - swiftobject171 Swift Object Service on swiftobject171 is CRITICAL: connect to address 10.0.17.126 and port 6000: No route to host
[06:55:53] <icinga-miraheze>	 RECOVERY - cloud16 Current Load on cloud16 is OK: LOAD OK - total load average: 8.22, 2.92, 1.06
[06:55:55] <icinga-miraheze>	 RECOVERY - Host mw161 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms
[06:55:57] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for puppet181 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[06:55:57] <icinga-miraheze>	 PROBLEM - reports171 php-fpm on reports171 is CRITICAL: connect to address 10.0.17.139 port 5666: No route to hostconnect to host 10.0.17.139 port 5666: No route to host
[06:55:57] <icinga-miraheze>	 PROBLEM - reports171 conntrack_table_size on reports171 is CRITICAL: connect to address 10.0.17.139 port 5666: No route to hostconnect to host 10.0.17.139 port 5666: No route to host
[06:55:59] <icinga-miraheze>	 RECOVERY - Host os162 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms
[06:56:00] <icinga-miraheze>	 PROBLEM - jobchron171 Redis Process on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[06:56:00] <icinga-miraheze>	 PROBLEM - ping on swiftac171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.125)
[06:56:00] <icinga-miraheze>	 PROBLEM - cp37 Puppet on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[06:56:00] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[06:56:01] <icinga-miraheze>	 PROBLEM - Host mw171 is DOWN: CRITICAL - Host Unreachable (10.0.17.122)
[06:56:01] <icinga-miraheze>	 RECOVERY - Host mem161 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms
[06:56:02] <icinga-miraheze>	 RECOVERY - Host swiftobject161 is UP: PING OK - Packet loss = 0%, RTA = 1.33 ms
[06:56:02] <icinga-miraheze>	 PROBLEM - db171 ferm_active on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host
[06:56:02] <icinga-miraheze>	 PROBLEM - mwtask171 ferm_active on mwtask171 is CRITICAL: connect to address 10.0.17.144 port 5666: No route to hostconnect to host 10.0.17.144 port 5666: No route to host
[06:56:02] <icinga-miraheze>	 PROBLEM - mwtask171 NTP time on mwtask171 is CRITICAL: connect to address 10.0.17.144 port 5666: No route to hostconnect to host 10.0.17.144 port 5666: No route to host
[06:56:03] <icinga-miraheze>	 RECOVERY - mw161 php-fpm on mw161 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2'
[06:56:03] <icinga-miraheze>	 PROBLEM - jobchron171 Current Load on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[06:57:40] <icinga-miraheze>	 PROBLEM - Host cloud17 is DOWN: PING CRITICAL - Packet loss = 100%
[06:57:45] <icinga-miraheze>	 PROBLEM - Host phorge171 is DOWN: CRITICAL - Host Unreachable (10.0.17.124)
[06:57:53] <icinga-miraheze>	 RECOVERY - swiftproxy161 Swift Proxy on swiftproxy161 is OK: TCP OK - 0.000 second response time on 10.0.16.135 port 80
[06:57:58] <icinga-miraheze>	 RECOVERY - mw162 php-fpm on mw162 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2'
[06:57:58] <icinga-miraheze>	 RECOVERY - mw162 Disk Space on mw162 is OK: DISK OK - free space: / 27483MiB (51% inode=86%);
[06:58:03] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for mw181 on cp36 is CRITICAL: connect to address localhost and port 8119: Connection refused
[06:58:03] <icinga-miraheze>	 RECOVERY - ping on mw162 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms
[06:58:03] <icinga-miraheze>	 RECOVERY - ping on os162 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms
[06:58:08] <icinga-miraheze>	 PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp36.wikitide.net port 443 after 0 ms: Couldn't connect to server
[07:06:32] <icinga-miraheze>	 RECOVERY - Host swiftac171 is UP: PING OK - Packet loss = 0%, RTA = 5.05 ms
[07:06:33] <icinga-miraheze>	 RECOVERY - Host ns1 is UP: PING OK - Packet loss = 0%, RTA = 1.44 ms
[07:06:38] <icinga-miraheze>	 RECOVERY - ldap171 PowerDNS Recursor on ldap171 is OK: DNS OK: 0.054 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[07:06:40] <icinga-miraheze>	 RECOVERY - ns1 Current Load on ns1 is OK: LOAD OK - total load average: 0.01, 0.09, 0.07
[07:06:41] <icinga-miraheze>	 RECOVERY - ldap171 Current Load on ldap171 is OK: LOAD OK - total load average: 0.02, 0.13, 0.08
[07:06:41] <icinga-miraheze>	 RECOVERY - ldap171 NTP time on ldap171 is OK: NTP OK: Offset -0.003489643335 secs
[07:06:46] <icinga-miraheze>	 PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:07:06] <icinga-miraheze>	 PROBLEM - swiftac171 Puppet on swiftac171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:07:07] <icinga-miraheze>	 RECOVERY - Host cp37 is UP: PING OK - Packet loss = 0%, RTA = 2.48 ms
[07:07:15] <icinga-miraheze>	 RECOVERY - Host cloud17 is UP: PING OK - Packet loss = 0%, RTA = 5.45 ms
[07:07:17] <icinga-miraheze>	 RECOVERY - Host reports171 is UP: PING OK - Packet loss = 0%, RTA = 2.22 ms
[07:07:18] <icinga-miraheze>	 RECOVERY - ping on swiftac171 is OK: PING OK - Packet loss = 0%, RTA = 1.89 ms
[07:07:21] <icinga-miraheze>	 RECOVERY - ldap171 ferm_active on ldap171 is OK: OK ferm input default policy is set
[07:07:24] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures
[07:07:26] <icinga-miraheze>	 RECOVERY - Host phorge171 is UP: PING OK - Packet loss = 0%, RTA = 2.96 ms
[07:07:26] <icinga-miraheze>	 RECOVERY - Host bots171 is UP: PING OK - Packet loss = 0%, RTA = 2.96 ms
[07:07:26] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw151 on cp37 is CRITICAL: connect to address localhost and port 8113: Connection refused
[07:07:46] <icinga-miraheze>	 RECOVERY - Host swiftobject171 is UP: PING OK - Packet loss = 0%, RTA = 2.25 ms
[07:07:50] <icinga-miraheze>	 RECOVERY - mem161 NTP time on mem161 is OK: NTP OK: Offset -0.001510262489 secs
[07:07:50] <icinga-miraheze>	 RECOVERY - reports171 NTP time on reports171 is OK: NTP OK: Offset -0.0007377266884 secs
[07:07:51] <icinga-miraheze>	 RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.195 seconds response time. wikitide.net returns 109.123.230.163,2400:d320:2161:9775::1
[07:07:51] <icinga-miraheze>	 RECOVERY - phorge171 ferm_active on phorge171 is OK: OK ferm input default policy is set
[07:07:53] <icinga-miraheze>	 RECOVERY - Host db171 is UP: PING OK - Packet loss = 0%, RTA = 6.20 ms
[07:07:56] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for matomo151 on cp37 is CRITICAL: connect to address localhost and port 8203: Connection refused
[07:08:00] <icinga-miraheze>	 [Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:08:01] <icinga-miraheze>	 PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:08:11] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw152 on cp37 is CRITICAL: connect to address localhost and port 8114: Connection refused
[07:08:12] <icinga-miraheze>	 RECOVERY - phorge171 Backups Phorge Static on phorge171 is OK: FILE_AGE OK: /var/log/phorge-backup.log is 1139297 seconds old and 91 bytes
[07:08:13] <icinga-miraheze>	 RECOVERY - ping on phorge171 is OK: PING OK - Packet loss = 0%, RTA = 3.77 ms
[07:08:15] <icinga-miraheze>	 RECOVERY - Host mw171 is UP: PING OK - Packet loss = 0%, RTA = 4.66 ms
[07:08:16] <icinga-miraheze>	 RECOVERY - Host swiftproxy171 is UP: PING OK - Packet loss = 0%, RTA = 1.11 ms
[07:08:23] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.212 second response time
[07:08:25] <icinga-miraheze>	 RECOVERY - ping on ns1 is OK: PING OK - Packet loss = 0%, RTA = 5.38 ms
[07:08:27] <icinga-miraheze>	 RECOVERY - swiftproxy171 conntrack_table_size on swiftproxy171 is OK: OK: nf_conntrack is 1 % full
[07:08:30] <icinga-miraheze>	 RECOVERY - swiftobject171 Swift Object Service on swiftobject171 is OK: TCP OK - 0.025 second response time on 10.0.17.126 port 6000
[07:08:32] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.118 second response time
[07:08:35] <icinga-miraheze>	 RECOVERY - swiftobject171 SSH on swiftobject171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[07:08:39] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.17, 0.14, 0.05
[07:08:39] <icinga-miraheze>	 RECOVERY - db171 NTP time on db171 is OK: NTP OK: Offset -0.01229390502 secs
[07:08:41] <icinga-miraheze>	 RECOVERY - db171 ferm_active on db171 is OK: OK ferm input default policy is set
[07:08:41] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mwtask181 on cp37 is CRITICAL: connect to address localhost and port 8160: Connection refused
[07:08:43] <icinga-miraheze>	 RECOVERY - swiftproxy171 PowerDNS Recursor on swiftproxy171 is OK: DNS OK: 0.053 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[07:08:51] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw171 on cp37 is CRITICAL: connect to address localhost and port 8117: Connection refused
[07:08:52] <icinga-miraheze>	 RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 5713MiB (64% inode=86%);
[07:08:52] <icinga-miraheze>	 RECOVERY - db171 Disk Space on db171 is OK: DISK OK - free space: / 660741MiB (74% inode=98%);
[07:08:52] <icinga-miraheze>	 RECOVERY - phorge171 SSH on phorge171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[07:08:52] <icinga-miraheze>	 RECOVERY - phorge171 PowerDNS Recursor on phorge171 is OK: DNS OK: 0.052 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[07:08:53] <icinga-miraheze>	 RECOVERY - Host jobchron171 is UP: PING OK - Packet loss = 0%, RTA = 4.63 ms
[07:08:56] <icinga-miraheze>	 RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17717 bytes in 0.212 second response time
[07:08:56] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for reports171 on cp37 is CRITICAL: connect to address localhost and port 8205: Connection refused
[07:08:56] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw182 on cp37 is CRITICAL: connect to address localhost and port 8120: Connection refused
[07:08:57] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.044 second response time
[07:09:01] <icinga-miraheze>	 RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.139 second response time
[07:09:01] <icinga-miraheze>	 PROBLEM - cp37 NTP time on cp37 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:09:01] <icinga-miraheze>	 RECOVERY - Host mw172 is UP: PING OK - Packet loss = 0%, RTA = 3.92 ms
[07:09:01] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTP on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.076 second response time
[07:09:02] <icinga-miraheze>	 RECOVERY - reports171 php-fpm on reports171 is OK: PROCS OK: 13 processes with command name 'php-fpm8.2'
[07:09:06] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mwtask171 on cp37 is CRITICAL: connect to address localhost and port 8161: Connection refused
[07:09:06] <icinga-miraheze>	 PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. PROBLEM - cp37 Nginx Backend for mw181 on cp37 is CRITICAL: connect to address localhost and port 8119: Connection refused
[07:09:06] <icinga-miraheze>	 RECOVERY - ns1 conntrack_table_size on ns1 is OK: OK: nf_conntrack is 0 % full
[07:09:10] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.403 second response time
[07:09:11] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 7.71 ms
[07:09:14] <icinga-miraheze>	 PROBLEM - eventgate181 NTP time on eventgate181 is WARNING: NTP WARNING: Offset 0.4259321094 secs
[07:09:15] <icinga-miraheze>	 RECOVERY - mw172 Disk Space on mw172 is OK: DISK OK - free space: / 30656MiB (57% inode=86%);
[07:09:16] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 22592MiB (25% inode=98%);
[07:09:17] <icinga-miraheze>	 RECOVERY - reports171 Current Load on reports171 is OK: LOAD OK - total load average: 0.02, 0.08, 0.03
[07:09:19] <icinga-miraheze>	 RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2'
[07:09:21] <icinga-miraheze>	 RECOVERY - Host mwtask171 is UP: PING OK - Packet loss = 0%, RTA = 2.39 ms
[07:09:22] <icinga-miraheze>	 RECOVERY - ldap171 LDAP on ldap171 is OK: LDAP OK - 0.048 seconds response time
[07:09:26] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for test151 on cp37 is CRITICAL: connect to address localhost and port 8181: Connection refused
[07:09:26] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mon181 on cp37 is CRITICAL: connect to address localhost and port 8201: Connection refused
[07:09:27] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.165 second response time
[07:09:29] <icinga-miraheze>	 RECOVERY - jobchron171 Disk Space on jobchron171 is OK: DISK OK - free space: / 5757MiB (65% inode=86%);
[07:09:32] <icinga-miraheze>	 RECOVERY - cp27 HTTPS on cp27 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3782 bytes in 0.917 second response time
[07:09:32] <icinga-miraheze>	 RECOVERY - phorge171 Disk Space on phorge171 is OK: DISK OK - free space: / 25277MiB (56% inode=93%);
[07:09:35] <icinga-miraheze>	 RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.498 second response time
[07:10:00] <icinga-miraheze>	 RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.429 second response time
[07:10:01] <icinga-miraheze>	 RECOVERY - jobchron171 JobChron Service on jobchron171 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[07:10:01] <icinga-miraheze>	 RECOVERY - reports171 Disk Space on reports171 is OK: DISK OK - free space: / 13638MiB (75% inode=92%);
[07:10:01] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for swiftproxy161 on cp37 is CRITICAL: connect to address localhost and port 8206: Connection refused
[07:10:03] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.409 second response time
[07:10:05] <icinga-miraheze>	 RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.358 second response time
[07:10:05] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[07:10:06] <icinga-miraheze>	 PROBLEM - cp37 PowerDNS Recursor on cp37 is CRITICAL: Domain 'wikitide.net' was not found by the server
[07:10:06] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 0.07, 0.13, 0.08
[07:10:06] <icinga-miraheze>	 RECOVERY - cloud17 Current Load on cloud17 is OK: LOAD OK - total load average: 9.37, 6.84, 3.76
[07:10:06] <icinga-miraheze>	 RECOVERY - ns1 ferm_active on ns1 is OK: OK ferm input default policy is set
[07:10:06] <icinga-miraheze>	 RECOVERY - phorge171 conntrack_table_size on phorge171 is OK: OK: nf_conntrack is 0 % full
[07:10:07] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for swiftproxy171 on cp37 is CRITICAL: connect to address localhost and port 8207: Connection refused
[07:10:07] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw161 on cp37 is CRITICAL: connect to address localhost and port 8115: Connection refused
[07:10:08] <icinga-miraheze>	 RECOVERY - mw172 ferm_active on mw172 is OK: OK ferm input default policy is set
[07:10:08] <icinga-miraheze>	 RECOVERY - ping on reports171 is OK: PING OK - Packet loss = 0%, RTA = 4.78 ms
[07:10:09] <icinga-miraheze>	 RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.089 second response time
[07:10:09] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[07:10:10] <icinga-miraheze>	 RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3784 bytes in 0.975 second response time
[07:11:32] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:11:33] <icinga-miraheze>	 RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:12:16] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:12:40] <icinga-miraheze>	 PROBLEM - isocasg.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:12:42] <icinga-miraheze>	 PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:13:00] <icinga-miraheze>	 [Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:14:01] <icinga-miraheze>	 RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[07:14:18] <icinga-miraheze>	 RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:15:13] <icinga-miraheze>	 RECOVERY - mon181 Current Load on mon181 is OK: LOAD OK - total load average: 0.60, 6.74, 5.44
[07:16:03] <icinga-miraheze>	 RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[07:18:15] <icinga-miraheze>	 RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:22:07] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[07:23:42] <icinga-miraheze>	 PROBLEM - eventgate181 NTP time on eventgate181 is CRITICAL: NTP CRITICAL: Offset 0.5064213574 secs
[07:24:29] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[07:25:56] <icinga-miraheze>	 RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:26:13] <icinga-miraheze>	 RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[07:26:35] <icinga-miraheze>	 PROBLEM - kafka181 NTP time on kafka181 is CRITICAL: NTP CRITICAL: Offset 0.5089354813 secs
[07:28:04] <icinga-miraheze>	 RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[07:28:13] <icinga-miraheze>	 RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[07:28:56] <icinga-miraheze>	 RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[07:31:01] <icinga-miraheze>	 RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[07:31:03] <icinga-miraheze>	 RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:31:13] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:31:35] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:32:34] <icinga-miraheze>	 RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[07:36:04] <icinga-miraheze>	 RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[07:36:51] <icinga-miraheze>	 RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:48:45] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 174.03 ms
[07:50:47] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 176.23 ms
[08:05:06] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 207.61 ms
[08:07:04] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 214.58 ms
[08:17:11] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 190.82 ms
[08:21:09] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 188.68 ms
[08:36:19] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 185.41 ms
[08:42:17] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 182.33 ms
[08:45:52] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 144.14 ms
[08:47:50] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 143.91 ms
[09:03:25] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.85 ms
[09:05:23] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 189.24 ms
[09:09:29] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 196.99 ms
[09:13:27] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 189.03 ms
[09:17:33] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 217.97 ms
[09:19:26] <RhinosF1>	 !log block a UA & IP
[09:19:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:19:32] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 218.14 ms
[09:26:14] <icinga-miraheze>	 PROBLEM - db171 Backups SQL on db171 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864226 seconds old and 122126 bytes
[09:34:32] <botifico>	 [02mw-config] 07RhinosF1 opened pull request 03#5545: Sitenotice: remove global notice - 13https://github.com/miraheze/mw-config/pull/5545
[09:35:21] <miraheze-github>	 miraheze/mw-config - RhinosF1 the build passed.
[09:35:40] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5545: Sitenotice: remove global notice - 13https://github.com/miraheze/mw-config/pull/5545
[09:35:43] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6488764550c7...ad768c379757
[09:35:46] <botifico>	 [02miraheze/mw-config] 07RhinosF1 03ad768c3 - Sitenotice: remove global notice (#5545)
[09:36:37] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[09:37:23] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[09:37:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:37:46] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 23s
[09:37:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:39:52] <icinga-miraheze>	 PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonciclopedia.org' expires in 15 day(s) (Tue 14 May 2024 09:22:18 AM GMT +0000).
[09:40:05] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/42453b9385ee...f874b589b2e7
[09:40:07] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03f874b58 - Bot: Update SSL cert for nonciclopedia.org
[09:40:24] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'enc.for.uz' expires in 15 day(s) (Tue 14 May 2024 09:31:34 AM GMT +0000).
[09:40:47] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f874b589b2e7...af267d74e7e4
[09:40:48] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03af267d7 - Bot: Update SSL cert for enc.for.uz
[09:54:13] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[09:54:14] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[09:54:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:54:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:00:36] <icinga-miraheze>	 PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pwiki.arkcls.com' expires in 15 day(s) (Tue 14 May 2024 09:47:28 AM GMT +0000).
[10:01:16] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/af267d74e7e4...0c6ca664569c
[10:01:18] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 030c6ca66 - Bot: Update SSL cert for pwiki.arkcls.com
[10:02:48] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[10:02:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:03:14] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 25s
[10:03:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:04:19] <icinga-miraheze>	 PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.graalmilitary.com' expires in 15 day(s) (Tue 14 May 2024 09:41:04 AM GMT +0000).
[10:04:41] <icinga-miraheze>	 PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'runzeppelin.ru' expires in 15 day(s) (Tue 14 May 2024 10:02:34 AM GMT +0000).
[10:05:09] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0c6ca664569c...5f124c79ffeb
[10:05:12] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 035f124c7 - Bot: Update SSL cert for runzeppelin.ru
[10:08:08] <icinga-miraheze>	 PROBLEM - wiki.mikrodev.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mikrodev.com' expires in 15 day(s) (Tue 14 May 2024 09:37:37 AM GMT +0000).
[10:09:44] <icinga-miraheze>	 RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Sat 27 Jul 2024 08:39:58 AM GMT +0000.
[10:09:47] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Sat 27 Jul 2024 08:40:42 AM GMT +0000.
[10:10:39] <icinga-miraheze>	 PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[10:28:01] <icinga-miraheze>	 PROBLEM - wiki.starship.digital - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.starship.digital' expires in 15 day(s) (Tue 14 May 2024 10:16:17 AM GMT +0000).
[10:28:16] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5f124c79ffeb...9fda1ad2b0ea
[10:28:17] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 039fda1ad - Bot: Update SSL cert for wiki.starship.digital
[10:33:03] <icinga-miraheze>	 PROBLEM - wiki.scvo.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.scvo.org' expires in 15 day(s) (Tue 14 May 2024 10:07:34 AM GMT +0000).
[10:33:22] <icinga-miraheze>	 PROBLEM - stablestate.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'stablestate.org' expires in 15 day(s) (Tue 14 May 2024 10:08:18 AM GMT +0000).
[10:33:35] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 186.16 ms
[10:33:56] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9fda1ad2b0ea...a6d8c914ba88
[10:33:59] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03a6d8c91 - Bot: Update SSL cert for stablestate.org
[10:35:10] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[10:36:22] <icinga-miraheze>	 PROBLEM - www.mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Tue 14 May 2024 10:25:39 AM GMT +0000).
[10:39:53] <icinga-miraheze>	 PROBLEM - mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Tue 14 May 2024 10:25:39 AM GMT +0000).
[10:40:11] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a6d8c914ba88...430e37b71305
[10:40:14] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03430e37b - Bot: Update SSL cert for mh142.com
[10:45:34] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 234.28 ms
[11:01:47] <icinga-miraheze>	 PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Tue 14 May 2024 10:37:44 AM GMT +0000).
[11:01:49] <icinga-miraheze>	 PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.s23.org' expires in 15 day(s) (Tue 14 May 2024 10:40:04 AM GMT +0000).
[11:02:02] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/430e37b71305...8b54b9baab34
[11:02:05] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 038b54b9b - Bot: Update SSL cert for dariawiki.org
[11:02:13] <icinga-miraheze>	 RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Sat 27 Jul 2024 09:33:50 AM GMT +0000.
[11:03:27] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[11:03:31] <icinga-miraheze>	 PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Tue 14 May 2024 10:37:44 AM GMT +0000).
[11:03:35] <icinga-miraheze>	 RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Sat 27 Jul 2024 09:05:03 AM GMT +0000.
[11:05:38] <icinga-miraheze>	 PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thunis.eu' expires in 15 day(s) (Tue 14 May 2024 10:36:45 AM GMT +0000).
[11:05:55] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8b54b9baab34...42b299ae4e5f
[11:05:56] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0342b299a - Bot: Update SSL cert for wiki.thunis.eu
[11:06:07] <icinga-miraheze>	 RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Sat 27 Jul 2024 09:40:05 AM GMT +0000.
[11:06:36] <icinga-miraheze>	 RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[11:06:38] <icinga-miraheze>	 PROBLEM - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cyberlaw.ccdcoe.org' expires in 15 day(s) (Tue 14 May 2024 10:46:26 AM GMT +0000).
[11:08:24] <icinga-miraheze>	 PROBLEM - programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Tue 14 May 2024 10:56:20 AM GMT +0000).
[11:08:42] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/42b299ae4e5f...08435a1a483d
[11:08:43] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0308435a1 - Bot: Update SSL cert for programming.red
[11:09:48] <icinga-miraheze>	 RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Sat 27 Jul 2024 09:40:05 AM GMT +0000.
[11:25:34] <icinga-miraheze>	 RECOVERY - wiki.starship.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.starship.digital' will expire on Sat 27 Jul 2024 09:28:10 AM GMT +0000.
[11:26:14] <icinga-miraheze>	 RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Sat 27 Jul 2024 09:00:44 AM GMT +0000.
[11:26:32] <icinga-miraheze>	 PROBLEM - www.programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Tue 14 May 2024 10:56:20 AM GMT +0000).
[11:28:36] <icinga-miraheze>	 PROBLEM - tep.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tep.wiki' expires in 15 day(s) (Tue 14 May 2024 11:21:41 AM GMT +0000).
[11:31:47] <icinga-miraheze>	 PROBLEM - kunwok.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kunwok.org' expires in 15 day(s) (Tue 14 May 2024 11:14:15 AM GMT +0000).
[11:32:01] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/08435a1a483d...903558f375ea
[11:32:04] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03903558f - Bot: Update SSL cert for kunwok.org
[11:33:42] <icinga-miraheze>	 PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediaofastrobiology.org' expires in 15 day(s) (Tue 14 May 2024 11:05:40 AM GMT +0000).
[11:33:45] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[11:33:53] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/903558f375ea...798268e9a032
[11:33:55] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03798268e - Bot: Update SSL cert for encyclopediaofastrobiology.org
[11:36:31] <icinga-miraheze>	 PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ao90.pinho.org' expires in 15 day(s) (Tue 14 May 2024 11:22:36 AM GMT +0000).
[11:57:25] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:00:59] <icinga-miraheze>	 PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'monarchists.wiki' expires in 15 day(s) (Tue 14 May 2024 11:38:44 AM GMT +0000).
[12:01:13] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/798268e9a032...4c919eaa8bad
[12:01:15] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 034c919ea - Bot: Update SSL cert for monarchists.wiki
[12:02:11] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[12:02:30] <icinga-miraheze>	 PROBLEM - podpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'podpedia.org' expires in 15 day(s) (Tue 14 May 2024 11:48:57 AM GMT +0000).
[12:02:45] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4c919eaa8bad...ade4f25a6fb2
[12:02:46] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03ade4f25 - Bot: Update SSL cert for podpedia.org
[12:03:13] <icinga-miraheze>	 RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Sat 27 Jul 2024 10:33:47 AM GMT +0000.
[12:03:54] <icinga-miraheze>	 RECOVERY - wiki.thunis.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Sat 27 Jul 2024 10:05:50 AM GMT +0000.
[12:07:37] <icinga-miraheze>	 RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Sat 27 Jul 2024 10:08:36 AM GMT +0000.
[12:07:37] <MirahezeRelay>	 <pixldev, replying to m3w> Erm, why? There’s no reason to make this private. Private matters are already discussed in the sre board discord
[12:09:52] <RhinosF1>	 !log ban 3 ips + challenge 1 ASN
[12:09:55] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[12:09:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:20:15] <icinga-miraheze>	 PROBLEM - replication-ops.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - replication-ops.com All nameservers failed to answer the query.
[12:22:52] <icinga-miraheze>	 PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ciptamedia.org' expires in 15 day(s) (Tue 14 May 2024 12:02:14 PM GMT +0000).
[12:23:42] <icinga-miraheze>	 RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Sat 27 Jul 2024 10:08:36 AM GMT +0000.
[12:27:59] <icinga-miraheze>	 RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Sat 27 Jul 2024 10:01:56 AM GMT +0000.
[12:30:20] <icinga-miraheze>	 RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Sat 27 Jul 2024 10:31:56 AM GMT +0000.
[12:31:03] <icinga-miraheze>	 RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Sat 27 Jul 2024 10:01:56 AM GMT +0000.
[12:33:41] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 203.45 ms
[12:33:49] <icinga-miraheze>	 PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.cineastas.pt' expires in 15 day(s) (Tue 14 May 2024 12:07:18 PM GMT +0000).
[12:34:00] <icinga-miraheze>	 PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.whentheycry.org' expires in 15 day(s) (Tue 14 May 2024 12:23:43 PM GMT +0000).
[12:34:30] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[12:34:42] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ade4f25a6fb2...c29676773c6a
[12:34:44] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03c296767 - Bot: Update SSL cert for wiki.whentheycry.org
[12:36:41] <icinga-miraheze>	 PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vitriol.spoon.army' expires in 15 day(s) (Tue 14 May 2024 12:30:56 PM GMT +0000).
[12:36:57] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c29676773c6a...daaf13ff5fdc
[12:37:00] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03daaf13f - Bot: Update SSL cert for vitriol.spoon.army
[12:37:55] <icinga-miraheze>	 PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.nonbinary.wiki' expires in 15 day(s) (Tue 14 May 2024 12:07:29 PM GMT +0000).
[12:39:39] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 217.68 ms
[12:53:57] <RhinosF1>	 !log limited a UA rule to only applying on some domains
[12:54:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:59:21] <icinga-miraheze>	 PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vise.dayid.org' expires in 15 day(s) (Tue 14 May 2024 12:56:37 PM GMT +0000).
[12:59:41] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/daaf13ff5fdc...f57d90532896
[12:59:44] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03f57d905 - Bot: Update SSL cert for vise.dayid.org
[13:00:38] <icinga-miraheze>	 PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.de' expires in 15 day(s) (Tue 14 May 2024 12:48:51 PM GMT +0000).
[13:01:22] <icinga-miraheze>	 PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nowchess.org' expires in 15 day(s) (Tue 14 May 2024 12:41:04 PM GMT +0000).
[13:02:51] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[13:02:59] <icinga-miraheze>	 RECOVERY - wiki.whentheycry.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Sat 27 Jul 2024 11:34:36 AM GMT +0000.
[13:03:40] <icinga-miraheze>	 PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.joust.ro' expires in 15 day(s) (Tue 14 May 2024 12:37:22 PM GMT +0000).
[13:04:10] <icinga-miraheze>	 PROBLEM - www.lab612.at - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.lab612.at' expires in 15 day(s) (Tue 14 May 2024 12:40:54 PM GMT +0000).
[13:04:25] <icinga-miraheze>	 PROBLEM - wiki.openhatch.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.openhatch.org' expires in 15 day(s) (Tue 14 May 2024 12:46:33 PM GMT +0000).
[13:04:30] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f57d90532896...a414aa2c91c8
[13:04:33] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03a414aa2 - Bot: Update SSL cert for www.lab612.at
[13:04:35] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 242.03 ms
[13:05:22] <icinga-miraheze>	 RECOVERY - vitriol.spoon.army - LetsEncrypt on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Sat 27 Jul 2024 11:36:52 AM GMT +0000.
[13:06:34] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 196.05 ms
[13:07:29] <icinga-miraheze>	 PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Tue 14 May 2024 12:50:53 PM GMT +0000).
[13:14:06] <RhinosF1>	 !log replace an ASN challenge with a block on some ips
[13:14:06] <wm-bot>	 Sorry, you are not authorized to perform this
[13:14:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:19:16] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 160.51 ms
[13:21:14] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 161.92 ms
[13:22:08] <RhinosF1>	 !log replace another ASN challenge with a UA block
[13:22:08] <wm-bot>	 Sorry, you are not authorized to perform this
[13:22:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:27:25] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[13:28:59] <icinga-miraheze>	 RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Sat 27 Jul 2024 11:01:07 AM GMT +0000.
[13:29:20] <icinga-miraheze>	 PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.valkyrienskies.org' expires in 15 day(s) (Tue 14 May 2024 01:03:16 PM GMT +0000).
[13:30:24] <icinga-miraheze>	 RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Sat 27 Jul 2024 11:02:39 AM GMT +0000.
[13:33:49] <icinga-miraheze>	 RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Sat 27 Jul 2024 12:04:24 PM GMT +0000.
[13:36:59] <icinga-miraheze>	 PROBLEM - bristolstudenthousingcoop.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bristolstudenthousingcoop.org' expires in 15 day(s) (Tue 14 May 2024 01:24:48 PM GMT +0000).
[13:37:13] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a414aa2c91c8...254d6bae5022
[13:37:14] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03254d6ba - Bot: Update SSL cert for bristolstudenthousingcoop.org
[13:37:47] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp51.wikitide.net - CNAME OK
[13:38:47] <icinga-miraheze>	 PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bebaskanpengetahuan.id' expires in 15 day(s) (Tue 14 May 2024 01:11:29 PM GMT +0000).
[13:39:00] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/254d6bae5022...71358c1c66f6
[13:39:03] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0371358c1 - Bot: Update SSL cert for bebaskanpengetahuan.id
[13:43:43] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 248.51 ms
[13:57:21] <icinga-miraheze>	 RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Sat 27 Jul 2024 11:59:36 AM GMT +0000.
[13:57:42] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 191.83 ms
[13:59:53] <icinga-miraheze>	 PROBLEM - www.portalsofphereon.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.portalsofphereon.com' expires in 15 day(s) (Tue 14 May 2024 01:43:04 PM GMT +0000).
[14:00:07] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/71358c1c66f6...c9b756f2a8db
[14:00:10] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03c9b756f - Bot: Update SSL cert for www.portalsofphereon.com
[14:06:41] <icinga-miraheze>	 PROBLEM - portalsofphereon.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.portalsofphereon.com' expires in 15 day(s) (Tue 14 May 2024 01:43:04 PM GMT +0000).
[14:06:46] <icinga-miraheze>	 RECOVERY - bristolstudenthousingcoop.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bristolstudenthousingcoop.org' will expire on Sat 27 Jul 2024 12:37:08 PM GMT +0000.
[14:06:55] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 177.66 ms
[14:08:40] <icinga-miraheze>	 RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Sat 27 Jul 2024 12:38:55 PM GMT +0000.
[14:08:54] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 196.19 ms
[14:10:36] <icinga-miraheze>	 PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[14:20:56] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 164.20 ms
[14:22:54] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 161.02 ms
[14:24:53] <RhinosF1>	 !log block 3 IPs
[14:24:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:28:45] <RhinosF1>	 !log mitigate an ongoing scrape
[14:28:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:31:05] <icinga-miraheze>	 PROBLEM - graphite151 NTP time on graphite151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[14:32:03] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 276.81 ms
[14:33:09] <icinga-miraheze>	 PROBLEM - graphite151 NTP time on graphite151 is CRITICAL: NTP CRITICAL: Offset 3.09016186 secs
[14:33:57] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/varnishlog_logs]
[14:35:23] <icinga-miraheze>	 RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Sat 27 Jul 2024 01:00:01 PM GMT +0000.
[14:36:02] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is WARNING: PING WARNING - Packet loss = 0%, RTA = 303.65 ms
[14:38:03] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 739.51 ms
[14:42:03] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 270.87 ms
[14:42:25] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:51:51] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 156.35 ms
[14:55:49] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 159.21 ms
[14:58:24] <icinga-miraheze>	 RECOVERY - www.portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Sat 27 Jul 2024 01:00:01 PM GMT +0000.
[15:02:22] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:05:53] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 161.81 ms
[15:06:38] <icinga-miraheze>	 RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[15:11:49] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 155.89 ms
[15:31:44] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 158.40 ms
[15:33:42] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 159.64 ms
[15:38:11] <botifico>	 [02MirahezeMagic] 07waki285 synchronize pull request 03#489: Add electionadmin - 13https://github.com/miraheze/MirahezeMagic/pull/489
[15:38:52] <botifico>	 [02MirahezeMagic] 07waki285 synchronize pull request 03#489: Add electionadmin - 13https://github.com/miraheze/MirahezeMagic/pull/489
[15:39:54] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[15:42:24] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 192.83 ms
[15:43:40] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 162.06 ms
[15:44:23] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 208.31 ms
[15:46:36] <MirahezeRelay>	 <m3w, replying to pixldev> More of better organized, if sre is busy and will look later. It gets flooded out
[15:47:37] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 154.64 ms
[15:55:31] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 184.60 ms
[15:57:29] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 187.01 ms
[16:06:36] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 190.67 ms
[16:07:25] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:08:45] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[16:12:34] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 199.33 ms
[16:29:50] <icinga-miraheze>	 PROBLEM - wiki.jill-jimmy.com - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[16:42:20] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:49:02] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/17c267dc0b90...21e5a47d6811
[16:49:05] <botifico>	 [02miraheze/puppet] 07Universal-Omega 0321e5a47 - Revert
[16:59:25] <icinga-miraheze>	 RECOVERY - wiki.jill-jimmy.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jill-jimmy.com' will expire on Mon 22 Jul 2024 10:18:43 PM GMT +0000.
[17:02:20] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:11:53] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 203.68 ms
[17:15:52] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 180.96 ms
[17:29:54] <icinga-miraheze>	 PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.villagecollaborative.net' expires in 15 day(s) (Tue 14 May 2024 05:25:35 PM GMT +0000).
[17:31:10] <icinga-miraheze>	 PROBLEM - history.sdtef.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.sdtef.org' expires in 15 day(s) (Tue 14 May 2024 05:27:07 PM GMT +0000).
[17:31:31] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c9b756f2a8db...9a23440e8a73
[17:31:33] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 039a23440 - Bot: Update SSL cert for history.sdtef.org
[17:32:13] <icinga-miraheze>	 PROBLEM - archive.stellurgists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archive.stellurgists.wiki' expires in 15 day(s) (Tue 14 May 2024 05:07:34 PM GMT +0000).
[17:37:24] <icinga-miraheze>	 PROBLEM - www.mcpk.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.mcpk.wiki' expires in 15 day(s) (Tue 14 May 2024 05:32:34 PM GMT +0000).
[17:37:49] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp51.wikitide.net - CNAME OK
[17:47:05] <botifico>	 [02dns] 07MacFan4000 opened pull request 03#515: Add wkmr.liao.media zone - 13https://github.com/miraheze/dns/pull/515
[17:54:19] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#515: Add wkmr.liao.media zone - 13https://github.com/miraheze/dns/pull/515
[17:54:21] <botifico>	 [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/8f3cf62ab63f...0fd55e2920b1
[17:54:24] <botifico>	 [02miraheze/dns] 07MacFan4000 030fd55e2 - Add wkmr.liao.media zone (#515)
[17:56:21] <icinga-miraheze>	 PROBLEM - infectowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'infectowiki.com' expires in 15 day(s) (Tue 14 May 2024 05:45:44 PM GMT +0000).
[17:56:36] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9a23440e8a73...84a58147d8b9
[17:56:38] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0384a5814 - Bot: Update SSL cert for infectowiki.com
[18:00:42] <icinga-miraheze>	 RECOVERY - history.sdtef.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.sdtef.org' will expire on Sat 27 Jul 2024 04:31:19 PM GMT +0000.
[18:01:50] <botifico>	 [02dns] 07MacFan4000 opened pull request 03#516: Add ACME challenges for orain.org - 13https://github.com/miraheze/dns/pull/516
[18:07:10] <icinga-miraheze>	 PROBLEM - www.bluepageswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bluepageswiki.org' expires in 15 day(s) (Tue 14 May 2024 05:49:47 PM GMT +0000).
[18:12:28] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 186.74 ms
[18:17:12] <icinga-miraheze>	 RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3754 bytes in 0.077 second response time
[18:17:13] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy
[18:17:15] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for swiftproxy171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8207
[18:17:21] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mwtask181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8160
[18:17:23] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8115
[18:17:23] <icinga-miraheze>	 RECOVERY - cp36 NTP time on cp36 is OK: NTP OK: Offset -0.03396469355 secs
[18:17:25] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw172 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8118
[18:17:26] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw162 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8116
[18:17:32] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw152 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8114
[18:17:49] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for phorge171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8202
[18:17:50] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for puppet181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8204
[18:17:51] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8113
[18:17:53] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mwtask171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8161
[18:18:09] <icinga-miraheze>	 RECOVERY - cp36 PowerDNS Recursor on cp36 is OK: DNS OK: 0.043 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[18:18:10] <icinga-miraheze>	 RECOVERY - isocasg.org - LetsEncrypt on sslhost is OK: OK - Certificate 'isocasg.org' will expire on Sun 21 Jul 2024 03:43:22 PM GMT +0000.
[18:18:19] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mon181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8201
[18:18:22] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8119
[18:18:32] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8117
[18:18:32] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw182 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8120
[18:18:32] <icinga-miraheze>	 RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[18:18:33] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for reports171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8205
[18:18:53] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for swiftproxy161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8206
[18:18:58] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for matomo151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8203
[18:19:00] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for test151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8181
[18:19:40] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw152 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8114
[18:19:45] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw162 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8116
[18:19:48] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8119
[18:19:50] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for reports171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8205
[18:19:54] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for matomo151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8203
[18:20:07] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw182 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8120
[18:20:08] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[18:20:10] <icinga-miraheze>	 RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[18:20:11] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8113
[18:20:18] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for swiftproxy171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8207
[18:20:19] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mon181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8201
[18:20:19] <icinga-miraheze>	 RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 5%
[18:20:22] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mwtask181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8160
[18:20:24] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for test151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8181
[18:20:26] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.83 ms
[18:20:30] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for phorge171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8202
[18:20:33] <icinga-miraheze>	 RECOVERY - cp37 PowerDNS Recursor on cp37 is OK: DNS OK: 0.053 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[18:20:37] <icinga-miraheze>	 PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.counterculturelabs.org' expires in 15 day(s) (Tue 14 May 2024 05:58:34 PM GMT +0000).
[18:20:39] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8115
[18:20:43] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw172 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8118
[18:20:49] <icinga-miraheze>	 RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3760 bytes in 0.080 second response time
[18:20:52] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for puppet181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8204
[18:20:59] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy
[18:21:03] <icinga-miraheze>	 RECOVERY - cp37 NTP time on cp37 is OK: NTP OK: Offset -0.00416469574 secs
[18:21:20] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for swiftproxy161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8206
[18:21:21] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mwtask171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8161
[18:21:24] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[18:21:29] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8117
[18:22:22] <icinga-miraheze>	 PROBLEM - wiki.b3313.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.b3313.org' expires in 15 day(s) (Tue 14 May 2024 06:05:40 PM GMT +0000).
[18:23:31] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#516: Add ACME challenges for orain.org - 13https://github.com/miraheze/dns/pull/516
[18:23:32] <botifico>	 [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/0fd55e2920b1...5f487a468973
[18:23:34] <botifico>	 [02miraheze/dns] 07MacFan4000 035f487a4 - Add ACME challenges for orain.org (#516)
[18:24:53] <botifico>	 [02MirahezeMagic] 07Universal-Omega closed pull request 03#489: Add electionadmin - 13https://github.com/miraheze/MirahezeMagic/pull/489
[18:24:56] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/52ec12711f95...3b957391ffe1
[18:24:58] <botifico>	 [02miraheze/MirahezeMagic] 07waki285 033b95739 - Add electionadmin (#489)
[18:25:11] <icinga-miraheze>	 RECOVERY - infectowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'infectowiki.com' will expire on Sat 27 Jul 2024 04:56:30 PM GMT +0000.
[18:27:55] <miraheze-github>	 miraheze/MirahezeMagic - waki285 the build has errored.
[18:28:33] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build has errored.
[18:29:07] <icinga-miraheze>	 PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'index.pdcommunity.ir' expires in 15 day(s) (Tue 14 May 2024 05:57:39 PM GMT +0000).
[18:29:17] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/84a58147d8b9...3cd84913d778
[18:29:20] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 033cd8491 - Bot: Update SSL cert for index.pdcommunity.ir
[18:33:00] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[18:33:02] <icinga-miraheze>	 PROBLEM - icclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'icclopedia.org' expires in 15 day(s) (Tue 14 May 2024 06:24:56 PM GMT +0000).
[18:33:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:33:09] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[18:33:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:33:13] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3cd84913d778...b35b4f427ffc
[18:33:14] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03b35b4f4 - Bot: Update SSL cert for icclopedia.org
[18:33:20] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[18:33:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:33:29] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[18:33:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:35:07] <icinga-miraheze>	 PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.minkyu.kim' expires in 15 day(s) (Tue 14 May 2024 06:10:31 PM GMT +0000).
[18:35:40] <icinga-miraheze>	 RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 16 Jul 2024 12:27:30 AM GMT +0000.
[18:37:32] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[18:37:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:37:41] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[18:37:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:37:49] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[18:37:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:37:57] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[18:38:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:44:24] <icinga-miraheze>	 PROBLEM - history.estill.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.estill.org' expires in 15 day(s) (Tue 14 May 2024 06:32:17 PM GMT +0000).
[18:44:39] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b35b4f427ffc...47cc63c770a7
[18:44:42] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0347cc63c - Bot: Update SSL cert for history.estill.org
[18:53:48] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/47cc63c770a7...3eafc3b7fd0c
[18:53:50] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 033eafc3b - Bot: Update SSL cert for orain.org
[18:54:07] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[18:54:08] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[18:54:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:54:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:54:16] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[18:54:17] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[18:54:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:54:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:54:26] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151
[18:54:27] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[18:54:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:54:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:58:39] <icinga-miraheze>	 RECOVERY - index.pdcommunity.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Sat 27 Jul 2024 05:29:12 PM GMT +0000.
[18:59:33] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.63 ms
[19:01:18] <icinga-miraheze>	 PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mineland.eu' expires in 15 day(s) (Tue 14 May 2024 06:53:49 PM GMT +0000).
[19:01:32] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.48 ms
[19:01:33] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3eafc3b7fd0c...b0f4e2577054
[19:01:35] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03b0f4e25 - Bot: Update SSL cert for wiki.mineland.eu
[19:02:35] <icinga-miraheze>	 RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Sat 27 Jul 2024 05:33:08 PM GMT +0000.
[19:05:43] <icinga-miraheze>	 PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mobilityengineer.com' expires in 15 day(s) (Tue 14 May 2024 06:55:00 PM GMT +0000).
[19:06:00] <icinga-miraheze>	 PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ricochetuniverse.com' expires in 15 day(s) (Tue 14 May 2024 06:42:50 PM GMT +0000).
[19:06:41] <icinga-miraheze>	 PROBLEM - iceria.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceria.org' expires in 15 day(s) (Tue 14 May 2024 06:58:14 PM GMT +0000).
[19:08:07] <icinga-miraheze>	 PROBLEM - wiki.danvs.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.danvs.net' expires in 15 day(s) (Tue 14 May 2024 06:44:36 PM GMT +0000).
[19:09:46] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/b0f4e2577054...211cf9b90251
[19:09:47] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03211cf9b - Bot: Add SSL cert for wkmr.liao.media
[19:10:07] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'franchise.franchising.org.ua' expires in 15 day(s) (Tue 14 May 2024 06:55:42 PM GMT +0000).
[19:12:25] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:17:02] <icinga-miraheze>	 PROBLEM - www.iceria.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceria.org' expires in 15 day(s) (Tue 14 May 2024 06:58:14 PM GMT +0000).
[19:17:13] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/211cf9b90251...a7cb8f4fe863
[19:17:15] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03a7cb8f4 - Bot: Update SSL cert for www.iceria.org
[19:19:57] <icinga-miraheze>	 RECOVERY - isv.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Sat 27 Jul 2024 05:53:43 PM GMT +0000.
[19:30:33] <icinga-miraheze>	 PROBLEM - wiki.ssij.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ssij.se' expires in 15 day(s) (Tue 14 May 2024 07:05:30 PM GMT +0000).
[19:30:47] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a7cb8f4fe863...f585a6caec68
[19:30:49] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03f585a6c - Bot: Update SSL cert for wiki.ssij.se
[19:30:58] <icinga-miraheze>	 RECOVERY - wiki.mineland.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Sat 27 Jul 2024 06:01:27 PM GMT +0000.
[19:31:54] <icinga-miraheze>	 RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Sat 27 Jul 2024 05:53:43 PM GMT +0000.
[19:35:58] <icinga-miraheze>	 PROBLEM - tl.awiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tl.awiki.org' expires in 15 day(s) (Tue 14 May 2024 07:22:51 PM GMT +0000).
[19:36:09] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f585a6caec68...c2cfad549976
[19:36:10] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03c2cfad5 - Bot: Update SSL cert for tl.awiki.org
[19:37:35] <icinga-miraheze>	 PROBLEM - fc.songcontests.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fc.songcontests.eu' expires in 15 day(s) (Tue 14 May 2024 07:32:36 PM GMT +0000).
[19:37:45] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c2cfad549976...cecaebc1de15
[19:37:48] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03cecaebc - Bot: Update SSL cert for fc.songcontests.eu
[19:42:25] <icinga-miraheze>	 RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Sat 27 Jul 2024 05:44:33 PM GMT +0000.
[19:43:53] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.57 ms
[19:45:52] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.78 ms
[19:48:02] <icinga-miraheze>	 RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Sat 27 Jul 2024 05:53:43 PM GMT +0000.
[19:48:31] <icinga-miraheze>	 PROBLEM - wkmr.liao.media - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1-32.azure-dns.com.', 'ns2-32.azure-dns.net.', 'ns3-32.azure-dns.org.', 'ns4-32.azure-dns.info.'], 'CNAME': None}
[19:50:36] <icinga-miraheze>	 PROBLEM - guia.esporo.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.esporo.net' expires in 15 day(s) (Tue 14 May 2024 07:37:22 PM GMT +0000).
[19:50:47] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cecaebc1de15...3e7c03938d87
[19:50:49] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 033e7c039 - Bot: Update SSL cert for guia.esporo.net
[19:59:22] <icinga-miraheze>	 PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pubwiki.lab.co.pl' expires in 15 day(s) (Tue 14 May 2024 07:49:08 PM GMT +0000).
[19:59:33] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3e7c03938d87...e6267fe68484
[19:59:34] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03e6267fe - Bot: Update SSL cert for pubwiki.lab.co.pl
[20:05:00] <icinga-miraheze>	 RECOVERY - wiki.ssij.se - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ssij.se' will expire on Sat 27 Jul 2024 06:30:41 PM GMT +0000.
[20:05:26] <icinga-miraheze>	 RECOVERY - tl.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Sat 27 Jul 2024 06:36:04 PM GMT +0000.
[20:05:28] <icinga-miraheze>	 PROBLEM - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lefrenchmelee.fr' expires in 15 day(s) (Tue 14 May 2024 07:48:47 PM GMT +0000).
[20:06:20] <icinga-miraheze>	 RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 27 Jul 2024 06:17:08 PM GMT +0000.
[20:07:25] <icinga-miraheze>	 RECOVERY - fc.songcontests.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'fc.songcontests.eu' will expire on Sat 27 Jul 2024 06:37:40 PM GMT +0000.
[20:08:02] <icinga-miraheze>	 RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 27 Jul 2024 06:17:08 PM GMT +0000.
[20:11:05] <icinga-miraheze>	 PROBLEM - wiki.insideearth.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.insideearth.info' expires in 15 day(s) (Tue 14 May 2024 07:50:13 PM GMT +0000).
[20:11:17] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e6267fe68484...85f0b1133601
[20:11:19] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0385f0b11 - Bot: Update SSL cert for wiki.insideearth.info
[20:12:45] <icinga-miraheze>	 PROBLEM - baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 15 day(s) (Tue 14 May 2024 08:02:17 PM GMT +0000).
[20:15:53] <icinga-miraheze>	 PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.warframestat.us' expires in 15 day(s) (Tue 14 May 2024 08:01:49 PM GMT +0000).
[20:16:04] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/85f0b1133601...bfa9892c3437
[20:16:05] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03bfa9892 - Bot: Update SSL cert for wiki.warframestat.us
[20:19:31] <icinga-miraheze>	 RECOVERY - guia.esporo.net - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.esporo.net' will expire on Sat 27 Jul 2024 06:50:42 PM GMT +0000.
[20:29:14] <icinga-miraheze>	 RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Sat 27 Jul 2024 06:59:28 PM GMT +0000.
[20:29:32] <icinga-miraheze>	 PROBLEM - wiki.triplescripts.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.triplescripts.org' expires in 15 day(s) (Tue 14 May 2024 08:04:03 PM GMT +0000).
[20:29:35] <icinga-miraheze>	 PROBLEM - www.baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 15 day(s) (Tue 14 May 2024 08:02:17 PM GMT +0000).
[20:29:43] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bfa9892c3437...17adcb1f5e6e
[20:29:45] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0317adcb1 - Bot: Update SSL cert for wiki.triplescripts.org
[20:29:53] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/17adcb1f5e6e...34488f90a667
[20:29:54] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0334488f9 - Bot: Update SSL cert for www.baharna.org
[20:33:34] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[20:36:26] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/34488f90a667...bd6821d5ec0a
[20:36:27] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03bd6821d - Bot: Update SSL cert for wiki.insideearth.info
[20:36:46] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bd6821d5ec0a...f33d385391e1
[20:36:49] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03f33d385 - Bot: Update SSL cert for wiki.warframestat.us
[20:37:57] <MirahezeLSBot>	 !log [universalomega@test151] starting deploy of {'force_upgrade': True, 'versions': '1.41', 'upgrade_extensions': 'MultiPurge'} to test151
[20:37:58] <MirahezeLSBot>	 !log [universalomega@test151] finished deploy of {'force_upgrade': True, 'versions': '1.41', 'upgrade_extensions': 'MultiPurge'} to test151 - SUCCESS in 1s
[20:38:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:38:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:38:32] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f33d385391e1...338455c0d70b
[20:38:35] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03338455c - Bot: Update SSL cert for wiki.lefrenchmelee.fr
[20:41:56] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/338455c0d70b...02be609fc882
[20:41:59] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0302be609 - Bot: Update SSL cert for franchise.franchising.org.ua
[20:45:20] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/02be609fc882...0739c6a52bb9
[20:45:21] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 030739c6a - Bot: Update SSL cert for wiki.danvs.net
[20:45:33] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0739c6a52bb9...54030af5d184
[20:45:36] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0354030af - Bot: Update SSL cert for wiki.ricochetuniverse.com
[20:50:57] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.01 ms
[20:52:56] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.43 ms
[20:58:32] <icinga-miraheze>	 RECOVERY - wiki.triplescripts.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.triplescripts.org' will expire on Sat 27 Jul 2024 07:29:37 PM GMT +0000.
[20:58:52] <icinga-miraheze>	 RECOVERY - www.baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Sat 27 Jul 2024 07:29:48 PM GMT +0000.
[21:01:20] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[21:03:23] <icinga-miraheze>	 RECOVERY - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Sat 27 Jul 2024 07:38:26 PM GMT +0000.
[21:05:45] <icinga-miraheze>	 PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.shaazzz.ir' expires in 15 day(s) (Tue 14 May 2024 08:35:09 PM GMT +0000).
[21:05:57] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/54030af5d184...3cffbf9fafa9
[21:06:00] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 033cffbf9 - Bot: Update SSL cert for wiki.shaazzz.ir
[21:07:24] <icinga-miraheze>	 PROBLEM - wiki.climatechange.ai - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.climatechange.ai' expires in 15 day(s) (Tue 14 May 2024 08:52:20 PM GMT +0000).
[21:07:35] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3cffbf9fafa9...e698a9911808
[21:07:36] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03e698a99 - Bot: Update SSL cert for wiki.climatechange.ai
[21:08:59] <icinga-miraheze>	 RECOVERY - wiki.insideearth.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.insideearth.info' will expire on Sat 27 Jul 2024 07:36:20 PM GMT +0000.
[21:11:09] <icinga-miraheze>	 RECOVERY - baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Sat 27 Jul 2024 07:29:48 PM GMT +0000.
[21:13:00] <icinga-miraheze>	 PROBLEM - wiki.cdntennis.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cdntennis.ca' expires in 15 day(s) (Tue 14 May 2024 08:59:10 PM GMT +0000).
[21:13:12] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e698a9911808...b66592b9bfc7
[21:13:14] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03b66592b - Bot: Update SSL cert for wiki.cdntennis.ca
[21:14:09] <botifico>	 [02CreateWiki] 07Universal-Omega closed pull request 03#403: T10674: Companion work for task; sets who grant an exemption for inactivity + timestamp - 13https://github.com/miraheze/CreateWiki/pull/403
[21:14:13] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03inactive-exempt-page
[21:14:15] <botifico>	 [02CreateWiki] 07Universal-Omega deleted branch 03inactive-exempt-page - 13https://github.com/miraheze/CreateWiki
[21:14:23] <botifico>	 [02ManageWiki] 07Universal-Omega closed pull request 03#399: T10674: Create Special:InactivityExemptWikis page - 13https://github.com/miraheze/ManageWiki/pull/399
[21:14:26] <botifico>	 [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03inactive-exempt-page
[21:14:28] <botifico>	 [02ManageWiki] 07Universal-Omega deleted branch 03inactive-exempt-page - 13https://github.com/miraheze/ManageWiki
[21:14:39] <botifico>	 [02ManageWiki] 07Universal-Omega closed pull request 03#375: Warn/lock if in read-only - 13https://github.com/miraheze/ManageWiki/pull/375
[21:15:20] <icinga-miraheze>	 RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Sat 27 Jul 2024 07:36:40 PM GMT +0000.
[21:15:21] <botifico>	 [02ManageWiki] 07Universal-Omega closed pull request 03#456: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/ManageWiki/pull/456
[21:15:24] <botifico>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/bd41729d7283...a0aaa702a536
[21:15:26] <botifico>	 [02miraheze/ManageWiki] 07dependabot[bot] 03a0aaa70 - Update php-parallel-lint/php-parallel-lint requirement (#456)
[21:15:27] <botifico>	 [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:15:30] <botifico>	 [02ManageWiki] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/ManageWiki
[21:15:42] <botifico>	 [02CreateWiki] 07Universal-Omega closed pull request 03#498: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/CreateWiki/pull/498
[21:15:44] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:15:47] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/36ea9424155e...33bb22ca9caf
[21:15:48] <botifico>	 [02miraheze/CreateWiki] 07dependabot[bot] 0333bb22c - Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 (#498)
[21:15:49] <botifico>	 [02CreateWiki] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/CreateWiki
[21:15:51] <botifico>	 [02CreateWiki] 07Universal-Omega closed pull request 03#499: Bump eslint-config-wikimedia from 0.26.0 to 0.27.0 - 13https://github.com/miraheze/CreateWiki/pull/499
[21:15:52] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.27.0
[21:15:54] <icinga-miraheze>	 PROBLEM - n64brew.dev - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'n64brew.dev' expires in 15 day(s) (Tue 14 May 2024 08:47:45 PM GMT +0000).
[21:15:54] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/33bb22ca9caf...237257b64e4e
[21:15:56] <botifico>	 [02miraheze/CreateWiki] 07dependabot[bot] 03237257b - Bump eslint-config-wikimedia from 0.26.0 to 0.27.0 (#499)
[21:15:58] <botifico>	 [02CreateWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.27.0 - 13https://github.com/miraheze/CreateWiki
[21:16:10] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b66592b9bfc7...914c23dbdb9d
[21:16:12] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03914c23d - Bot: Update SSL cert for n64brew.dev
[21:16:22] <botifico>	 [02MirahezeMagic] 07Universal-Omega closed pull request 03#486: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/MirahezeMagic/pull/486
[21:16:24] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/3b957391ffe1...a4f62b38e403
[21:16:27] <botifico>	 [02miraheze/MirahezeMagic] 07dependabot[bot] 03a4f62b3 - Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 (#486)
[21:16:28] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:16:30] <botifico>	 [02MirahezeMagic] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/MirahezeMagic
[21:17:11] <botifico>	 [02MirahezeMagic] 07Universal-Omega closed pull request 03#488: Consistent capitalization in a message - 13https://github.com/miraheze/MirahezeMagic/pull/488
[21:17:12] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/a4f62b38e403...d3e8911ea480
[21:17:14] <botifico>	 [02miraheze/MirahezeMagic] 07amire80 03d3e8911 - Consistent capitalization in a message (#488)
[21:17:55] <botifico>	 [02IncidentReporting] 07Universal-Omega closed pull request 03#78: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/IncidentReporting/pull/78
[21:17:57] <botifico>	 [02miraheze/IncidentReporting] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:17:58] <botifico>	 [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/68bdc01d29a6...20686f89cf75
[21:18:01] <botifico>	 [02miraheze/IncidentReporting] 07dependabot[bot] 0320686f8 - Update php-parallel-lint/php-parallel-lint requirement (#78)
[21:18:03] <botifico>	 [02IncidentReporting] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/IncidentReporting
[21:18:04] <botifico>	 [02IncidentReporting] 07Universal-Omega closed pull request 03#79: Bump eslint-config-wikimedia from 0.26.0 to 0.27.0 - 13https://github.com/miraheze/IncidentReporting/pull/79
[21:18:07] <botifico>	 [02miraheze/IncidentReporting] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.27.0
[21:18:08] <botifico>	 [02IncidentReporting] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.27.0 - 13https://github.com/miraheze/IncidentReporting
[21:18:09] <botifico>	 [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/IncidentReporting/compare/20686f89cf75...7e04cae7e276
[21:18:12] <botifico>	 [02miraheze/IncidentReporting] 07dependabot[bot] 037e04cae - Bump eslint-config-wikimedia from 0.26.0 to 0.27.0 (#79)
[21:18:20] <botifico>	 [02WikiDiscover] 07Universal-Omega closed pull request 03#114: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/WikiDiscover/pull/114
[21:18:23] <botifico>	 [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/1a7af36a1c53...3c8f46e3aad5
[21:18:24] <botifico>	 [02miraheze/WikiDiscover] 07dependabot[bot] 033c8f46e - Update php-parallel-lint/php-parallel-lint requirement (#114)
[21:18:25] <botifico>	 [02miraheze/WikiDiscover] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:18:28] <botifico>	 [02WikiDiscover] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/WikiDiscover
[21:18:58] <botifico>	 [02landing] 07Universal-Omega closed pull request 03#147: Use jsdelivr - 13https://github.com/miraheze/landing/pull/147
[21:18:59] <botifico>	 [02landing] 07Universal-Omega commented on pull request 03#147: Use jsdelivr - 13https://github.com/miraheze/landing/pull/147#issuecomment-2081659649
[21:19:29] <botifico>	 [02YouTube] 07Universal-Omega closed pull request 03#26: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/YouTube/pull/26
[21:19:31] <botifico>	 [02miraheze/YouTube] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/YouTube/compare/118939def53b...7a6dee595cb5
[21:19:32] <botifico>	 [02miraheze/YouTube] 07dependabot[bot] 037a6dee5 - Update php-parallel-lint/php-parallel-lint requirement (#26)
[21:19:33] <botifico>	 [02miraheze/YouTube] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:19:35] <botifico>	 [02YouTube] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/YouTube
[21:20:12] <botifico>	 [02RequestSSL] 07Universal-Omega closed pull request 03#42: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RequestSSL/pull/42
[21:20:14] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[21:20:14] <botifico>	 [02miraheze/RequestSSL] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/compare/77c22683a9a8...4a730e5acaea
[21:20:15] <botifico>	 [02miraheze/RequestSSL] 07dependabot[bot] 034a730e5 - Update php-parallel-lint/php-parallel-lint requirement (#42)
[21:20:16] <botifico>	 [02miraheze/RequestSSL] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:20:18] <botifico>	 [02RequestSSL] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/RequestSSL
[21:20:25] <botifico>	 [02RequestSSL] 07Universal-Omega closed pull request 03#43: Bump eslint-config-wikimedia from 0.26.0 to 0.27.0 - 13https://github.com/miraheze/RequestSSL/pull/43
[21:20:26] <botifico>	 [02miraheze/RequestSSL] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/RequestSSL/compare/4a730e5acaea...21e2bf773e40
[21:20:27] <botifico>	 [02miraheze/RequestSSL] 07dependabot[bot] 0321e2bf7 - Bump eslint-config-wikimedia from 0.26.0 to 0.27.0 (#43)
[21:20:30] <botifico>	 [02miraheze/RequestSSL] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.27.0
[21:20:32] <botifico>	 [02RequestSSL] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.27.0 - 13https://github.com/miraheze/RequestSSL
[21:20:38] <botifico>	 [02RottenLinks] 07Universal-Omega closed pull request 03#78: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RottenLinks/pull/78
[21:20:40] <botifico>	 [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/56b882420168...cf398b0125bb
[21:20:41] <botifico>	 [02miraheze/RottenLinks] 07dependabot[bot] 03cf398b0 - Update php-parallel-lint/php-parallel-lint requirement (#78)
[21:20:44] <botifico>	 [02miraheze/RottenLinks] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:20:47] <botifico>	 [02RottenLinks] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/RottenLinks
[21:20:55] <botifico>	 [02DataDump] 07Universal-Omega closed pull request 03#85: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/DataDump/pull/85
[21:20:58] <botifico>	 [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/65d548a589ef...3889d76ddcb9
[21:21:00] <botifico>	 [02miraheze/DataDump] 07dependabot[bot] 033889d76 - Update php-parallel-lint/php-parallel-lint requirement (#85)
[21:21:02] <botifico>	 [02miraheze/DataDump] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:21:03] <botifico>	 [02DataDump] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/DataDump
[21:21:18] <botifico>	 [02MatomoAnalytics] 07Universal-Omega closed pull request 03#122: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/122
[21:21:19] <botifico>	 [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/7032b175a768...c21d7eb5859f
[21:21:20] <botifico>	 [02miraheze/MatomoAnalytics] 07dependabot[bot] 03c21d7eb - Update php-parallel-lint/php-parallel-lint requirement (#122)
[21:21:21] <botifico>	 [02miraheze/MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:21:23] <botifico>	 [02MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/MatomoAnalytics
[21:21:29] <botifico>	 [02MatomoAnalytics] 07Universal-Omega closed pull request 03#123: Bump eslint-config-wikimedia from 0.26.0 to 0.27.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/123
[21:21:30] <botifico>	 [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/compare/c21d7eb5859f...e63b8a3f55ba
[21:21:33] <botifico>	 [02miraheze/MatomoAnalytics] 07dependabot[bot] 03e63b8a3 - Bump eslint-config-wikimedia from 0.26.0 to 0.27.0 (#123)
[21:21:34] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build has errored.
[21:21:36] <botifico>	 [02miraheze/MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.27.0
[21:21:39] <botifico>	 [02MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.27.0 - 13https://github.com/miraheze/MatomoAnalytics
[21:21:47] <botifico>	 [02GlobalNewFiles] 07Universal-Omega closed pull request 03#55: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/GlobalNewFiles/pull/55
[21:21:48] <miraheze-github>	 miraheze/YouTube - koreirozzu the build has errored.
[21:21:49] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/c2365a858cc6...6b7eaef7eab4
[21:21:52] <botifico>	 [02miraheze/GlobalNewFiles] 07dependabot[bot] 036b7eaef - Update php-parallel-lint/php-parallel-lint requirement (#55)
[21:21:54] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:21:56] <botifico>	 [02GlobalNewFiles] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/GlobalNewFiles
[21:22:10] <botifico>	 [02PDFEmbed] 07Universal-Omega closed pull request 03#20: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/PDFEmbed/pull/20
[21:22:12] <botifico>	 [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/9cc54cbbc859...f3212c2f0052
[21:22:13] <botifico>	 [02miraheze/PDFEmbed] 07dependabot[bot] 03f3212c2 - Update php-parallel-lint/php-parallel-lint requirement (#20)
[21:22:15] <botifico>	 [02miraheze/PDFEmbed] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0
[21:22:16] <botifico>	 [02PDFEmbed] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/PDFEmbed
[21:22:20] <miraheze-github>	 miraheze/IncidentReporting - Universal-Omega the build passed.
[21:22:44] <miraheze-github>	 miraheze/WikiDiscover - Universal-Omega the build passed.
[21:22:51] <botifico>	 [02.github] 07Universal-Omega closed pull request 03#47: Bump actions/cache from 4.0.1 to 4.0.2 - 13https://github.com/miraheze/.github/pull/47
[21:22:53] <botifico>	 [02miraheze/.github] 07Universal-Omega deleted branch 03dependabot/github_actions/actions/cache-4.0.2
[21:22:55] <botifico>	 [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/8a00ac8eb122...a78688425a4d
[21:22:58] <botifico>	 [02miraheze/.github] 07dependabot[bot] 03a786884 - Bump actions/cache from 4.0.1 to 4.0.2 (#47)
[21:23:01] <botifico>	 [02.github] 07Universal-Omega deleted branch 03dependabot/github_actions/actions/cache-4.0.2 - 13https://github.com/miraheze/.github
[21:23:24] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[21:23:25] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[21:23:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:23:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:23:34] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[21:23:35] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[21:23:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:23:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:23:44] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151
[21:23:45] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[21:23:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:23:49] <miraheze-github>	 miraheze/YouTube - Universal-Omega the build passed.
[21:23:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:24:26] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[21:25:11] <miraheze-github>	 miraheze/RottenLinks - Universal-Omega the build passed.
[21:25:41] <icinga-miraheze>	 PROBLEM - hololive.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hololive.wiki' expires in 15 day(s) (Tue 14 May 2024 08:59:19 PM GMT +0000).
[21:25:46] <miraheze-github>	 miraheze/DataDump - Universal-Omega the build passed.
[21:25:52] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/914c23dbdb9d...9ff0b0c18c92
[21:25:53] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 039ff0b0c - Bot: Update SSL cert for hololive.wiki
[21:26:08] <miraheze-github>	 miraheze/MatomoAnalytics - Universal-Omega the build passed.
[21:26:20] <miraheze-github>	 miraheze/GlobalNewFiles - Universal-Omega the build passed.
[21:26:44] <miraheze-github>	 miraheze/PDFEmbed - Universal-Omega the build passed.
[21:26:51] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9ff0b0c18c92...35694f97b619
[21:26:52] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0335694f9 - Bot: Update SSL cert for wiki.mobilityengineer.com
[21:28:16] <icinga-miraheze>	 PROBLEM - wiki.vinesh.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.vinesh.eu.org' expires in 15 day(s) (Tue 14 May 2024 09:23:39 PM GMT +0000).
[21:28:30] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/35694f97b619...37eea3868c33
[21:28:31] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0337eea38 - Bot: Update SSL cert for wiki.vinesh.eu.org
[21:29:19] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[21:30:13] <icinga-miraheze>	 PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aman.awiki.org' expires in 15 day(s) (Tue 14 May 2024 09:16:41 PM GMT +0000).
[21:30:19] <icinga-miraheze>	 RECOVERY - wiki.danvs.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.danvs.net' will expire on Sat 27 Jul 2024 07:45:15 PM GMT +0000.
[21:30:26] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/37eea3868c33...00042d4bcb1e
[21:30:29] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0300042d4 - Bot: Update SSL cert for aman.awiki.org
[21:31:10] <icinga-miraheze>	 RECOVERY - franchise.franchising.org.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'franchise.franchising.org.ua' will expire on Sat 27 Jul 2024 07:41:51 PM GMT +0000.
[21:32:53] <icinga-miraheze>	 RECOVERY - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ricochetuniverse.com' will expire on Sat 27 Jul 2024 07:45:28 PM GMT +0000.
[21:33:16] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[21:33:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:33:26] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[21:33:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:33:38] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[21:33:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:33:47] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[21:33:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:34:50] <icinga-miraheze>	 RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Sat 27 Jul 2024 08:05:51 PM GMT +0000.
[21:36:46] <icinga-miraheze>	 RECOVERY - wiki.climatechange.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Sat 27 Jul 2024 08:07:29 PM GMT +0000.
[21:37:20] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[21:37:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:37:28] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[21:37:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:37:39] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[21:37:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:37:46] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 7s
[21:37:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:38:33] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/00042d4bcb1e...2e79a04e45ec
[21:38:34] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 032e79a04 - Bot: Update SSL cert for wiki.minkyu.kim
[21:38:57] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[21:55:30] <icinga-miraheze>	 RECOVERY - hololive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hololive.wiki' will expire on Sat 27 Jul 2024 08:25:47 PM GMT +0000.
[21:57:24] <icinga-miraheze>	 RECOVERY - wiki.vinesh.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.vinesh.eu.org' will expire on Sat 27 Jul 2024 08:28:24 PM GMT +0000.
[22:00:06] <icinga-miraheze>	 RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Sat 27 Jul 2024 08:38:28 PM GMT +0000.
[22:00:12] <icinga-miraheze>	 RECOVERY - aman.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'aman.awiki.org' will expire on Sat 27 Jul 2024 08:30:19 PM GMT +0000.
[22:00:16] <icinga-miraheze>	 RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Sat 27 Jul 2024 08:26:46 PM GMT +0000.
[22:08:02] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[22:12:54] <icinga-miraheze>	 RECOVERY - wiki.cdntennis.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cdntennis.ca' will expire on Sat 27 Jul 2024 08:13:06 PM GMT +0000.
[22:13:09] <icinga-miraheze>	 RECOVERY - n64brew.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'n64brew.dev' will expire on Sat 27 Jul 2024 08:16:03 PM GMT +0000.
[22:22:25] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:37:46] <botifico>	 [02CreateWiki] 07pixDeVl synchronize pull request 03#511: Notify Wiki Creators about requesters with over X(5) request (T12075)  - 13https://github.com/miraheze/CreateWiki/pull/511
[22:40:38] <botifico>	 [02CreateWiki] 07pixDeVl synchronize pull request 03#511: Notify Wiki Creators about requesters with over X(5) request (T12075)  - 13https://github.com/miraheze/CreateWiki/pull/511
[22:41:22] <botifico>	 [02CreateWiki] 07pixDeVl synchronize pull request 03#511: Notify Wiki Creators about requesters with over X(5) request (T12075)  - 13https://github.com/miraheze/CreateWiki/pull/511
[22:44:26] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2e79a04e45ec...069ded5cdac8
[22:44:29] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03069ded5 - Bot: Update SSL cert for wiki.b3313.org
[22:47:10] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/069ded5cdac8...8d7f9a8d8cd5
[22:47:11] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 038d7f9a8 - Bot: Update SSL cert for wiki.counterculturelabs.org
[22:48:23] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8d7f9a8d8cd5...11a717cba55c
[22:48:26] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0311a717c - Bot: Update SSL cert for www.bluepageswiki.org
[22:50:51] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/11a717cba55c...b6932e9cda70
[22:50:54] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03b6932e9 - Bot: Update SSL cert for www.mcpk.wiki
[22:51:45] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b6932e9cda70...8349275a3859
[22:51:46] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 038349275 - Bot: Update SSL cert for archive.stellurgists.wiki
[22:53:34] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8349275a3859...4cd643503d92
[22:53:37] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 034cd6435 - Bot: Update SSL cert for wiki.villagecollaborative.net
[22:57:09] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4cd643503d92...baf826dedd04
[22:57:11] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03baf826d - Bot: Update SSL cert for wiki.valkyrienskies.org
[22:59:33] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/baf826dedd04...20477420f831
[22:59:35] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 032047742 - Bot: Update SSL cert for wiki.zymonic.com
[23:00:45] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/20477420f831...3bd115310008
[23:00:46] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 033bd1153 - Bot: Update SSL cert for wiki.openhatch.org
[23:01:30] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3bd115310008...a52d09a42548
[23:01:31] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03a52d09a - Bot: Update SSL cert for wiki.joust.ro
[23:02:16] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a52d09a42548...3450fc99ccff
[23:02:18] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 033450fc9 - Bot: Update SSL cert for wiki.nowchess.org
[23:02:55] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3450fc99ccff...411165bad52e
[23:02:57] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03411165b - Bot: Update SSL cert for wiki.mxlinuxusers.de
[23:04:08] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/411165bad52e...580d8decb88f
[23:04:11] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03580d8de - Bot: Update SSL cert for data.nonbinary.wiki
[23:04:47] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/580d8decb88f...4ed95893593e
[23:04:49] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 034ed9589 - Bot: Update SSL cert for guia.cineastas.pt
[23:04:58] <icinga-miraheze>	 RECOVERY - www.bluepageswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluepageswiki.org' will expire on Sat 27 Jul 2024 09:48:17 PM GMT +0000.
[23:05:27] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4ed95893593e...58c615ef47df
[23:05:28] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0358c615e - Bot: Update SSL cert for wiki.ciptamedia.org
[23:06:11] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/58c615ef47df...0154e90a0f0c
[23:06:12] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 030154e90 - Bot: Update SSL cert for ao90.pinho.org
[23:06:59] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0154e90a0f0c...72512527c514
[23:07:01] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 037251252 - Bot: Update SSL cert for tep.wiki
[23:07:53] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/72512527c514...b47002c7b4e3
[23:07:55] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03b47002c - Bot: Update SSL cert for cyberlaw.ccdcoe.org
[23:09:04] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b47002c7b4e3...dfa0c8f4eda7
[23:09:05] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03dfa0c8f - Bot: Update SSL cert for wiki.s23.org
[23:10:01] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dfa0c8f4eda7...40183aa38bc2
[23:10:02] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0340183aa - Bot: Update SSL cert for wiki.scvo.org
[23:10:06] <icinga-miraheze>	 RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Sat 27 Jul 2024 09:47:05 PM GMT +0000.
[23:10:40] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/40183aa38bc2...a8bc149501b8
[23:10:41] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03a8bc149 - Bot: Update SSL cert for wiki.mikrodev.com
[23:11:00] <icinga-miraheze>	 RECOVERY - wiki.b3313.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.b3313.org' will expire on Sat 27 Jul 2024 09:44:21 PM GMT +0000.
[23:11:32] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a8bc149501b8...6f6c9447a0bb
[23:11:35] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 036f6c944 - Bot: Update SSL cert for wiki.graalmilitary.com
[23:12:55] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6f6c9447a0bb...018b31978527
[23:12:56] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03018b319 - Bot: Update SSL cert for wiki.grottocenter.org
[23:13:40] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/018b31978527...d24f530fcca7
[23:13:42] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03d24f530 - Bot: Update SSL cert for wiki.worlduniversityandschool.org
[23:14:33] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d24f530fcca7...217f2a25454d
[23:14:35] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03217f2a2 - Bot: Update SSL cert for wiki.macc.nyc
[23:14:57] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/217f2a25454d...c18f765f4ca1
[23:14:58] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03c18f765 - Bot: Update SSL cert for ie.unigon.net
[23:16:00] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c18f765f4ca1...1c0b55915118
[23:16:02] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 031c0b559 - Bot: Update SSL cert for wiki.kkstudio.eu.org
[23:20:51] <icinga-miraheze>	 RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Sat 27 Jul 2024 10:06:54 PM GMT +0000.
[23:21:02] <icinga-miraheze>	 RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Sat 27 Jul 2024 10:02:49 PM GMT +0000.
[23:22:31] <icinga-miraheze>	 RECOVERY - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberlaw.ccdcoe.org' will expire on Sat 27 Jul 2024 10:07:47 PM GMT +0000.
[23:22:47] <icinga-miraheze>	 RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Sat 27 Jul 2024 09:53:28 PM GMT +0000.
[23:24:17] <icinga-miraheze>	 RECOVERY - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worlduniversityandschool.org' will expire on Sat 27 Jul 2024 10:13:34 PM GMT +0000.
[23:24:54] <icinga-miraheze>	 RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Sat 27 Jul 2024 10:04:41 PM GMT +0000.
[23:25:06] <icinga-miraheze>	 RECOVERY - wiki.openhatch.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.openhatch.org' will expire on Sat 27 Jul 2024 10:00:39 PM GMT +0000.
[23:26:10] <icinga-miraheze>	 RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scvo.org' will expire on Sat 27 Jul 2024 10:09:55 PM GMT +0000.
[23:26:41] <icinga-miraheze>	 RECOVERY - ie.unigon.net - LetsEncrypt on sslhost is OK: OK - Certificate 'ie.unigon.net' will expire on Sat 27 Jul 2024 10:14:51 PM GMT +0000.
[23:27:23] <icinga-miraheze>	 RECOVERY - archive.stellurgists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.stellurgists.wiki' will expire on Sat 27 Jul 2024 09:51:40 PM GMT +0000.
[23:27:57] <icinga-miraheze>	 RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Sat 27 Jul 2024 10:01:25 PM GMT +0000.
[23:28:36] <icinga-miraheze>	 RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Sat 27 Jul 2024 10:14:27 PM GMT +0000.
[23:32:14] <icinga-miraheze>	 RECOVERY - www.mcpk.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mcpk.wiki' will expire on Sat 27 Jul 2024 09:50:45 PM GMT +0000.
[23:33:25] <icinga-miraheze>	 RECOVERY - wiki.grottocenter.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grottocenter.org' will expire on Sat 27 Jul 2024 10:12:50 PM GMT +0000.
[23:34:20] <icinga-miraheze>	 RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mikrodev.com' will expire on Sat 27 Jul 2024 10:10:35 PM GMT +0000.
[23:34:54] <icinga-miraheze>	 RECOVERY - wiki.kkstudio.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kkstudio.eu.org' will expire on Sat 27 Jul 2024 10:15:55 PM GMT +0000.
[23:35:22] <icinga-miraheze>	 RECOVERY - data.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.nonbinary.wiki' will expire on Sat 27 Jul 2024 10:04:02 PM GMT +0000.
[23:35:26] <icinga-miraheze>	 RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Sat 27 Jul 2024 10:06:05 PM GMT +0000.
[23:35:34] <icinga-miraheze>	 RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Sat 27 Jul 2024 09:59:27 PM GMT +0000.
[23:38:18] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[23:38:49] <icinga-miraheze>	 RECOVERY - wiki.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.graalmilitary.com' will expire on Sat 27 Jul 2024 10:11:27 PM GMT +0000.
[23:41:20] <icinga-miraheze>	 RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Sat 27 Jul 2024 10:08:59 PM GMT +0000.
[23:42:22] <icinga-miraheze>	 RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Sat 27 Jul 2024 10:05:21 PM GMT +0000.
[23:45:36] <icinga-miraheze>	 RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Sat 27 Jul 2024 09:57:04 PM GMT +0000.
[23:47:49] <icinga-miraheze>	 RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Sat 27 Jul 2024 10:02:10 PM GMT +0000.