[00:00:07] <icinga-miraheze>	 RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[00:00:17] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[00:00:24] <icinga-miraheze>	 RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[00:00:32] <icinga-miraheze>	 RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:00:34] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[00:00:41] <icinga-miraheze>	 RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:01:03] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:02:18] <icinga-miraheze>	 RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[00:02:21] <icinga-miraheze>	 RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:02:27] <icinga-miraheze>	 RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:02:38] <icinga-miraheze>	 RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[00:03:19] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[00:05:08] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:05:18] <icinga-miraheze>	 RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:05:18] <icinga-miraheze>	 RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:06:21] <icinga-miraheze>	 RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[00:07:37] <icinga-miraheze>	 RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:07:48] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[00:08:15] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[00:08:41] <icinga-miraheze>	 RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:09:31] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[00:10:02] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:44:01] <MirahezeRelay>	 <pixldev> any beta stewards/other global permissions available? need a perm fixed to test a PR(cc @reception123 )
[01:44:19] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[01:44:41] <MirahezeRelay>	 <pixldev> hush
[01:55:15] <MirahezeRelay>	 <orduin, replying to pixldev> What needs doing?
[01:56:33] <MirahezeRelay>	 <pixldev, replying to orduin> crate on metabeta still just has the legacy `managewiki` right from when it was all one thing, could you grant it `managewiki-permissions` and the other core one so i can fix it up and set up the ability to grant groups as well>
[01:58:57] <MacFan4000>	 Done
[02:01:17] <MirahezeRelay>	 <pixldev> thank you!
[02:01:27] <MirahezeRelay>	 <pixldev> time to take over beta
[02:03:24] <MirahezeRelay>	 <pixldev> huh wiki creator dont exist on beta
[02:03:36] <MirahezeRelay>	 <pixldev> since i need to test that for now ill just make one rq ig
[02:05:28] <MirahezeRelay>	 <pixldev> [1/2] wha
[02:05:29] <MirahezeRelay>	 <pixldev> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1280710293068185710/saEnowM.png?ex=66d911e8&is=66d7c068&hm=6fe1f2ca3446f4bd745d15ebae6d4edd9f811895c6d0649b5869ea1513960864&
[02:06:23] <MacFan4000>	 Pretty sure createwiki rights aren’t manageable via the ManageWiki UI
[02:06:38] <MirahezeRelay>	 <pixldev> ..por que
[02:07:30] <MirahezeRelay>	 <pixldev> why is `view-private-ssl-requests` in MW and createwiki isn't...
[02:07:32] <MirahezeRelay>	 <pixldev> welp
[02:07:39] <MirahezeRelay>	 <pixldev> ill have to figure this out tmr
[02:09:15] <MirahezeRelay>	 <pixldev, replying to MacFan4000> if its assigned in config will creating a wikicreator group on beta inherit the rights?
[02:09:54] <MirahezeRelay>	 <pixldev> or is it on prod meta only
[02:11:21] <MacFan4000>	 Hmm that perm probably should have been added to the list
[02:11:55] <MirahezeRelay>	 <pixldev> yeeah
[02:12:06] <MacFan4000>	 https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L3402 controls what is assignable via the UI
[02:12:23] <MirahezeRelay>	 <pixldev> [1/2] oh it worked
[02:12:23] <MirahezeRelay>	 <pixldev> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1280712032169099377/pAk92Uc.png?ex=66d91386&is=66d7c206&hm=eebb4b0b282ae831b3edcc74c4549c8fec02d91e42f460424b0cd995c02aa872&
[02:12:41] <MirahezeRelay>	 <pixldev> ironically the delete wiki request isn't in the same way
[02:12:45] <MacFan4000>	 https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L3293
[02:12:45] <MirahezeRelay>	 <pixldev> fuckin weird
[02:12:52] <MirahezeRelay>	 <pixldev> ill make a pr tmr maybe
[02:12:52] <MacFan4000>	 already is assigned
[02:13:22] <MirahezeRelay>	 <pixldev> why did we do this
[02:13:34] <MirahezeRelay>	 <pixldev> whahttps://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L3240
[02:13:59] <MirahezeRelay>	 <pixldev> reading this makes me cry
[02:14:33] <MirahezeRelay>	 <pixldev> oh fuck ut
[02:15:37] <MirahezeRelay>	 <pixldev> i almost thought it didnt work
[02:15:39] <MirahezeRelay>	 <pixldev> well
[02:15:51] <MirahezeRelay>	 <pixldev> it wont if you dont give yourself the right PIXL
[02:16:12] <MirahezeRelay>	 <pixldev> or also good
[02:16:17] <MirahezeRelay>	 <pixldev> the right to give yourself the right
[02:17:20] <MirahezeRelay>	 <pixldev> [1/2] okay cool
[02:17:20] <MirahezeRelay>	 <pixldev> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1280713279408439308/5Xe5KYO.png?ex=66d914b0&is=66d7c330&hm=64a7f209069f59b7738f4c65d254f019681632a3a1c38c2872302a4c8443de3d&
[02:18:01] <MirahezeRelay>	 <pixldev> MacFan4000: is the box checked f youhttps://meta.mirabeta.org/wiki/Special:RequestWikiQueue/21 ?
[02:22:56] <MirahezeRelay>	 <pixldev> anyways i eep now bye bte world
[02:23:35] <botifico>	 [02mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cf7ab9f07d55...66d1da5107c5
[02:23:38] <botifico>	 [02mw-config] 07MacFan4000 0366d1da5 - add handle-ssl-requests and view-private-ssl-requests and request-ssl to wgManageWikiPermissionsDisallowedRights
[02:23:48] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[02:23:49] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[02:23:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:23:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:24:29] <miraheze-github>	 miraheze/mw-config - MacFan4000 the build passed.
[03:02:57] <icinga-miraheze>	 PROBLEM - mon181 Backups Grafana on mon181 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864155 seconds old and 93 bytes
[03:13:33] <icinga-miraheze>	 PROBLEM - wonderfuleveryday.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wonderfuleveryday.org' expires in 15 day(s) (Fri 20 Sep 2024 03:07:54 AM GMT +0000).
[03:13:48] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/754debc7c884...1b73d17ec33d
[03:13:50] <botifico>	 [02ssl] 07WikiTideSSLBot 031b73d17 - Bot: Update SSL cert for wonderfuleveryday.org
[03:52:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:11:37] <icinga-miraheze>	 RECOVERY - wonderfuleveryday.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wonderfuleveryday.org' will expire on Tue 03 Dec 2024 02:15:11 AM GMT +0000.
[04:31:40] <icinga-miraheze>	 PROBLEM - irad.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[05:39:08] <icinga-miraheze>	 PROBLEM - swiftobject161 NTP time on swiftobject161 is CRITICAL: NTP CRITICAL: No response from NTP server
[05:39:33] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable
[05:39:59] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTP on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable
[05:40:37] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: CRITICAL - Host Unreachable (2407:3641:2161:9774::1)
[05:41:23] <icinga-miraheze>	 PROBLEM - swiftobject161 NTP time on swiftobject161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:42:02] <icinga-miraheze>	 PROBLEM - swiftobject161 PowerDNS Recursor on swiftobject161 is CRITICAL: Domain 'wikitide.net' was not found by the server
[05:42:28] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 162.14 ms
[05:43:19] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 2 backends are down. mw152 mw162
[05:43:25] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 2 backends are down. mw152 mw162
[05:43:57] <icinga-miraheze>	 PROBLEM - mw152 NTP time on mw152 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:44:10] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw152 mw162
[05:44:30] <icinga-miraheze>	 PROBLEM - bots171 NTP time on bots171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:44:33] <icinga-miraheze>	 PROBLEM - mw162 NTP time on mw162 is CRITICAL: NTP CRITICAL: No response from NTP server
[05:44:36] <icinga-miraheze>	 PROBLEM - mw152 PowerDNS Recursor on mw152 is CRITICAL: Domain 'wikitide.net' was not found by the server
[05:44:38] <icinga-miraheze>	 PROBLEM - swiftproxy171 NTP time on swiftproxy171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:44:39] <icinga-miraheze>	 PROBLEM - swiftobject151 NTP time on swiftobject151 is CRITICAL: NTP CRITICAL: No response from NTP server
[05:44:41] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw152 mw162
[05:44:58] <icinga-miraheze>	 PROBLEM - os161 NTP time on os161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:44:59] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw152 mw162
[05:45:28] <icinga-miraheze>	 PROBLEM - os162 NTP time on os162 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:45:58] <icinga-miraheze>	 PROBLEM - os162 PowerDNS Recursor on os162 is CRITICAL: No response from DNS ::1
[05:45:59] <icinga-miraheze>	 PROBLEM - mw162 PowerDNS Recursor on mw162 is CRITICAL: Domain 'wikitide.net' was not found by the server
[05:46:14] <icinga-miraheze>	 PROBLEM - os161 PowerDNS Recursor on os161 is CRITICAL: No response from DNS ::1
[05:46:22] <icinga-miraheze>	 PROBLEM - swiftobject151 PowerDNS Recursor on swiftobject151 is CRITICAL: No response from DNS ::1
[05:46:45] <icinga-miraheze>	 PROBLEM - swiftproxy171 NTP time on swiftproxy171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:46:46] <icinga-miraheze>	 PROBLEM - mw162 NTP time on mw162 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:46:53] <icinga-miraheze>	 PROBLEM - swiftobject151 NTP time on swiftobject151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:50:53] <icinga-miraheze>	 PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[05:58:38] <icinga-miraheze>	 PROBLEM - graylog161 NTP time on graylog161 is CRITICAL: NTP CRITICAL: No response from NTP server
[05:59:16] <icinga-miraheze>	 PROBLEM - swiftobject161 Puppet on swiftobject161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:00:40] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:00:42] <icinga-miraheze>	 PROBLEM - graylog161 NTP time on graylog161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[06:00:55] <icinga-miraheze>	 PROBLEM - graylog161 PowerDNS Recursor on graylog161 is CRITICAL: Domain 'wikitide.net' was not found by the server
[06:01:21] <icinga-miraheze>	 PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:01:36] <RhinosF1>	 Erm
[06:02:14] <RhinosF1>	 MacFan4000, TheresNoTime: ping
[06:02:41] <icinga-miraheze>	 PROBLEM - graylog161 Puppet on graylog161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:03:39] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:04:32] <icinga-miraheze>	 PROBLEM - bots171 IRC-Discord Relay Bot relaybot2 on bots171 is CRITICAL: PROCS CRITICAL: 1 process with args 'relaybot2/'
[06:06:37] <RhinosF1>	 MacFan4000, TheresNoTime: incident comms on discord - bridge down
[06:08:47] <icinga-miraheze>	 PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:10:15] <icinga-miraheze>	 RECOVERY - bots171 IRC-Discord Relay Bot relaybot2 on bots171 is OK: PROCS OK: 2 processes with args 'relaybot2/'
[06:10:59] <icinga-miraheze>	 PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:11:03] <BlankEclair>	 cool, any of them gonna connect?
[06:12:55] <BlankEclair>	 imagine if icinga-miraheze disconnects
[06:17:22] <icinga-miraheze>	 PROBLEM - os162 Puppet on os162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:23:36] <icinga-miraheze>	 PROBLEM - bots171 IRC-Discord Relay Bot relaybot2 on bots171 is CRITICAL: PROCS CRITICAL: 1 process with args 'relaybot2/'
[06:25:41] <RhinosF1>	 BlankEclair: something is screwy
[06:25:44] <RhinosF1>	 No idea what
[06:26:01] <RhinosF1>	 It seems to be internal only
[06:26:09] <RhinosF1>	 Cause external dns is working fine
[06:27:25] <icinga-miraheze>	 RECOVERY - bots171 IRC-Discord Relay Bot relaybot2 on bots171 is OK: PROCS OK: 2 processes with args 'relaybot2/'
[06:27:48] <icinga-miraheze>	 PROBLEM - bots171 IRC-Discord Relay Bot relaybot on bots171 is CRITICAL: PROCS CRITICAL: 1 process with args 'relaybot/'
[06:29:46] <icinga-miraheze>	 RECOVERY - bots171 IRC-Discord Relay Bot relaybot on bots171 is OK: PROCS OK: 2 processes with args 'relaybot/'
[06:30:34] <RhinosF1>	 You need a crash loop back off icinga-miraheze
[06:38:34] <icinga-miraheze>	 PROBLEM - bots171 IRC-Discord Relay Bot relaybot on bots171 is CRITICAL: PROCS CRITICAL: 1 process with args 'relaybot/'
[06:42:34] <icinga-miraheze>	 RECOVERY - bots171 IRC-Discord Relay Bot relaybot on bots171 is OK: PROCS OK: 2 processes with args 'relaybot/'
[06:51:59] <icinga-miraheze>	 PROBLEM - bots171 NTP time on bots171 is CRITICAL: NTP CRITICAL: Offset 1.06835404 secs
[06:52:47] <MirahezeLSBot>	 !log [void@bots171] reboot bots171
[06:52:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:53:35] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[06:54:02] <icinga-miraheze>	 RECOVERY - bots171 NTP time on bots171 is OK: NTP OK: Offset -5.057454109e-05 secs
[06:54:03] <icinga-miraheze>	 PROBLEM - swiftobject151 NTP time on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: Connection refusedconnect to host 10.0.15.117 port 5666: Connection refused
[06:54:55] <MirahezeLSBot>	 !log [void@cloud15] reboot swiftobject151 and os162
[06:55:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:56:13] <icinga-miraheze>	 PROBLEM - swiftobject151 NTP time on swiftobject151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[06:56:25] <icinga-miraheze>	 PROBLEM - graylog161 NTP time on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: Connection refusedconnect to host 10.0.16.130 port 5666: Connection refused
[06:56:47] <icinga-miraheze>	 PROBLEM - swiftproxy171 NTP time on swiftproxy171 is WARNING: NTP WARNING: Offset 0.3668488562 secs
[06:58:19] <icinga-miraheze>	 PROBLEM - graylog161 HTTPS on graylog161 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 113 bytes in 0.016 second response time
[06:58:39] <icinga-miraheze>	 PROBLEM - graylog161 NTP time on graylog161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[06:58:43] <icinga-miraheze>	 PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: connect to address 10.0.16.133 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[06:58:47] <icinga-miraheze>	 RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:59:02] <icinga-miraheze>	 PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw162.wikitide.net port 443 after 4 ms: Couldn't connect to server
[07:00:55] <icinga-miraheze>	 RECOVERY - swiftproxy171 NTP time on swiftproxy171 is OK: NTP OK: Offset -9.036064148e-05 secs
[07:03:21] <icinga-miraheze>	 PROBLEM - db182 NTP time on db182 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:03:21] <icinga-miraheze>	 PROBLEM - db182 MariaDB Connections on db182 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne
[07:03:21] <icinga-miraheze>	  on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66
[07:03:33] <icinga-miraheze>	 PROBLEM - db182 PowerDNS Recursor on db182 is CRITICAL: Domain 'wikitide.net' was not found by the server
[07:03:43] <RhinosF1>	 Oh poop
[07:05:35] <icinga-miraheze>	 PROBLEM - db182 MariaDB on db182 is UNKNOWN: <Timeout exceeded.><Terminated by signal 15 (Terminated).>
[07:07:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:09:58] <icinga-miraheze>	 PROBLEM - mw152 NTP time on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: Connection refusedconnect to host 10.0.15.115 port 5666: Connection refused
[07:10:28] <icinga-miraheze>	 PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:10:34] <icinga-miraheze>	 RECOVERY - mw152 PowerDNS Recursor on mw152 is OK: DNS OK: 0.038 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[07:11:57] <icinga-miraheze>	 RECOVERY - mw152 NTP time on mw152 is OK: NTP OK: Offset -0.0008927285671 secs
[07:12:39] <icinga-miraheze>	 RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:14:37] <icinga-miraheze>	 PROBLEM - bots171 NTP time on bots171 is CRITICAL: NTP CRITICAL: No response from NTP server
[07:15:33] <icinga-miraheze>	 PROBLEM - swiftobject151 NTP time on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: Connection refusedconnect to host 10.0.15.117 port 5666: Connection refused
[07:16:44] <icinga-miraheze>	 PROBLEM - bots171 NTP time on bots171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:17:42] <icinga-miraheze>	 PROBLEM - swiftobject151 NTP time on swiftobject151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:19:16] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:21:15] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0002903342247 secs
[07:28:27] <icinga-miraheze>	 PROBLEM - graylog161 NTP time on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: Connection refusedconnect to host 10.0.16.130 port 5666: Connection refused
[07:31:00] <icinga-miraheze>	 PROBLEM - graylog161 conntrack_table_size on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: No route to hostconnect to host 10.0.16.130 port 5666: No route to host
[07:31:00] <icinga-miraheze>	 PROBLEM - graylog161 Disk Space on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: No route to hostconnect to host 10.0.16.130 port 5666: No route to host
[07:31:35] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:32:00] <icinga-miraheze>	 PROBLEM - Host graylog161 is DOWN: CRITICAL - Host Unreachable (10.0.16.130)
[07:33:34] <icinga-miraheze>	 PROBLEM - bots171 IRC-Discord Relay Bot relaybot on bots171 is CRITICAL: PROCS CRITICAL: 1 process with args 'relaybot/'
[07:35:32] <icinga-miraheze>	 RECOVERY - bots171 IRC-Discord Relay Bot relaybot on bots171 is OK: PROCS OK: 2 processes with args 'relaybot/'
[07:35:55] <icinga-miraheze>	 RECOVERY - Host graylog161 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms
[07:36:19] <icinga-miraheze>	 RECOVERY - graylog161 NTP time on graylog161 is OK: NTP OK: Offset 0.003110170364 secs
[07:36:19] <icinga-miraheze>	 RECOVERY - graylog161 HTTPS on graylog161 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 0.020 second response time
[07:36:49] <icinga-miraheze>	 RECOVERY - graylog161 Disk Space on graylog161 is OK: DISK OK - free space: / 9472MiB (52% inode=93%);
[07:36:49] <icinga-miraheze>	 RECOVERY - graylog161 conntrack_table_size on graylog161 is OK: OK: nf_conntrack is 0 % full
[07:37:04] <icinga-miraheze>	 RECOVERY - graylog161 PowerDNS Recursor on graylog161 is OK: DNS OK: 0.050 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[07:37:09] <icinga-miraheze>	 RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:37:51] <icinga-miraheze>	 PROBLEM - mw162 NTP time on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[07:38:52] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host PROBLEM - mw162 Disk Space on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[07:38:55] <icinga-miraheze>	 PROBLEM - ping on mw162 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.133)
[07:38:59] <icinga-miraheze>	 PROBLEM - Host mw162 is DOWN: CRITICAL - Host Unreachable (10.0.16.133)
[07:43:20] <icinga-miraheze>	 PROBLEM - bots171 IRC-Discord Relay Bot relaybot on bots171 is CRITICAL: PROCS CRITICAL: 1 process with args 'relaybot/'
[07:45:02] <icinga-miraheze>	 RECOVERY - Host mw162 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms
[07:45:02] <icinga-miraheze>	 RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.247 second response time
[07:45:50] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[07:45:55] <icinga-miraheze>	 RECOVERY - mw162 PowerDNS Recursor on mw162 is OK: DNS OK: 0.051 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[07:46:43] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.327 second response time
[07:46:49] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 5.25, 1.72, 0.61
[07:46:49] <icinga-miraheze>	 RECOVERY - mw162 Disk Space on mw162 is OK: DISK OK - free space: / 20672MiB (38% inode=86%);
[07:46:58] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[07:46:59] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy
[07:46:59] <icinga-miraheze>	 RECOVERY - ping on mw162 is OK: PING OK - Packet loss = 0%, RTA = 0.27 ms
[07:47:15] <icinga-miraheze>	 RECOVERY - bots171 IRC-Discord Relay Bot relaybot on bots171 is OK: PROCS OK: 2 processes with args 'relaybot/'
[07:47:19] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[07:47:20] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[07:47:25] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy
[07:47:47] <icinga-miraheze>	 RECOVERY - mw162 NTP time on mw162 is OK: NTP OK: Offset -5.716085434e-05 secs
[07:51:27] <icinga-miraheze>	 PROBLEM - os162 NTP time on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[07:52:36] <icinga-miraheze>	 PROBLEM - os161 NTP time on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host
[07:53:21] <icinga-miraheze>	 PROBLEM - os162 conntrack_table_size on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[07:53:24] <icinga-miraheze>	 PROBLEM - os162 APT on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[07:53:24] <icinga-miraheze>	 PROBLEM - os162 ferm_active on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[07:53:27] <icinga-miraheze>	 PROBLEM - os161 ferm_active on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host
[07:53:27] <icinga-miraheze>	 PROBLEM - os161 APT on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host
[07:53:36] <icinga-miraheze>	 PROBLEM - os161 Current Load on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host
[07:53:36] <icinga-miraheze>	 PROBLEM - os161 conntrack_table_size on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host
[07:53:52] <icinga-miraheze>	 PROBLEM - os162 SSH on os162 is CRITICAL: connect to address 10.0.16.141 and port 22: No route to host
[07:53:52] <icinga-miraheze>	 PROBLEM - os162 Disk Space on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[07:53:57] <icinga-miraheze>	 PROBLEM - ping on os161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.129)
[07:54:00] <icinga-miraheze>	 PROBLEM - Host os161 is DOWN: CRITICAL - Host Unreachable (10.0.16.129)
[07:54:02] <icinga-miraheze>	 PROBLEM - ping on os162 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.141)
[07:54:21] <icinga-miraheze>	 PROBLEM - Host os162 is DOWN: CRITICAL - Host Unreachable (10.0.16.141)
[08:08:00] <icinga-miraheze>	 RECOVERY - Host os161 is UP: PING OK - Packet loss = 0%, RTA = 0.89 ms
[08:08:01] <icinga-miraheze>	 RECOVERY - ping on os161 is OK: PING OK - Packet loss = 0%, RTA = 0.27 ms
[08:08:18] <icinga-miraheze>	 RECOVERY - os161 PowerDNS Recursor on os161 is OK: DNS OK: 0.157 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[08:08:22] <icinga-miraheze>	 RECOVERY - Host os162 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms
[08:09:08] <icinga-miraheze>	 RECOVERY - os162 conntrack_table_size on os162 is OK: OK: nf_conntrack is 0 % full
[08:09:15] <icinga-miraheze>	 RECOVERY - os161 conntrack_table_size on os161 is OK: OK: nf_conntrack is 0 % full
[08:09:17] <icinga-miraheze>	 RECOVERY - os162 NTP time on os162 is OK: NTP OK: Offset -0.0001413524151 secs
[08:09:21] <icinga-miraheze>	 RECOVERY - os161 ferm_active on os161 is OK: OK ferm input default policy is set
[08:09:22] <icinga-miraheze>	 RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[08:09:24] <icinga-miraheze>	 RECOVERY - os161 APT on os161 is OK: APT OK: 66 packages available for upgrade (0 critical updates).
[08:09:27] <icinga-miraheze>	 RECOVERY - os162 APT on os162 is OK: APT OK: 67 packages available for upgrade (0 critical updates).
[08:09:34] <icinga-miraheze>	 RECOVERY - os162 ferm_active on os162 is OK: OK ferm input default policy is set
[08:09:49] <icinga-miraheze>	 RECOVERY - os162 SSH on os162 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[08:09:54] <icinga-miraheze>	 RECOVERY - os162 Disk Space on os162 is OK: DISK OK - free space: / 417090MiB (94% inode=99%);
[08:09:54] <icinga-miraheze>	 RECOVERY - ping on os162 is OK: PING OK - Packet loss = 0%, RTA = 1.03 ms
[08:09:59] <icinga-miraheze>	 RECOVERY - os162 PowerDNS Recursor on os162 is OK: DNS OK: 0.052 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[08:10:25] <icinga-miraheze>	 RECOVERY - os161 NTP time on os161 is OK: NTP OK: Offset -0.002215564251 secs
[08:10:28] <icinga-miraheze>	 RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:13:29] <icinga-miraheze>	 PROBLEM - graylog161 Current Load on graylog161 is WARNING: LOAD WARNING - total load average: 7.64, 5.21, 2.96
[08:13:36] <icinga-miraheze>	 PROBLEM - swiftobject151 NTP time on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host
[08:13:38] <icinga-miraheze>	 PROBLEM - bots171 NTP time on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[08:14:42] <icinga-miraheze>	 PROBLEM - bots171 IRC-Discord Relay Bot relaybot2 on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[08:14:45] <icinga-miraheze>	 PROBLEM - Host bots171 is DOWN: CRITICAL - Host Unreachable (10.0.17.143)
[08:14:46] <icinga-miraheze>	 PROBLEM - swiftobject161 NTP time on swiftobject161 is CRITICAL: connect to address 10.0.16.134 port 5666: No route to hostconnect to host 10.0.16.134 port 5666: No route to host
[08:15:25] <icinga-miraheze>	 PROBLEM - db182 NTP time on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[08:15:43] <icinga-miraheze>	 PROBLEM - swiftobject151 conntrack_table_size on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host
[08:15:54] <icinga-miraheze>	 PROBLEM - swiftobject151 Swift Object Service on swiftobject151 is CRITICAL: connect to address 10.0.15.117 and port 6000: No route to host
[08:16:07] <icinga-miraheze>	 PROBLEM - swiftobject151 Disk Space on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host
[08:16:14] <icinga-miraheze>	 PROBLEM - Host swiftobject151 is DOWN: CRITICAL - Host Unreachable (10.0.15.117)
[08:16:34] <icinga-miraheze>	 PROBLEM - Host swiftobject161 is DOWN: CRITICAL - Host Unreachable (10.0.16.134)
[08:17:19] <icinga-miraheze>	 PROBLEM - ping on db182 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.103)
[08:17:20] <icinga-miraheze>	 PROBLEM - swiftproxy161 HTTP on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable
[08:17:26] <icinga-miraheze>	 PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.011 second response time
[08:17:33] <icinga-miraheze>	 PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.47, 3.33, 1.94
[08:17:43] <icinga-miraheze>	 PROBLEM - Host db182 is DOWN: CRITICAL - Host Unreachable (10.0.18.103)
[08:17:55] <icinga-miraheze>	 PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable
[08:18:00] <icinga-miraheze>	 PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4279 bytes in 2.024 second response time
[08:18:29] <icinga-miraheze>	 PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error
[08:19:29] <icinga-miraheze>	 RECOVERY - graylog161 Current Load on graylog161 is OK: LOAD OK - total load average: 6.20, 6.27, 4.24
[08:19:33] <icinga-miraheze>	 RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 3.37, 3.33, 2.11
[08:28:28] <icinga-miraheze>	 RECOVERY - Host bots171 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms
[08:29:35] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[08:29:41] <icinga-miraheze>	 RECOVERY - bots171 NTP time on bots171 is OK: NTP OK: Offset -4.59253788e-05 secs
[08:30:03] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTP on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.014 second response time
[08:30:14] <icinga-miraheze>	 RECOVERY - bots171 IRC-Discord Relay Bot relaybot2 on bots171 is OK: PROCS OK: 2 processes with args 'relaybot2/'
[08:30:23] <icinga-miraheze>	 RECOVERY - Host swiftobject151 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms
[08:30:28] <icinga-miraheze>	 RECOVERY - Host swiftobject161 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms
[08:30:52] <icinga-miraheze>	 RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[08:30:52] <icinga-miraheze>	 RECOVERY - swiftobject161 NTP time on swiftobject161 is OK: NTP OK: Offset -1.591444016e-05 secs
[08:31:16] <icinga-miraheze>	 RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:31:19] <icinga-miraheze>	 RECOVERY - swiftobject151 NTP time on swiftobject151 is OK: NTP OK: Offset -0.0002345740795 secs
[08:31:20] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time
[08:31:29] <icinga-miraheze>	 RECOVERY - swiftobject151 conntrack_table_size on swiftobject151 is OK: OK: nf_conntrack is 0 % full
[08:31:31] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.023 second response time
[08:31:53] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.040 second response time
[08:31:54] <icinga-miraheze>	 RECOVERY - swiftobject151 Swift Object Service on swiftobject151 is OK: TCP OK - 0.000 second response time on 10.0.15.117 port 6000
[08:31:59] <icinga-miraheze>	 RECOVERY - swiftobject161 PowerDNS Recursor on swiftobject161 is OK: DNS OK: 0.040 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[08:32:04] <icinga-miraheze>	 RECOVERY - swiftobject151 Disk Space on swiftobject151 is OK: DISK OK - free space: / 215338MiB (16% inode=86%);
[08:32:19] <icinga-miraheze>	 RECOVERY - swiftobject151 PowerDNS Recursor on swiftobject151 is OK: DNS OK: 0.045 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[08:40:27] <icinga-miraheze>	 RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.035 second response time
[08:41:42] <icinga-miraheze>	 RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.313 second response time
[08:41:46] <icinga-miraheze>	 RECOVERY - Host db182 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms
[08:41:59] <icinga-miraheze>	 RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19630 bytes in 0.054 second response time
[08:42:19] <icinga-miraheze>	 RECOVERY - db182 MariaDB on db182 is OK: Uptime: 122 Threads: 6 Questions: 83970 Slow queries: 0 Opens: 241 Open tables: 234 Queries per second avg: 688.278
[08:42:27] <icinga-miraheze>	 RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:43:14] <icinga-miraheze>	 RECOVERY - ping on db182 is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms
[08:43:24] <icinga-miraheze>	 RECOVERY - db182 MariaDB Connections on db182 is OK: OK connection usage: 1.2%Current connections: 6
[08:43:44] <icinga-miraheze>	 RECOVERY - db182 PowerDNS Recursor on db182 is OK: DNS OK: 0.041 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[08:43:44] <icinga-miraheze>	 RECOVERY - db182 NTP time on db182 is OK: NTP OK: Offset 6.508827209e-05 secs
[08:52:25] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:35:56] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:37:56] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0005894899368 secs
[10:51:19] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw161
[10:51:20] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw161
[10:51:25] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw161
[10:51:50] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw161
[10:52:59] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw161
[10:53:11] <icinga-miraheze>	 PROBLEM - puppet181 PowerDNS Recursor on puppet181 is CRITICAL: Domain 'wikitide.net' was not found by the server
[10:53:37] <icinga-miraheze>	 PROBLEM - mw161 NTP time on mw161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:53:54] <icinga-miraheze>	 PROBLEM - mw161 PowerDNS Recursor on mw161 is CRITICAL: No response from DNS ::1
[10:53:54] <icinga-miraheze>	 PROBLEM - puppet181 NTP time on puppet181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:54:14] <icinga-miraheze>	 PROBLEM - os161 PowerDNS Recursor on os161 is CRITICAL: No response from DNS ::1
[10:54:28] <icinga-miraheze>	 PROBLEM - os161 NTP time on os161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:55:03] <icinga-miraheze>	 PROBLEM - phorge171 NTP time on phorge171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:55:58] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:56:11] <icinga-miraheze>	 PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:57:17] <icinga-miraheze>	 PROBLEM - swiftobject161 Puppet on swiftobject161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:57:56] <icinga-miraheze>	 PROBLEM - swiftac171 Puppet on swiftac171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:58:22] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:00:06] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:00:40] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:00:42] <icinga-miraheze>	 PROBLEM - mem161 Puppet on mem161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:00:47] <icinga-miraheze>	 PROBLEM - swiftobject181 Puppet on swiftobject181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:01:11] <icinga-miraheze>	 PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:01:35] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:01:40] <icinga-miraheze>	 PROBLEM - graphite151 Puppet on graphite151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:02:00] <icinga-miraheze>	 PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:02:24] <MirahezeRelay>	 <cosmicalpha> This is very bad
[11:02:26] <icinga-miraheze>	 PROBLEM - bast181 Puppet on bast181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:02:58] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:02:59] <MirahezeRelay>	 <bluemoon0332> puppet181 is not responding to even pings
[11:03:05] <MirahezeRelay>	 <bluemoon0332> that's the reason for the puppet failures
[11:03:06] <icinga-miraheze>	 PROBLEM - reports171 Puppet on reports171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:03:15] <icinga-miraheze>	 PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:03:15] <icinga-miraheze>	 PROBLEM - db171 Puppet on db171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:04:18] <MirahezeRelay>	 <cosmicalpha, replying to bluemoon0332> Well mw161 and db182 seems to have failed also as well as Swift. We seem to be in a lot larger issues then just that.
[11:04:24] <MirahezeRelay>	 <cosmicalpha> PowerDNS is failing everywhere
[11:04:26] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet],Exec[git_pull_ssl],Exec[git_pull_mediawiki-repos]
[11:04:35] <icinga-miraheze>	 PROBLEM - rdb151 Puppet on rdb151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:04:41] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:04:41] <icinga-miraheze>	 PROBLEM - graylog161 Puppet on graylog161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:04:47] <icinga-miraheze>	 PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:04:54] <icinga-miraheze>	 PROBLEM - db161 Puppet on db161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:05:10] <icinga-miraheze>	 PROBLEM - eventgate181 Puppet on eventgate181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:05:16] <MirahezeRelay>	 <cosmicalpha> @bluemoon0332 try rebooting cloud18 (stop all VMs first though)
[11:05:27] <icinga-miraheze>	 PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:05:34] <icinga-miraheze>	 PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:05:51] <icinga-miraheze>	 PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:07:00] <icinga-miraheze>	 PROBLEM - mem151 Puppet on mem151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:07:04] <icinga-miraheze>	 PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:08:28] <icinga-miraheze>	 PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:08:35] <icinga-miraheze>	 PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:08:38] <icinga-miraheze>	 PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:09:00] <MirahezeLSBot>	 !log [alex@cloud18.wikitide.net] stopped puppet181
[11:09:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:09:15] <icinga-miraheze>	 PROBLEM - puppet181 NTP time on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[11:09:22] <MirahezeRelay>	 <bluemoon0332> if I shutdown bast181 I'm going to be disconnected, I have to change bastions first
[11:09:46] <icinga-miraheze>	 PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:10:21] <icinga-miraheze>	 PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:10:29] <MirahezeRelay>	 <cosmicalpha, replying to bluemoon0332> oh right
[11:10:47] <icinga-miraheze>	 PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[11:11:12] <icinga-miraheze>	 PROBLEM - puppet181 conntrack_table_size on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[11:11:22] <icinga-miraheze>	 PROBLEM - ping on puppet181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.100)
[11:11:22] <icinga-miraheze>	 PROBLEM - puppet181 ferm_active on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[11:11:22] <icinga-miraheze>	 PROBLEM - puppet181 Current Load on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[11:11:25] <icinga-miraheze>	 PROBLEM - puppet181 Backups Private on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: No route to hostconnect to host 10.0.18.100 port 5666: No route to host
[11:11:45] <icinga-miraheze>	 PROBLEM - Host puppet181 is DOWN: CRITICAL - Host Unreachable (10.0.18.100)
[11:12:15] <MirahezeLSBot>	 !log [alex@cloud18.wikitide.net] stopped the rest of vms
[11:12:31] <MirahezeRelay>	 <bluemoon0332> !log rebooted cloud18
[11:12:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:13:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:13:18] <MirahezeRelay>	 <cosmicalpha> @bluemoon0332 most servers wont boot back up immediately btw, so will have to be manually started.
[11:16:37] <MirahezeRelay>	 <agentisai> oh nice, outage
[11:16:43] <BlankEclair>	 > [04/09/2024 16:12] <BlankEclair> imagine if icinga-miraheze disconnects
[11:16:51] <BlankEclair>	 icinga-miraheze i was joking please reconnect
[11:17:57] <MirahezeRelay>	 <bluemoon0332> bringing the vms back up now
[11:18:02] <BlankEclair>	 oh okay
[11:19:10] <MirahezeLSBot>	 !log [alex@cloud18.wikitide.net] brought all vms back up
[11:19:32] <BlankEclair>	 i wonder, what does LS in MirahezeLSBot stand for? log sudo?
[11:19:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:19:51] <MirahezeRelay>	 <bluemoon0332> I think it stands for Log Server?
[11:19:51] <MirahezeRelay>	 <agentisai> Log Server?
[11:19:53] <MirahezeRelay>	 <agentisai> yeah
[11:19:54] <MirahezeRelay>	 <bluemoon0332> I don't really know
[11:24:04] <MirahezeLSBot>	 !log [alex@cloud17.wikitide.net] restarted phorge171
[11:24:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:27:17] <MirahezeRelay>	 <cosmicalpha, replying to agentisai> Yes PowerDNS decided it was tired and should go to sleep, kinda like I do sometimes.
[11:27:27] <MirahezeRelay>	 <agentisai> lol
[11:28:21] <MirahezeRelay>	 <agentisai> it's always amazing how stuff we're going to deprecate ends up breaking right before we migrate
[11:28:32] <MirahezeRelay>	 <cosmicalpha> Yep lol
[11:28:39] <MirahezeRelay>	 <cosmicalpha> Its like it knows
[11:28:45] <MirahezeRelay>	 <cosmicalpha> and goes out with a bang
[11:28:46] <MirahezeRelay>	 <agentisai> this reminds me of when db131 broke and took out the whole farm
[11:28:52] <MirahezeRelay>	 <agentisai, replying to cosmicalpha> heh yes
[11:29:04] <MirahezeRelay>	 <bluemoon0332> I think networking in cloud17 is messed up
[11:29:14] <MirahezeRelay>	 <cosmicalpha> Reboot cloud17
[11:29:53] <MirahezeRelay>	 <cosmicalpha> We are already down quite a bit rebooting cloud17 is fine right now.
[11:30:52] <BlankEclair>	 we aren't deprecating powerdns though, right?
[11:31:24] <MirahezeRelay>	 <bluemoon0332> say goodbye to the relay everyone
[11:31:29] <MirahezeRelay>	 <agentisai> RIP
[11:31:34] <BlankEclair>	 noooo
[11:33:39] <BlankEclair>	 yay pixl!
[11:33:48] <PixDeVl>	 I GO TO SLEEP
[11:33:50] <Agent>	 1984
[11:33:51] <PixDeVl>	 AND WHEN I WAKE UP
[11:33:55] <BlankEclair>	 before you do
[11:33:57] <PixDeVl>	 EVERYTHING IS ON FUCKING FIRE
[11:34:03] <BlankEclair>	 i want to let you know that i filed a bug in firefox
[11:34:04] <BlankEclair>	 https://bugzilla.mozilla.org/show_bug.cgi?id=1916642
[11:34:11] <PixDeVl>	 oh congrats!!!
[11:34:14] <PixDeVl>	 anyways
[11:34:29] <PixDeVl>	 PHORGE. META. DB171 APPARENTLY. EVEN GRAFANA
[11:34:48] <PixDeVl>	 was that relay
[11:34:50] <BlankEclair>	 yes
[11:34:53] <PixDeVl>	 no
[11:34:54] <BlankEclair>	 > [04/09/2024 21:31] <MirahezeRelay> <bluemoon0332> say goodbye to the relay everyone
[11:34:57] <BlankEclair>	 oh
[11:34:58] <BlankEclair>	 not yet
[11:35:02] <BlankEclair>	 MirahezeRelay: still there with us?
[11:35:07] <PixDeVl>	 WHAT IS GOING ON
[11:35:12] <Agent>	 powerdns ded
[11:35:22] <Agent>	 so a good portion of the farm died with it
[11:35:23] <BlankEclair>	 and entire fucking server ded
[11:35:32] <BlankEclair>	 so we're rebooting the cloud servers
[11:35:39] <BlankEclair>	 (i think, i spent most of the day debugging firefox...)
[11:36:01] <PixDeVl>	 Is it too much to assume it was expected downtime
[11:36:03] <PixDeVl>	 NOOOOOOOOO
[11:36:08] <Agent>	 BlankEclair: to answer your question, we were planning to deprecate wikitide.net being resolvable internally by PowerDNS rather than doing an external query
[11:36:53] <PixDeVl>	 Is powerdns the new one? I thought we were on gdnsn and moving to BIND
[11:37:35] <BlankEclair>	 powerdns is the recursive resolver on the servers
[11:37:38] <Agent>	 PowerDNS would catch queries for wikitide.net and internally route then to gdnsd 
[11:37:43] <BlankEclair>	 uh
[11:37:46] <BlankEclair>	 huh
[11:38:00] <PixDeVl>	 so completely unrelated
[11:38:03] <BlankEclair>	 i thought that powerdns is basically a recursive resolver on the servers
[11:38:21] <BlankEclair>	 yeah i don't think it's the authoritative nameservers
[11:38:49] <PixDeVl>	 is this because i overslet
[11:39:03] <BlankEclair>	 lmao
[11:39:23] <BlankEclair>	 you oversleeping caused me to come across three bugs in firefox and the entire dns setup in miraheze to come crashing down
[11:41:00] <BlankEclair>	 firefox networking code is hard to understand...
[12:22:29] <Guest10>	 Hi, I'd like to report a possible issue? It seems some pictures on our wiki are not loading. For instance: https://bgo.miraheze.org/wiki/File:TheDragon.png
[12:27:35] <BlankEclair>	 Guest10: the servers are being rebooted as there is a problem with the servers (dns died, maybe bad networking? i haven't really paid attention to it myself as i'm kinda busy, and i don't have any privileges to fix it anyway)
[12:27:57] <BlankEclair>	 i dunnot the status... normally the irc <-> discord bridge and icinga would give info here, but they're both down
[12:32:55] <Guest10>	 Okay, thanks! :)
[14:19:12] <MacFan4000>	 I posted notices
[14:19:24] <MacFan4000>	 (I know nobody can hear me right now)
[14:44:53] <MirahezeRelay>	 <bluemoon0332> I restarted cloud17 and it got IPv6 back
[14:45:12] <MirahezeRelay>	 <bluemoon0332> also welcome back relaybot
[14:47:41] <MirahezeRelay>	 <bluemoon0332> also welcome back Phorge
[14:48:28] <MirahezeRelay>	 <bluemoon0332> vms in cloud16 and cloud18 are also acting up
[14:56:04] <MacFan4000>	 Yay relay is back
[14:57:21] <MirahezeRelay>	 <bluemoon0332> yeah, not quite out of the woods just yet
[15:02:01] <MirahezeRelay>	 <bluemoon0332> FiberState said they'd investigate the issue, but I hope just restarting those two allows us to put this bizarre issue behind
[15:02:59] <MirahezeRelay>	 <bluemoon0332> As for the VMs affected there, puppet181, mw161 and mw181 are still not reachable over the network
[15:03:52] <MirahezeRelay>	 <bluemoon0332> `s/mw181/mw182`
[15:08:01] <MirahezeRelay>	 <bluemoon0332> actually, we can add as well bast181 to that list
[15:09:41] <MirahezeLSBot>	 !log [alex@cloud18.wikitide.net] stopped all VMs
[15:10:05] <MirahezeRelay>	 <bluemoon0332> !log rebooted cloud18
[15:10:19] <PiscesXYZ>	 anyone available in Phorge?
[15:10:19] <PiscesXYZ>	 I need to change my username there to Pisces
[15:10:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:10:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:13:27] <MacFan4000>	 PiscesXYZ: the phorge admins would be busy with the current incident
[15:14:05] <MacFan4000>	 I would open a phorge ticket and then they’ll handle when they have time
[15:14:15] <PiscesXYZ>	 MacFan4000: I forgot about it, tks for lmk tho
[15:16:42] <MirahezeLSBot>	 !log [alex@cloud18.wikitide.net] brought up all vms
[15:17:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:17:57] <icinga-miraheze>	 RECOVERY - wiki.secondrenaissance.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.secondrenaissance.net' will expire on Thu 28 Nov 2024 01:41:45 PM GMT +0000.
[15:17:57] <icinga-miraheze>	 RECOVERY - 321nailswiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:17:57] <icinga-miraheze>	 RECOVERY - unhingedwiki.paralleldv.org - Cloudflare on sslhost is OK: OK - Certificate 'unhingedwiki.paralleldv.org' will expire on Thu 28 Nov 2024 01:36:49 PM GMT +0000.
[15:17:59] <MirahezeRelay>	 <bluemoon0332> puppet181 is back
[15:18:05] <icinga-miraheze>	 RECOVERY - wiki.kscucf.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.kscucf.org' will expire on Thu 28 Nov 2024 12:48:30 PM GMT +0000.
[15:18:08] <icinga-miraheze>	 RECOVERY - kodiak.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kodiak.wiki' will expire on Mon 11 Nov 2024 04:12:08 AM GMT +0000.
[15:18:17] <icinga-miraheze>	 RECOVERY - removededm.com - LetsEncrypt on sslhost is OK: OK - Certificate 'removededm.com' will expire on Mon 25 Nov 2024 01:47:38 PM GMT +0000.
[15:18:18] <icinga-miraheze>	 RECOVERY - fanojo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fanojo.wiki' will expire on Tue 22 Oct 2024 01:41:50 AM GMT +0000.
[15:18:18] <icinga-miraheze>	 RECOVERY - wiki.jill-jimmy.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.jill-jimmy.com' will expire on Thu 28 Nov 2024 01:39:01 PM GMT +0000.
[15:18:19] <icinga-miraheze>	 RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Fri 11 Oct 2024 01:04:25 PM GMT +0000.
[15:18:19] <icinga-miraheze>	 RECOVERY - wiki.solarconquest.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.solarconquest.net' will expire on Thu 28 Nov 2024 01:41:55 PM GMT +0000.
[15:18:24] <icinga-miraheze>	 RECOVERY - alternatewiki.tombricks.com - Cloudflare on sslhost is OK: OK - Certificate 'alternatewiki.tombricks.com' will expire on Thu 28 Nov 2024 01:24:51 PM GMT +0000.
[15:18:25] <icinga-miraheze>	 RECOVERY - orain.org - Cloudflare on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 29 Nov 2024 04:24:26 PM GMT +0000.
[15:18:26] <icinga-miraheze>	 RECOVERY - wiki.widwwa.co.uk - Cloudflare on sslhost is OK: OK - Certificate 'wiki.widwwa.co.uk' will expire on Thu 28 Nov 2024 01:42:51 PM GMT +0000.
[15:18:27] <icinga-miraheze>	 RECOVERY - xomnipedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'xomnipedia.org' will expire on Thu 17 Oct 2024 09:33:16 AM GMT +0000.
[15:18:28] <icinga-miraheze>	 RECOVERY - beta.sagan4.org - Cloudflare on sslhost is OK: OK - Certificate 'beta.sagan4.org' will expire on Thu 28 Nov 2024 01:25:01 PM GMT +0000.
[15:18:29] <icinga-miraheze>	 RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'warframestat.us' will expire on Tue 19 Nov 2024 12:19:13 AM GMT +0000.
[15:18:31] <icinga-miraheze>	 RECOVERY - wiki.potatotransit.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.potatotransit.xyz' will expire on Thu 28 Nov 2024 01:41:30 PM GMT +0000.
[15:18:34] <icinga-miraheze>	 RECOVERY - wikimoma.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimoma.art' will expire on Thu 10 Oct 2024 10:39:49 AM GMT +0000.
[15:18:36] <icinga-miraheze>	 RECOVERY - cp27 Nginx Backend for swiftproxy161 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8206
[15:18:37] <icinga-miraheze>	 RECOVERY - wiki.themanaworld.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.themanaworld.org' will expire on Wed 27 Nov 2024 04:29:56 PM GMT +0000.
[15:18:38] <icinga-miraheze>	 RECOVERY - reviwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Sat 16 Nov 2024 01:16:18 PM GMT +0000.
[15:18:39] <icinga-miraheze>	 RECOVERY - kafka181 PowerDNS Recursor on kafka181 is OK: DNS OK: 0.078 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[15:18:39] <icinga-miraheze>	 RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[15:18:40] <icinga-miraheze>	 RECOVERY - cities.simulz.kr - Cloudflare on sslhost is OK: OK - Certificate 'cities.simulz.kr' will expire on Thu 28 Nov 2024 01:25:13 PM GMT +0000.
[15:18:46] <icinga-miraheze>	 RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 0%
[15:18:47] <icinga-miraheze>	 RECOVERY - cp27 Nginx Backend for mwtask171 on cp27 is OK: TCP OK - 0.000 second response time on localhost port 8161
[15:18:48] <icinga-miraheze>	 RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[15:18:48] <icinga-miraheze>	 RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[15:18:48] <icinga-miraheze>	 RECOVERY - wiki.qadrishattari.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.qadrishattari.xyz' will expire on Thu 28 Nov 2024 01:41:35 PM GMT +0000.
[15:18:50] <icinga-miraheze>	 RECOVERY - coffeewiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'coffeewiki.net' will expire on Sun 24 Nov 2024 10:57:49 PM GMT +0000.
[15:18:51] <icinga-miraheze>	 RECOVERY - worldtriggerwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'worldtriggerwiki.com' will expire on Fri 18 Oct 2024 01:12:55 AM GMT +0000.
[15:18:51] <MirahezeRelay>	 <bluemoon0332> and mon181, apparently :p
[15:18:51] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms
[15:18:52] <icinga-miraheze>	 RECOVERY - cp26 NTP time on cp26 is OK: NTP OK: Offset 0.002093136311 secs
[15:18:52] <icinga-miraheze>	 RECOVERY - ns2 ferm_active on ns2 is OK: OK ferm input default policy is set
[15:18:53] <icinga-miraheze>	 RECOVERY - wiki.meeusen.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Fri 29 Nov 2024 07:58:03 PM GMT +0000.
[15:18:53] <icinga-miraheze>	 RECOVERY - terraforming.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'terraforming.wiki' will expire on Wed 25 Sep 2024 08:31:50 PM GMT +0000.
[15:18:54] <icinga-miraheze>	 RECOVERY - gimkit.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:18:54] <icinga-miraheze>	 RECOVERY - wiki.sequoiafabrica.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.sequoiafabrica.org' will expire on Thu 28 Nov 2024 01:41:45 PM GMT +0000.
[15:18:55] <icinga-miraheze>	 RECOVERY - pokemonwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'pokemonwiki.info' will expire on Sun 20 Oct 2024 04:47:56 PM GMT +0000.
[15:18:56] <icinga-miraheze>	 RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[15:18:56] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[15:18:57] <icinga-miraheze>	 RECOVERY - cp26 Nginx Backend for mw151 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8113
[15:18:57] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.23 ms
[15:18:59] <icinga-miraheze>	 RECOVERY - db181 NTP time on db181 is OK: NTP OK: Offset -8.252263069e-05 secs
[15:19:00] <icinga-miraheze>	 RECOVERY - wiki.danvs.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.danvs.net' will expire on Thu 28 Nov 2024 12:47:17 PM GMT +0000.
[15:19:01] <icinga-miraheze>	 RECOVERY - eventgate181 Puppet on eventgate181 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[15:19:01] <icinga-miraheze>	 RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.orain.org' will expire on Fri 29 Nov 2024 04:30:28 PM GMT +0000.
[15:19:01] <icinga-miraheze>	 RECOVERY - rainverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rainverse.wiki' will expire on Fri 22 Nov 2024 01:56:56 AM GMT +0000.
[15:19:01] <icinga-miraheze>	 RECOVERY - db182 NTP time on db182 is OK: NTP OK: Offset -0.0001636743546 secs
[15:19:13] <icinga-miraheze>	 RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:19:13] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[15:19:17] <icinga-miraheze>	 RECOVERY - www.mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mwcosmos.com' will expire on Fri 29 Nov 2024 05:06:15 PM GMT +0000.
[15:19:19] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[15:19:21] <icinga-miraheze>	 RECOVERY - wiki.giantesselsie.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.giantesselsie.net' will expire on Thu 28 Nov 2024 01:38:42 PM GMT +0000.
[15:19:21] <icinga-miraheze>	 RECOVERY - wiki.sadboyzpod.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.sadboyzpod.com' will expire on Thu 28 Nov 2024 01:41:42 PM GMT +0000.
[15:19:21] <icinga-miraheze>	 PROBLEM - cloud17 Puppet on cloud17 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[15:19:23] <icinga-miraheze>	 PROBLEM - graylog161 Current Load on graylog161 is WARNING: LOAD WARNING - total load average: 7.62, 4.72, 3.07
[15:19:24] <icinga-miraheze>	 RECOVERY - mon181 NTP time on mon181 is OK: NTP OK: Offset -6.774067879e-05 secs
[15:19:25] <icinga-miraheze>	 RECOVERY - wiki.maribelhearn.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.maribelhearn.com' will expire on Thu 28 Nov 2024 01:39:21 PM GMT +0000.
[15:19:26] <icinga-miraheze>	 RECOVERY - encyclopediarobotica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediarobotica.org' will expire on Tue 26 Nov 2024 03:12:49 PM GMT +0000.
[15:19:26] <icinga-miraheze>	 RECOVERY - mario.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:19:26] <icinga-miraheze>	 RECOVERY - wiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:19:29] <icinga-miraheze>	 RECOVERY - wiki.mobilityengineer.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Fri 29 Nov 2024 07:58:05 PM GMT +0000.
[15:19:31] <icinga-miraheze>	 RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:19:32] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:19:33] <icinga-miraheze>	 RECOVERY - robloxapi.wiki - Cloudflare on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Thu 28 Nov 2024 01:58:02 PM GMT +0000.
[15:19:35] <icinga-miraheze>	 RECOVERY - familiacorsi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'familiacorsi.com' will expire on Tue 22 Oct 2024 05:16:02 PM GMT +0000.
[15:19:35] <icinga-miraheze>	 RECOVERY - wiki.altilunium.my.id - Cloudflare on sslhost is OK: OK - Certificate 'wiki.altilunium.my.id' will expire on Thu 28 Nov 2024 01:37:49 PM GMT +0000.
[15:19:36] <icinga-miraheze>	 RECOVERY - squareenix.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:19:37] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:19:41] <MirahezeRelay>	 <bluemoon0332> moving to cloud16 now
[15:19:41] <icinga-miraheze>	 RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'thesimswiki.com' will expire on Tue 03 Dec 2024 05:30:24 AM GMT +0000.
[15:19:44] <icinga-miraheze>	 RECOVERY - beidipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beidipedia.com' will expire on Sat 09 Nov 2024 03:43:42 PM GMT +0000.
[15:19:45] <icinga-miraheze>	 RECOVERY - pubwiki.lab.co.pl - Cloudflare on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Thu 28 Nov 2024 01:32:46 PM GMT +0000.
[15:19:45] <icinga-miraheze>	 RECOVERY - wiki.buryland.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.buryland.net' will expire on Thu 28 Nov 2024 01:38:01 PM GMT +0000.
[15:19:45] <icinga-miraheze>	 RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Mon 21 Oct 2024 03:28:07 AM GMT +0000.
[15:19:46] <icinga-miraheze>	 RECOVERY - bluearchive.wiki - Cloudflare on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Thu 28 Nov 2024 01:59:07 PM GMT +0000.
[15:19:46] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sun 20 Oct 2024 04:53:18 PM GMT +0000.
[15:19:47] <icinga-miraheze>	 RECOVERY - lab.wikimimo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimimo.com' will expire on Tue 08 Oct 2024 10:24:23 AM GMT +0000.
[15:19:51] <icinga-miraheze>	 RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.dariawiki.org' will expire on Thu 28 Nov 2024 04:21:50 PM GMT +0000.
[15:19:54] <icinga-miraheze>	 RECOVERY - wiki.junkstore.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.junkstore.xyz' will expire on Thu 28 Nov 2024 01:39:05 PM GMT +0000.
[15:19:56] <icinga-miraheze>	 RECOVERY - clubpenguinfanon.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'clubpenguinfanon.wiki' will expire on Mon 28 Oct 2024 12:43:18 AM GMT +0000.
[15:19:58] <icinga-miraheze>	 RECOVERY - polcompball.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:19:59] <icinga-miraheze>	 RECOVERY - wiki.tulpa.info - Cloudflare on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Thu 28 Nov 2024 01:42:45 PM GMT +0000.
[15:20:01] <icinga-miraheze>	 RECOVERY - wiki.esufranchise.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.esufranchise.com' will expire on Thu 28 Nov 2024 01:38:35 PM GMT +0000.
[15:20:02] <icinga-miraheze>	 RECOVERY - wiki.mostly.vet - LetsEncrypt on sslhost is OK: OK - Certificate 'mostly.vet' will expire on Fri 18 Oct 2024 12:05:32 AM GMT +0000.
[15:20:02] <icinga-miraheze>	 RECOVERY - commonswiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:20:03] <icinga-miraheze>	 RECOVERY - crustypedia.org - Cloudflare on sslhost is OK: OK - Certificate 'crustypedia.org' will expire on Thu 28 Nov 2024 01:25:18 PM GMT +0000.
[15:20:04] <icinga-miraheze>	 RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Thu 10 Oct 2024 12:53:31 AM GMT +0000.
[15:20:06] <icinga-miraheze>	 RECOVERY - wiki.simorgh.me - Cloudflare on sslhost is OK: OK - Certificate 'wiki.simorgh.me' will expire on Thu 28 Nov 2024 01:41:49 PM GMT +0000.
[15:20:07] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0002547204494 secs
[15:20:08] <icinga-miraheze>	 RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'poserdazfreebies.orain.org' will expire on Fri 29 Nov 2024 04:24:24 PM GMT +0000.
[15:20:09] <icinga-miraheze>	 RECOVERY - fid.koymi.net - Cloudflare on sslhost is OK: OK - Certificate 'fid.koymi.net' will expire on Thu 28 Nov 2024 01:50:41 PM GMT +0000.
[15:20:09] <icinga-miraheze>	 RECOVERY - www.rippaversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rippaversewiki.com' will expire on Thu 28 Nov 2024 04:33:43 PM GMT +0000.
[15:20:14] <icinga-miraheze>	 RECOVERY - donkeykong.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:20:15] <icinga-miraheze>	 RECOVERY - www.dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.dcmultiversewiki.com' will expire on Fri 29 Nov 2024 05:08:41 PM GMT +0000.
[15:20:16] <icinga-miraheze>	 RECOVERY - www.allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.allthetropes.org' will expire on Thu 28 Nov 2024 04:23:00 PM GMT +0000.
[15:20:17] <icinga-miraheze>	 RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 27 Sep 2024 11:58:22 PM GMT +0000.
[15:20:18] <icinga-miraheze>	 RECOVERY - wiki.closai.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.closai.com' will expire on Fri 29 Nov 2024 07:57:29 PM GMT +0000.
[15:20:22] <icinga-miraheze>	 RECOVERY - wiki.lesunk.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.lesunk.com' will expire on Thu 28 Nov 2024 01:39:15 PM GMT +0000.
[15:20:23] <icinga-miraheze>	 RECOVERY - wiki.strangereons.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.strangereons.com' will expire on Thu 28 Nov 2024 01:42:00 PM GMT +0000.
[15:20:28] <icinga-miraheze>	 RECOVERY - guia.esporo.net - Cloudflare on sslhost is OK: OK - Certificate 'guia.esporo.net' will expire on Thu 28 Nov 2024 01:30:52 PM GMT +0000.
[15:20:32] <icinga-miraheze>	 RECOVERY - yoshi.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:20:35] <icinga-miraheze>	 RECOVERY - steem.wiki - Cloudflare on sslhost is OK: OK - Certificate 'steem.wiki' will expire on Thu 28 Nov 2024 01:34:25 PM GMT +0000.
[15:20:37] <icinga-miraheze>	 RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.publictestwiki.com' will expire on Fri 29 Nov 2024 07:59:52 PM GMT +0000.
[15:20:39] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.gab.pt.eu.org' will expire on Thu 28 Nov 2024 01:38:42 PM GMT +0000.
[15:20:45] <icinga-miraheze>	 RECOVERY - wiki.yonicstudios.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.yonicstudios.com' will expire on Thu 28 Nov 2024 01:42:58 PM GMT +0000.
[15:20:47] <icinga-miraheze>	 RECOVERY - small.polandballwiki.com - Cloudflare on sslhost is OK: OK - Certificate 'small.polandballwiki.com' will expire on Thu 28 Nov 2024 01:34:19 PM GMT +0000.
[15:20:49] <icinga-miraheze>	 RECOVERY - wiki.creaturathegame.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.creaturathegame.com' will expire on Thu 28 Nov 2024 01:38:15 PM GMT +0000.
[15:20:49] <icinga-miraheze>	 RECOVERY - junxions.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'junxions.wiki' will expire on Mon 07 Oct 2024 04:04:33 PM GMT +0000.
[15:20:50] <icinga-miraheze>	 RECOVERY - 321nails.crpteam.club - Cloudflare on sslhost is OK: OK - Certificate '321nails.crpteam.club' will expire on Thu 28 Nov 2024 02:04:00 PM GMT +0000.
[15:20:54] <icinga-miraheze>	 RECOVERY - archives.nsgov.org - Cloudflare on sslhost is OK: OK - Certificate 'archives.nsgov.org' will expire on Thu 28 Nov 2024 01:24:58 PM GMT +0000.
[15:20:55] <icinga-miraheze>	 RECOVERY - atlas.starworld.zone - Cloudflare on sslhost is OK: OK - Certificate 'atlas.starworld.zone' will expire on Thu 28 Nov 2024 01:24:59 PM GMT +0000.
[15:20:56] <icinga-miraheze>	 RECOVERY - wiki.gtnewhorizons.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.gtnewhorizons.com' will expire on Thu 28 Nov 2024 01:38:49 PM GMT +0000.
[15:20:58] <icinga-miraheze>	 RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:21:00] <icinga-miraheze>	 RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Wed 09 Oct 2024 08:01:13 AM GMT +0000.
[15:21:02] <icinga-miraheze>	 RECOVERY - uncyclopediakr.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:21:03] <icinga-miraheze>	 RECOVERY - amps.wiki.gd - Cloudflare on sslhost is OK: OK - Certificate 'amps.wiki.gd' will expire on Thu 28 Nov 2024 01:24:55 PM GMT +0000.
[15:21:03] <icinga-miraheze>	 RECOVERY - grayzonewarfare.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'grayzonewarfare.wiki' will expire on Sat 09 Nov 2024 08:51:50 PM GMT +0000.
[15:21:06] <icinga-miraheze>	 RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Wed 09 Oct 2024 09:56:34 AM GMT +0000.
[15:21:09] <icinga-miraheze>	 RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'scvo.org' will expire on Fri 29 Nov 2024 07:51:04 AM GMT +0000.
[15:21:10] <icinga-miraheze>	 RECOVERY - islegoblin.wiki - Cloudflare on sslhost is OK: OK - Certificate 'islegoblin.wiki' will expire on Thu 28 Nov 2024 01:31:28 PM GMT +0000.
[15:21:10] <icinga-miraheze>	 RECOVERY - cheeseepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cheeseepedia.org' will expire on Fri 04 Oct 2024 09:40:03 AM GMT +0000.
[15:21:14] <icinga-miraheze>	 RECOVERY - crashspyro.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:21:20] <icinga-miraheze>	 RECOVERY - wiki.lewisgill.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.lewisgill.com' will expire on Thu 28 Nov 2024 01:39:15 PM GMT +0000.
[15:21:21] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Thu 10 Oct 2024 07:27:39 PM GMT +0000.
[15:21:24] <icinga-miraheze>	 RECOVERY - www.dovearchives.wiki - Cloudflare on sslhost is OK: OK - Certificate 'www.dovearchives.wiki' will expire on Thu 28 Nov 2024 01:43:19 PM GMT +0000.
[15:21:25] <icinga-miraheze>	 RECOVERY - dlfm-wiki.top - Cloudflare on sslhost is OK: OK - Certificate 'dlfm-wiki.top' will expire on Thu 28 Nov 2024 02:15:47 PM GMT +0000.
[15:21:26] <icinga-miraheze>	 RECOVERY - swests.org - Cloudflare on sslhost is OK: OK - Certificate 'swests.org' will expire on Thu 28 Nov 2024 01:34:28 PM GMT +0000.
[15:21:27] <icinga-miraheze>	 RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Thu 10 Oct 2024 04:29:46 PM GMT +0000.
[15:21:27] <icinga-miraheze>	 RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Fri 04 Oct 2024 04:16:45 PM GMT +0000.
[15:21:28] <icinga-miraheze>	 RECOVERY - puzzles.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.puzzles.wiki' will expire on Thu 10 Oct 2024 09:52:21 AM GMT +0000.
[15:21:28] <icinga-miraheze>	 RECOVERY - en.religiononfire.mar.in.ua - Cloudflare on sslhost is OK: OK - Certificate 'en.religiononfire.mar.in.ua' will expire on Thu 28 Nov 2024 01:27:10 PM GMT +0000.
[15:21:29] <icinga-miraheze>	 RECOVERY - wiki.corgicam.tv - Cloudflare on sslhost is OK: OK - Certificate 'wiki.corgicam.tv' will expire on Thu 28 Nov 2024 01:38:12 PM GMT +0000.
[15:21:30] <icinga-miraheze>	 RECOVERY - wiki.iurevar.world - LetsEncrypt on sslhost is OK: OK - Certificate 'iurevar.world' will expire on Sat 23 Nov 2024 05:34:13 AM GMT +0000.
[15:21:31] <icinga-miraheze>	 RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'potabi.com' will expire on Thu 17 Oct 2024 04:34:59 PM GMT +0000.
[15:21:38] <icinga-miraheze>	 RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 29 Nov 2024 04:19:24 PM GMT +0000.
[15:21:40] <icinga-miraheze>	 RECOVERY - iol.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'iol.wiki' will expire on Fri 11 Oct 2024 01:25:12 AM GMT +0000.
[15:21:40] <icinga-miraheze>	 RECOVERY - 100acg.wiki - Cloudflare on sslhost is OK: OK - Certificate '100acg.wiki' will expire on Thu 28 Nov 2024 02:29:50 PM GMT +0000.
[15:21:41] <icinga-miraheze>	 RECOVERY - otherkin.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'otherkin.wiki' will expire on Sat 09 Nov 2024 03:36:03 PM GMT +0000.
[15:21:41] <icinga-miraheze>	 RECOVERY - mirabeta.org - Cloudflare on sslhost is OK: OK - Certificate 'mirabeta.org' will expire on Fri 29 Nov 2024 06:53:23 AM GMT +0000.
[15:21:46] <icinga-miraheze>	 RECOVERY - acgn.sfdev.eu.org - Cloudflare on sslhost is OK: OK - Certificate 'acgn.sfdev.eu.org' will expire on Thu 28 Nov 2024 02:06:59 PM GMT +0000.
[15:21:47] <icinga-miraheze>	 RECOVERY - osrwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'osrwiki.org' will expire on Sat 09 Nov 2024 09:06:39 PM GMT +0000.
[15:21:50] <icinga-miraheze>	 RECOVERY - wiki.drycleaninglobby.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.drycleaninglobby.net' will expire on Thu 28 Nov 2024 01:38:28 PM GMT +0000.
[15:21:51] <icinga-miraheze>	 RECOVERY - wiki.apico.buzz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.apico.buzz' will expire on Thu 28 Nov 2024 01:37:49 PM GMT +0000.
[15:21:51] <icinga-miraheze>	 RECOVERY - wiki.luemir.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.luemir.xyz' will expire on Thu 28 Nov 2024 01:39:18 PM GMT +0000.
[15:21:52] <icinga-miraheze>	 RECOVERY - strinova.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:21:53] <icinga-miraheze>	 RECOVERY - wiki.ouro.one - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ouro.one' will expire on Thu 28 Nov 2024 01:40:41 PM GMT +0000.
[15:21:54] <icinga-miraheze>	 RECOVERY - wiki.dfonline.dev - Cloudflare on sslhost is OK: OK - Certificate 'wiki.dfonline.dev' will expire on Thu 28 Nov 2024 01:38:25 PM GMT +0000.
[15:21:55] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:21:57] <icinga-miraheze>	 RECOVERY - grk.repository.archiopedia.org - Cloudflare on sslhost is OK: OK - Certificate 'grk.repository.archiopedia.org' will expire on Fri 29 Nov 2024 07:55:09 PM GMT +0000.
[15:22:00] <icinga-miraheze>	 RECOVERY - rippaversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rippaversewiki.com' will expire on Thu 31 Oct 2024 06:35:20 PM GMT +0000.
[15:22:01] <icinga-miraheze>	 RECOVERY - www.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:22:03] <icinga-miraheze>	 RECOVERY - wiki.ate42.ru - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ate42.ru' will expire on Thu 28 Nov 2024 01:37:54 PM GMT +0000.
[15:22:07] <icinga-miraheze>	 RECOVERY - tech.exputra.com - Cloudflare on sslhost is OK: OK - Certificate 'tech.exputra.com' will expire on Thu 28 Nov 2024 01:34:29 PM GMT +0000.
[15:22:09] <icinga-miraheze>	 RECOVERY - wiki.evscope.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.evscope.org' will expire on Thu 28 Nov 2024 01:38:38 PM GMT +0000.
[15:22:14] <icinga-miraheze>	 RECOVERY - oxboxtra.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'oxboxtra.wiki' will expire on Fri 15 Nov 2024 07:42:02 AM GMT +0000.
[15:22:14] <icinga-miraheze>	 RECOVERY - www.lostidols.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lostidols.wiki' will expire on Fri 29 Nov 2024 07:05:01 PM GMT +0000.
[15:22:24] <icinga-miraheze>	 RECOVERY - rarewarewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewarewiki.com' will expire on Fri 11 Oct 2024 05:28:02 PM GMT +0000.
[15:22:31] <icinga-miraheze>	 RECOVERY - wiki.globasa.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.globasa.net' will expire on Thu 28 Nov 2024 01:38:45 PM GMT +0000.
[15:22:32] <icinga-miraheze>	 RECOVERY - wiki.lefrenchmelee.fr - Cloudflare on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Thu 28 Nov 2024 01:39:13 PM GMT +0000.
[15:22:32] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[15:22:35] <icinga-miraheze>	 RECOVERY - wiki.surf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.surf' will expire on Sun 01 Dec 2024 03:10:28 AM GMT +0000.
[15:22:36] <icinga-miraheze>	 RECOVERY - www.iceria.org - Cloudflare on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Thu 28 Nov 2024 03:57:06 PM GMT +0000.
[15:22:39] <icinga-miraheze>	 RECOVERY - jackbox.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'jackbox.wiki' will expire on Mon 25 Nov 2024 03:33:03 PM GMT +0000.
[15:22:42] <icinga-miraheze>	 RECOVERY - wiki.sukotuze.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.sukotuze.com' will expire on Fri 29 Nov 2024 07:58:49 PM GMT +0000.
[15:22:43] <icinga-miraheze>	 RECOVERY - piwik.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:22:46] <icinga-miraheze>	 RECOVERY - bobobay.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bobobay.wiki' will expire on Fri 29 Nov 2024 02:57:01 PM GMT +0000.
[15:22:47] <icinga-miraheze>	 RECOVERY - siam.info.gf - Cloudflare on sslhost is OK: OK - Certificate 'siam.info.gf' will expire on Thu 28 Nov 2024 01:34:19 PM GMT +0000.
[15:22:48] <icinga-miraheze>	 RECOVERY - wiki.infomedia.co.uk - Cloudflare on sslhost is OK: OK - Certificate 'wiki.infomedia.co.uk' will expire on Thu 28 Nov 2024 01:38:55 PM GMT +0000.
[15:22:50] <icinga-miraheze>	 RECOVERY - status.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:22:51] <icinga-miraheze>	 RECOVERY - wiki.jagtek.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.jagtek.org' will expire on Thu 28 Nov 2024 01:38:59 PM GMT +0000.
[15:22:57] <icinga-miraheze>	 RECOVERY - isocasg.org - Cloudflare on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Sat 27 Aug 2039 04:37:00 PM GMT +0000.
[15:22:58] <icinga-miraheze>	 RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Wed 09 Oct 2024 08:07:14 AM GMT +0000.
[15:22:59] <icinga-miraheze>	 RECOVERY - wiki.milkywayidle.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.milkywayidle.com' will expire on Thu 28 Nov 2024 01:39:28 PM GMT +0000.
[15:23:01] <icinga-miraheze>	 RECOVERY - www.ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Fri 25 Oct 2024 04:43:10 PM GMT +0000.
[15:23:05] <icinga-miraheze>	 RECOVERY - tl.awiki.org - Cloudflare on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Thu 28 Nov 2024 01:35:48 PM GMT +0000.
[15:23:06] <icinga-miraheze>	 RECOVERY - cg.songngu.xyz - Cloudflare on sslhost is OK: OK - Certificate 'cg.songngu.xyz' will expire on Thu 28 Nov 2024 11:52:53 AM GMT +0000.
[15:23:10] <icinga-miraheze>	 RECOVERY - wiki.leapit.no - Cloudflare on sslhost is OK: OK - Certificate 'wiki.leapit.no' will expire on Fri 29 Nov 2024 07:57:56 PM GMT +0000.
[15:23:12] <icinga-miraheze>	 RECOVERY - wiki.seamly.io - Cloudflare on sslhost is OK: OK - Certificate 'wiki.seamly.io' will expire on Thu 28 Nov 2024 01:41:43 PM GMT +0000.
[15:23:16] <icinga-miraheze>	 RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Fri 11 Oct 2024 01:07:26 AM GMT +0000.
[15:23:23] <icinga-miraheze>	 RECOVERY - www.baharna.org - Cloudflare on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Thu 28 Nov 2024 04:02:31 PM GMT +0000.
[15:23:24] <icinga-miraheze>	 RECOVERY - minescape.wiki - Cloudflare on sslhost is OK: OK - Certificate 'minescape.wiki' will expire on Thu 28 Nov 2024 01:31:55 PM GMT +0000.
[15:23:26] <icinga-miraheze>	 RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Sun 13 Oct 2024 01:32:41 PM GMT +0000.
[15:23:31] <icinga-miraheze>	 RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Tue 08 Oct 2024 09:57:49 PM GMT +0000.
[15:23:32] <icinga-miraheze>	 RECOVERY - wiki.ciptamedia.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Thu 28 Nov 2024 01:38:08 PM GMT +0000.
[15:23:32] <icinga-miraheze>	 RECOVERY - polandballwiki.com - Cloudflare on sslhost is OK: OK - Certificate 'polandballwiki.com' will expire on Sat 23 Nov 2024 03:25:15 AM GMT +0000.
[15:23:39] <icinga-miraheze>	 RECOVERY - fc.songcontests.eu - Cloudflare on sslhost is OK: OK - Certificate 'fc.songcontests.eu' will expire on Thu 28 Nov 2024 01:28:08 PM GMT +0000.
[15:23:39] <icinga-miraheze>	 RECOVERY - wiki.meregos.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.meregos.com' will expire on Thu 28 Nov 2024 01:39:25 PM GMT +0000.
[15:23:44] <icinga-miraheze>	 RECOVERY - tno.wiki - Cloudflare on sslhost is OK: OK - Certificate 'tno.wiki' will expire on Thu 28 Nov 2024 02:03:20 PM GMT +0000.
[15:23:45] <icinga-miraheze>	 RECOVERY - wiki.rosestulipsandliberty.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.rosestulipsandliberty.com' will expire on Thu 28 Nov 2024 01:41:39 PM GMT +0000.
[15:23:45] <icinga-miraheze>	 RECOVERY - wiki.astralprojections.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.astralprojections.org' will expire on Thu 28 Nov 2024 01:37:52 PM GMT +0000.
[15:23:45] <icinga-miraheze>	 RECOVERY - wiki.minkyu.kim - Cloudflare on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Thu 28 Nov 2024 01:39:30 PM GMT +0000.
[15:23:46] <icinga-miraheze>	 RECOVERY - mhdh.pj568.eu.org - Cloudflare on sslhost is OK: OK - Certificate 'mhdh.pj568.eu.org' will expire on Thu 28 Nov 2024 01:31:48 PM GMT +0000.
[15:23:47] <MirahezeLSBot>	 !log [alex@cloud16.wikitide.net] brought down all vms
[15:23:53] <icinga-miraheze>	 RECOVERY - wiki.b3313.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.b3313.org' will expire on Thu 28 Nov 2024 01:37:58 PM GMT +0000.
[15:23:56] <icinga-miraheze>	 RECOVERY - wiki.aoemods.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.aoemods.com' will expire on Fri 29 Nov 2024 07:57:25 PM GMT +0000.
[15:23:56] <icinga-miraheze>	 PROBLEM - graylog161 Current Load on graylog161 is CRITICAL: connect to address 10.0.16.130 port 5666: No route to hostconnect to host 10.0.16.130 port 5666: No route to host
[15:23:59] <icinga-miraheze>	 RECOVERY - resources.africanvision.org - Cloudflare on sslhost is OK: OK - Certificate 'resources.africanvision.org' will expire on Fri 29 Nov 2024 07:56:19 PM GMT +0000.
[15:24:05] <icinga-miraheze>	 RECOVERY - wiki.zergmafia.top - LetsEncrypt on sslhost is OK: OK - Certificate 'zergmafia.top' will expire on Sat 30 Nov 2024 06:49:15 PM GMT +0000.
[15:24:05] <icinga-miraheze>	 RECOVERY - uk.religiononfire.mar.in.ua - Cloudflare on sslhost is OK: OK - Certificate 'uk.religiononfire.mar.in.ua' will expire on Thu 28 Nov 2024 01:36:49 PM GMT +0000.
[15:24:06] <icinga-miraheze>	 PROBLEM - os161 NTP time on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host
[15:24:07] <icinga-miraheze>	 RECOVERY - wiki.col6.de - Cloudflare on sslhost is OK: OK - Certificate 'wiki.col6.de' will expire on Thu 28 Nov 2024 01:38:09 PM GMT +0000.
[15:24:14] <icinga-miraheze>	 PROBLEM - mw161 NTP time on mw161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[15:24:22] <icinga-miraheze>	 RECOVERY - wiki.myehs.eu - Cloudflare on sslhost is OK: OK - Certificate 'wiki.myehs.eu' will expire on Thu 28 Nov 2024 01:39:35 PM GMT +0000.
[15:24:28] <icinga-miraheze>	 RECOVERY - wiki.thunis.eu - Cloudflare on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Thu 28 Nov 2024 01:42:41 PM GMT +0000.
[15:24:33] <icinga-miraheze>	 RECOVERY - www.glitchcity.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.glitchcity.wiki' will expire on Fri 29 Nov 2024 07:59:35 PM GMT +0000.
[15:24:34] <icinga-miraheze>	 RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberfurs.org' will expire on Thu 14 Nov 2024 07:49:33 PM GMT +0000.
[15:24:37] <icinga-miraheze>	 RECOVERY - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bikinibottomdiscord.org' will expire on Thu 21 Nov 2024 01:38:48 AM GMT +0000.
[15:24:42] <icinga-miraheze>	 RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:24:46] <icinga-miraheze>	 RECOVERY - history.sdtef.org - Cloudflare on sslhost is OK: OK - Certificate 'history.sdtef.org' will expire on Thu 28 Nov 2024 01:30:58 PM GMT +0000.
[15:24:56] <icinga-miraheze>	 RECOVERY - wiki.showtapeselection.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.showtapeselection.com' will expire on Thu 28 Nov 2024 01:41:48 PM GMT +0000.
[15:25:00] <icinga-miraheze>	 RECOVERY - wiki.chevrine.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.chevrine.com' will expire on Thu 28 Nov 2024 01:38:05 PM GMT +0000.
[15:25:00] <icinga-miraheze>	 RECOVERY - wiki.albrusmusic.ru - Cloudflare on sslhost is OK: OK - Certificate 'wiki.albrusmusic.ru' will expire on Thu 28 Nov 2024 12:47:19 PM GMT +0000.
[15:25:04] <icinga-miraheze>	 RECOVERY - ao90.pinho.org - Cloudflare on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Fri 29 Nov 2024 07:42:23 PM GMT +0000.
[15:25:07] <icinga-miraheze>	 RECOVERY - phorge.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:25:14] <icinga-miraheze>	 RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[15:25:15] <icinga-miraheze>	 RECOVERY - wiki.fatcatfablab.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.fatcatfablab.org' will expire on Thu 28 Nov 2024 01:38:38 PM GMT +0000.
[15:25:17] <icinga-miraheze>	 PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time
[15:25:20] <icinga-miraheze>	 RECOVERY - wiki.dwarvesrp.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.dwarvesrp.com' will expire on Thu 28 Nov 2024 01:38:28 PM GMT +0000.
[15:25:20] <icinga-miraheze>	 RECOVERY - de.berlinwiki.org - Cloudflare on sslhost is OK: OK - Certificate 'de.berlinwiki.org' will expire on Thu 28 Nov 2024 01:25:21 PM GMT +0000.
[15:25:21] <icinga-miraheze>	 PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[15:25:21] <Orange_Star>	 !log rebooted cloud16
[15:25:22] <icinga-miraheze>	 PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time
[15:25:22] <icinga-miraheze>	 PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[15:25:23] <icinga-miraheze>	 RECOVERY - graphite151 Puppet on graphite151 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[15:25:24] <icinga-miraheze>	 PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[15:25:25] <icinga-miraheze>	 PROBLEM - mw161 SSH on mw161 is CRITICAL: connect to address 10.0.16.132 and port 22: No route to host
[15:25:25] <icinga-miraheze>	 PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[15:25:27] <icinga-miraheze>	 PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[15:25:27] <icinga-miraheze>	 PROBLEM - ping on os161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.129)
[15:25:29] <icinga-miraheze>	 PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: connect to address 10.0.16.133 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket
[15:25:29] <icinga-miraheze>	 PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw162.wikitide.net port 443 after 2615 ms: Couldn't connect to server
[15:25:30] <icinga-miraheze>	 PROBLEM - db161 SSH on db161 is CRITICAL: connect to address 10.0.16.128 and port 22: No route to host
[15:25:31] <icinga-miraheze>	 PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[15:25:33] <icinga-miraheze>	 RECOVERY - sonicpedia.org - Cloudflare on sslhost is OK: OK - Certificate 'sonicpedia.org' will expire on Thu 28 Nov 2024 01:34:21 PM GMT +0000.
[15:25:33] <icinga-miraheze>	 PROBLEM - cp27 HTTPS on cp27 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[15:25:34] <icinga-miraheze>	 RECOVERY - www.kinitopedia.lol - Cloudflare on sslhost is OK: OK - Certificate 'www.kinitopedia.lol' will expire on Thu 28 Nov 2024 01:43:28 PM GMT +0000.
[15:25:37] <icinga-miraheze>	 PROBLEM - ping on db161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.128)
[15:25:37] <icinga-miraheze>	 PROBLEM - bast161 NTP time on bast161 is CRITICAL: connect to address 10.0.16.127 port 5666: No route to hostconnect to host 10.0.16.127 port 5666: No route to host
[15:25:38] <icinga-miraheze>	 PROBLEM - bast161 APT on bast161 is CRITICAL: connect to address 10.0.16.127 port 5666: No route to hostconnect to host 10.0.16.127 port 5666: No route to host
[15:25:38] <icinga-miraheze>	 PROBLEM - Host bast161 is DOWN: CRITICAL - Host Unreachable (10.0.16.127)
[15:25:39] <icinga-miraheze>	 PROBLEM - ping on mem161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.131)
[15:25:39] <icinga-miraheze>	 RECOVERY - closinglogosgroup.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:25:40] <Orange_Star>	 the relay got rate limited or something, messages aren't relaying right now
[15:25:43] <icinga-miraheze>	 PROBLEM - ping on graylog161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.130)
[15:25:44] <icinga-miraheze>	 RECOVERY - wiki.k2cho.me - Cloudflare on sslhost is OK: OK - Certificate 'wiki.k2cho.me' will expire on Thu 28 Nov 2024 01:39:08 PM GMT +0000.
[15:25:48] <icinga-miraheze>	 PROBLEM - os161 Disk Space on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: No route to hostconnect to host 10.0.16.129 port 5666: No route to host
[15:25:48] <icinga-miraheze>	 RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:25:49] <icinga-miraheze>	 PROBLEM - Host db161 is DOWN: CRITICAL - Host Unreachable (10.0.16.128)
[15:25:50] <icinga-miraheze>	 PROBLEM - Host graylog161 is DOWN: CRITICAL - Host Unreachable (10.0.16.130)
[15:25:50] <Orange_Star>	 which is not surprising given this
[15:25:51] <icinga-miraheze>	 RECOVERY - wiki.enkro.co - Cloudflare on sslhost is OK: OK - Certificate 'wiki.enkro.co' will expire on Thu 28 Nov 2024 01:38:32 PM GMT +0000.
[15:25:52] <icinga-miraheze>	 PROBLEM - Host os161 is DOWN: CRITICAL - Host Unreachable (10.0.16.129)
[15:25:54] <icinga-miraheze>	 RECOVERY - wiki.wikimedia.cat - Cloudflare on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Thu 28 Nov 2024 01:42:53 PM GMT +0000.
[15:25:56] <icinga-miraheze>	 PROBLEM - Host swiftobject161 is DOWN: CRITICAL - Host Unreachable (10.0.16.134)
[15:25:56] <icinga-miraheze>	 PROBLEM - Host mw162 is DOWN: CRITICAL - Host Unreachable (10.0.16.133)
[15:25:57] <icinga-miraheze>	 RECOVERY - wiki.grab-tools.live - Cloudflare on sslhost is OK: OK - Certificate 'wiki.grab-tools.live' will expire on Thu 28 Nov 2024 01:38:48 PM GMT +0000.
[15:25:58] <icinga-miraheze>	 RECOVERY - lpfteam.nl - Cloudflare on sslhost is OK: OK - Certificate 'lpfteam.nl' will expire on Thu 28 Nov 2024 01:31:41 PM GMT +0000.
[15:26:09] <icinga-miraheze>	 RECOVERY - ie.unigon.net - Cloudflare on sslhost is OK: OK - Certificate 'ie.unigon.net' will expire on Thu 28 Nov 2024 01:31:21 PM GMT +0000.
[15:26:09] <icinga-miraheze>	 RECOVERY - wiki.omegabuild.uk - Cloudflare on sslhost is OK: OK - Certificate 'wiki.omegabuild.uk' will expire on Thu 28 Nov 2024 01:39:41 PM GMT +0000.
[15:26:09] <icinga-miraheze>	 RECOVERY - www.satepedia.com - Cloudflare on sslhost is OK: OK - Certificate 'www.satepedia.com' will expire on Fri 29 Nov 2024 05:09:13 PM GMT +0000.
[15:26:10] <icinga-miraheze>	 RECOVERY - wiki.digitalcandela.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.digitalcandela.com' will expire on Thu 28 Nov 2024 01:38:25 PM GMT +0000.
[15:26:10] <icinga-miraheze>	 PROBLEM - mem161 APT on mem161 is CRITICAL: connect to address 10.0.16.131 port 5666: No route to hostconnect to host 10.0.16.131 port 5666: No route to host
[15:26:12] <icinga-miraheze>	 RECOVERY - ndgkb.nena.org - Cloudflare on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Thu 28 Nov 2024 01:31:55 PM GMT +0000.
[15:26:13] <icinga-miraheze>	 RECOVERY - vitriol.spoon.army - Cloudflare on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Thu 28 Nov 2024 01:37:38 PM GMT +0000.
[15:26:14] <icinga-miraheze>	 RECOVERY - mason.sagan4.org - Cloudflare on sslhost is OK: OK - Certificate 'mason.sagan4.org' will expire on Thu 28 Nov 2024 01:31:45 PM GMT +0000.
[15:26:18] <icinga-miraheze>	 PROBLEM - mem161 memcached on mem161 is CRITICAL: connect to address 10.0.16.131 and port 11211: No route to host
[15:26:18] <icinga-miraheze>	 RECOVERY - ajhalili2006.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:26:19] <icinga-miraheze>	 RECOVERY - wiki.harlonserver.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.harlonserver.net' will expire on Thu 28 Nov 2024 01:38:52 PM GMT +0000.
[15:26:20] <icinga-miraheze>	 PROBLEM - mw161 conntrack_table_size on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[15:26:21] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for mon181 on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[15:26:22] <icinga-miraheze>	 PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[15:26:24] <icinga-miraheze>	 RECOVERY - wiki.s23.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Thu 28 Nov 2024 01:41:41 PM GMT +0000.
[15:26:25] <icinga-miraheze>	 RECOVERY - luxuryelevator.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:26:27] <icinga-miraheze>	 RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.permanentfuturelab.wiki' will expire on Fri 29 Nov 2024 05:09:06 PM GMT +0000.
[15:26:29] <icinga-miraheze>	 PROBLEM - mw161 Disk Space on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[15:26:30] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for mwtask181 on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[15:26:30] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for mw161 on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[15:26:30] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for mw171 on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[15:26:30] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for matomo151 on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[15:26:30] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTP on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable
[15:26:32] <icinga-miraheze>	 PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[15:26:36] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for puppet181 on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[15:26:39] <icinga-miraheze>	 PROBLEM - swiftproxy161 SSH on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 and port 22: No route to host
[15:26:39] <icinga-miraheze>	 PROBLEM - swiftproxy161 ferm_active on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: No route to hostconnect to host 10.0.16.135 port 5666: No route to host
[15:26:40] <icinga-miraheze>	 RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[15:26:42] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[15:26:42] <icinga-miraheze>	 PROBLEM - ping on swiftproxy161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.135)
[15:26:42] <icinga-miraheze>	 PROBLEM - cp36 PowerDNS Recursor on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[15:26:43] <icinga-miraheze>	 PROBLEM - cp36 Puppet on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[15:26:43] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[15:26:43] <icinga-miraheze>	 PROBLEM - mem161 SSH on mem161 is CRITICAL: connect to address 10.0.16.131 and port 22: No route to host
[15:26:43] <icinga-miraheze>	 PROBLEM - Host mem161 is DOWN: CRITICAL - Host Unreachable (10.0.16.131)
[15:26:45] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time
[15:26:45] <icinga-miraheze>	 PROBLEM - swiftproxy161 PowerDNS Recursor on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: No route to hostconnect to host 10.0.16.135 port 5666: No route to host
[15:26:45] <icinga-miraheze>	 PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: No route to hostconnect to host 10.0.16.135 port 5666: No route to host
[15:26:46] <icinga-miraheze>	 PROBLEM - cp36 APT on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[15:26:46] <icinga-miraheze>	 RECOVERY - wiki.cowgl.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cowgl.xyz' will expire on Thu 28 Nov 2024 01:38:15 PM GMT +0000.
[15:26:46] <icinga-miraheze>	 PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[15:26:47] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 3.193 second response time
[15:26:48] <icinga-miraheze>	 PROBLEM - os162 ferm_active on os162 is CRITICAL: connect to address 10.0.16.141 port 5666: No route to hostconnect to host 10.0.16.141 port 5666: No route to host
[15:26:48] <icinga-miraheze>	 PROBLEM - ping on os162 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.141)
[15:27:01] <icinga-miraheze>	 RECOVERY - www.christipedia.nl - Cloudflare on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Fri 29 Nov 2024 05:09:33 PM GMT +0000.
[15:27:01] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable
[15:27:03] <icinga-miraheze>	 PROBLEM - swiftproxy161 conntrack_table_size on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: No route to hostconnect to host 10.0.16.135 port 5666: No route to host
[15:27:03] <icinga-miraheze>	 PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket
[15:27:06] <icinga-miraheze>	 PROBLEM - Host os162 is DOWN: CRITICAL - Host Unreachable (10.0.16.141)
[15:27:06] <icinga-miraheze>	 PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time
[15:27:06] <icinga-miraheze>	 RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:27:06] <icinga-miraheze>	 PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: No route to hostconnect to host 10.0.16.135 port 5666: No route to host
[15:27:07] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - Cloudflare on sslhost is OK: OK - Certificate 'lcn.zfc.id.lv' will expire on Fri 29 Nov 2024 07:55:16 PM GMT +0000.
[15:27:09] <icinga-miraheze>	 RECOVERY - wiki.nowchess.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Fri 29 Nov 2024 07:58:06 PM GMT +0000.
[15:27:11] <icinga-miraheze>	 RECOVERY - wiki.amplicity.fr - Cloudflare on sslhost is OK: OK - Certificate 'wiki.amplicity.fr' will expire on Thu 28 Nov 2024 01:37:48 PM GMT +0000.
[15:27:11] <icinga-miraheze>	 RECOVERY - wiki.aridia.space - Cloudflare on sslhost is OK: OK - Certificate 'wiki.aridia.space' will expire on Fri 29 Nov 2024 07:57:26 PM GMT +0000.
[15:27:12] <icinga-miraheze>	 RECOVERY - nexttide.org - Cloudflare on sslhost is OK: OK - Certificate 'nexttide.org' will expire on Fri 29 Nov 2024 04:52:29 PM GMT +0000.
[15:27:13] <icinga-miraheze>	 RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:27:14] <icinga-miraheze>	 RECOVERY - thelangyalist.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:27:14] <icinga-miraheze>	 RECOVERY - wiki.cor.fyi - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cor.fyi' will expire on Thu 28 Nov 2024 01:38:11 PM GMT +0000.
[15:27:15] <icinga-miraheze>	 PROBLEM - Host swiftproxy161 is DOWN: CRITICAL - Host Unreachable (10.0.16.135)
[15:27:16] <icinga-miraheze>	 PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:27:19] <icinga-miraheze>	 RECOVERY - www.pomologia.org - Cloudflare on sslhost is OK: OK - Certificate 'www.pomologia.org' will expire on Fri 29 Nov 2024 05:09:02 PM GMT +0000.
[15:27:20] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 100%
[15:27:21] <icinga-miraheze>	 PROBLEM - cloud16 SSH on cloud16 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:27:24] <icinga-miraheze>	 RECOVERY - wiki.knowledgerevolution.eu - Cloudflare on sslhost is OK: OK - Certificate 'wiki.knowledgerevolution.eu' will expire on Thu 28 Nov 2024 01:39:11 PM GMT +0000.
[15:27:27] <icinga-miraheze>	 RECOVERY - elsterclopedie.marithaime.nl - Cloudflare on sslhost is OK: OK - Certificate 'elsterclopedie.marithaime.nl' will expire on Thu 28 Nov 2024 01:26:26 PM GMT +0000.
[15:27:28] <icinga-miraheze>	 RECOVERY - internalwiki.coolstation.space - Cloudflare on sslhost is OK: OK - Certificate 'internalwiki.coolstation.space' will expire on Thu 28 Nov 2024 01:31:25 PM GMT +0000.
[15:27:28] <icinga-miraheze>	 RECOVERY - wiki.starshipsailing.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.starshipsailing.net' will expire on Thu 28 Nov 2024 01:41:59 PM GMT +0000.
[15:27:30] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 49%
[15:27:31] <icinga-miraheze>	 RECOVERY - wiki.orivium.io - Cloudflare on sslhost is OK: OK - Certificate 'wiki.orivium.io' will expire on Thu 28 Nov 2024 01:45:56 PM GMT +0000.
[15:27:33] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:27:37] <icinga-miraheze>	 RECOVERY - wiki.pokrew.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.pokrew.net' will expire on Thu 28 Nov 2024 01:41:28 PM GMT +0000.
[15:27:38] <icinga-miraheze>	 RECOVERY - wiki.whentheycry.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Thu 28 Nov 2024 01:42:50 PM GMT +0000.
[15:27:39] <icinga-miraheze>	 RECOVERY - wiki.mega-earth.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mega-earth.com' will expire on Thu 28 Nov 2024 01:39:25 PM GMT +0000.
[15:27:46] <icinga-miraheze>	 RECOVERY - wiki.funkey-project.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.funkey-project.com' will expire on Thu 28 Nov 2024 01:38:40 PM GMT +0000.
[15:27:51] <icinga-miraheze>	 RECOVERY - www.pyramidgames.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pyramidgames.wiki' will expire on Thu 28 Nov 2024 04:29:09 PM GMT +0000.
[15:28:05] <icinga-miraheze>	 RECOVERY - wiki.cosmicdeathfungus.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cosmicdeathfungus.com' will expire on Sun 24 Nov 2024 05:06:01 PM GMT +0000.
[15:28:06] <icinga-miraheze>	 RECOVERY - www.widedata.org - Cloudflare on sslhost is OK: OK - Certificate 'www.widedata.org' will expire on Thu 28 Nov 2024 04:20:57 PM GMT +0000.
[15:28:08] <icinga-miraheze>	 RECOVERY - wiki.otir.nl - Cloudflare on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Thu 28 Nov 2024 01:40:41 PM GMT +0000.
[15:28:09] <icinga-miraheze>	 RECOVERY - www.lebork.info.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lebork.info.pl' will expire on Thu 28 Nov 2024 04:02:44 PM GMT +0000.
[15:28:12] <icinga-miraheze>	 RECOVERY - itora.bunnies1230.com - Cloudflare on sslhost is OK: OK - Certificate 'itora.bunnies1230.com' will expire on Thu 28 Nov 2024 01:31:29 PM GMT +0000.
[15:28:26] <icinga-miraheze>	 RECOVERY - wiki.virtualjet.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.virtualjet.net' will expire on Thu 28 Nov 2024 01:42:48 PM GMT +0000.
[15:28:28] <icinga-miraheze>	 RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mikrodev.com' will expire on Fri 08 Nov 2024 05:10:40 AM GMT +0000.
[15:28:31] <icinga-miraheze>	 RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:28:32] <icinga-miraheze>	 RECOVERY - pwiki.drydraytonvillagehall.org.uk - Cloudflare on sslhost is OK: OK - Certificate 'pwiki.drydraytonvillagehall.org.uk' will expire on Thu 28 Nov 2024 01:32:48 PM GMT +0000.
[15:28:34] <icinga-miraheze>	 RECOVERY - abeyancewiki.bumblebee-studios.se - Cloudflare on sslhost is OK: OK - Certificate 'abeyancewiki.bumblebee-studios.se' will expire on Thu 28 Nov 2024 02:03:51 PM GMT +0000.
[15:28:35] <icinga-miraheze>	 RECOVERY - wiki.cube-conflict.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cube-conflict.com' will expire on Thu 28 Nov 2024 01:38:19 PM GMT +0000.
[15:28:37] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 53%
[15:28:38] <icinga-miraheze>	 RECOVERY - dc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:28:42] <icinga-miraheze>	 RECOVERY - www.lgbtqia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lgbtqia.wiki' will expire on Fri 29 Nov 2024 11:15:50 AM GMT +0000.
[15:28:47] <icinga-miraheze>	 RECOVERY - meijinversus.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:28:48] <icinga-miraheze>	 RECOVERY - Host cp36 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms
[15:28:51] <icinga-miraheze>	 RECOVERY - wiki.thelastdimension.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.thelastdimension.net' will expire on Thu 28 Nov 2024 01:42:38 PM GMT +0000.
[15:28:51] <icinga-miraheze>	 RECOVERY - Host mw161 is UP: PING OK - Packet loss = 16%, RTA = 325.88 ms
[15:28:52] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for mw181 on cp36 is CRITICAL: connect to address localhost and port 8119: Connection refused
[15:28:56] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.091 second response time
[15:29:00] <icinga-miraheze>	 RECOVERY - wiki.taes.com.br - Cloudflare on sslhost is OK: OK - Certificate 'wiki.taes.com.br' will expire on Thu 28 Nov 2024 01:42:02 PM GMT +0000.
[15:29:01] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8115
[15:29:02] <icinga-miraheze>	 PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp36.wikitide.net port 443 after 0 ms: Couldn't connect to server
[15:29:02] <icinga-miraheze>	 RECOVERY - wiki.neonai.dev - Cloudflare on sslhost is OK: OK - Certificate 'wiki.neonai.dev' will expire on Thu 28 Nov 2024 01:39:38 PM GMT +0000.
[15:29:02] <icinga-miraheze>	 RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.294 second response time
[15:29:03] <icinga-miraheze>	 RECOVERY - grk.archiopedia.org - Cloudflare on sslhost is OK: OK - Certificate 'grk.archiopedia.org' will expire on Thu 28 Nov 2024 01:29:19 PM GMT +0000.
[15:29:06] <MirahezeLSBot>	 !log [alex@cloud16.wikitide.net] brought up all vms
[15:29:07] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for mw182 on cp36 is CRITICAL: connect to address localhost and port 8120: Connection refused
[15:29:07] <icinga-miraheze>	 RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.068 second response time
[15:29:08] <icinga-miraheze>	 RECOVERY - Host os162 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms
[15:29:11] <icinga-miraheze>	 RECOVERY - lgbtqia.wiki - Cloudflare on sslhost is OK: OK - Certificate 'lgbtqia.wiki' will expire on Thu 28 Nov 2024 01:31:48 PM GMT +0000.
[15:29:12] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for mw152 on cp36 is CRITICAL: connect to address localhost and port 8114: Connection refused
[15:29:12] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for test151 on cp36 is CRITICAL: connect to address localhost and port 8181: Connection refused
[15:29:12] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for swiftproxy171 on cp36 is CRITICAL: connect to address localhost and port 8207: Connection refused
[15:29:13] <icinga-miraheze>	 RECOVERY - Host swiftproxy161 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms
[15:29:14] <icinga-miraheze>	 RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.308 second response time
[15:29:15] <icinga-miraheze>	 RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.345 second response time
[15:29:15] <icinga-miraheze>	 RECOVERY - removededmsongs.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:29:15] <icinga-miraheze>	 RECOVERY - wiki.candelabrem.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.candelabrem.com' will expire on Thu 28 Nov 2024 01:38:01 PM GMT +0000.
[15:29:16] <icinga-miraheze>	 RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[15:29:17] <icinga-miraheze>	 PROBLEM - cp36 NTP time on cp36 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[15:29:17] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for swiftproxy161 on cp36 is CRITICAL: connect to address localhost and port 8206: Connection refused
[15:29:17] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for mwtask171 on cp36 is CRITICAL: connect to address localhost and port 8161: Connection refused
[15:29:17] <icinga-miraheze>	 RECOVERY - mw161 SSH on mw161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:29:18] <icinga-miraheze>	 RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.064 second response time
[15:29:18] <icinga-miraheze>	 RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.064 second response time
[15:29:19] <icinga-miraheze>	 RECOVERY - cloud16 SSH on cloud16 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:29:20] <icinga-miraheze>	 RECOVERY - wiki.namegames.dev - Cloudflare on sslhost is OK: OK - Certificate 'wiki.namegames.dev' will expire on Thu 28 Nov 2024 01:39:35 PM GMT +0000.
[15:29:31] <icinga-miraheze>	 RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.078 second response time
[15:29:31] <icinga-miraheze>	 RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 33%
[15:29:32] <icinga-miraheze>	 RECOVERY - wiki.wildport.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.wildport.com' will expire on Thu 28 Nov 2024 01:42:53 PM GMT +0000.
[15:29:32] <icinga-miraheze>	 RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4008 bytes in 1.117 second response time
[15:29:33] <icinga-miraheze>	 RECOVERY - wiki.alathramc.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.alathramc.com' will expire on Thu 28 Nov 2024 01:37:45 PM GMT +0000.
[15:29:34] <icinga-miraheze>	 RECOVERY - cp27 HTTPS on cp27 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4009 bytes in 0.944 second response time
[15:29:35] <icinga-miraheze>	 RECOVERY - Host bast161 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms
[15:29:44] <icinga-miraheze>	 RECOVERY - chinafake.wiki - Cloudflare on sslhost is OK: OK - Certificate 'chinafake.wiki' will expire on Thu 28 Nov 2024 01:25:15 PM GMT +0000.
[15:29:46] <icinga-miraheze>	 RECOVERY - Host graylog161 is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms
[15:29:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:29:50] <icinga-miraheze>	 RECOVERY - Host db161 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms
[15:29:53] <icinga-miraheze>	 RECOVERY - Host swiftobject161 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms
[15:29:55] <icinga-miraheze>	 RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[15:29:57] <icinga-miraheze>	 RECOVERY - wmworld.sktz.live - Cloudflare on sslhost is OK: OK - Certificate 'wmworld.sktz.live' will expire on Fri 29 Nov 2024 07:59:33 PM GMT +0000.
[15:29:57] <icinga-miraheze>	 RECOVERY - Host os161 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms
[15:29:57] <icinga-miraheze>	 RECOVERY - graylog161 Current Load on graylog161 is OK: LOAD OK - total load average: 3.24, 0.90, 0.31
[15:29:58] <MirahezeRelay>	 <bluemoon0332> cloud16 and all of its vms are back up
[15:29:59] <icinga-miraheze>	 RECOVERY - Host mw162 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms
[15:30:03] <icinga-miraheze>	 RECOVERY - wiki.mineplay.nl - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mineplay.nl' will expire on Thu 28 Nov 2024 01:39:28 PM GMT +0000.
[15:30:03] <icinga-miraheze>	 RECOVERY - wiki.eggsdstudios.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.eggsdstudios.com' will expire on Thu 28 Nov 2024 01:38:29 PM GMT +0000.
[15:30:06] <icinga-miraheze>	 RECOVERY - tfuk.app - Cloudflare on sslhost is OK: OK - Certificate 'tfuk.app' will expire on Thu 28 Nov 2024 01:35:45 PM GMT +0000.
[15:30:08] <icinga-miraheze>	 RECOVERY - os161 NTP time on os161 is OK: NTP OK: Offset -6.377696991e-06 secs
[15:30:10] <icinga-miraheze>	 RECOVERY - mw161 NTP time on mw161 is OK: NTP OK: Offset -8.264183998e-05 secs
[15:30:13] <icinga-miraheze>	 RECOVERY - meta.sagan4.org - Cloudflare on sslhost is OK: OK - Certificate 'meta.sagan4.org' will expire on Thu 28 Nov 2024 01:31:48 PM GMT +0000.
[15:30:13] <icinga-miraheze>	 RECOVERY - mw161 PowerDNS Recursor on mw161 is OK: DNS OK: 0.044 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:30:14] <icinga-miraheze>	 RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.819 second response time
[15:30:17] <icinga-miraheze>	 RECOVERY - wiki.villagecollaborative.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Fri 29 Nov 2024 07:59:06 PM GMT +0000.
[15:30:20] <icinga-miraheze>	 RECOVERY - mw161 conntrack_table_size on mw161 is OK: OK: nf_conntrack is 2 % full
[15:30:20] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[15:30:21] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTP on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.026 second response time
[15:30:24] <icinga-miraheze>	 RECOVERY - mw161 Disk Space on mw161 is OK: DISK OK - free space: / 20442MiB (38% inode=86%);
[15:30:24] <icinga-miraheze>	 RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.080 second response time
[15:30:24] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[15:30:32] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 44%
[15:30:35] <icinga-miraheze>	 RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 4%
[15:30:36] <icinga-miraheze>	 RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:30:37] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 0.56, 0.39, 0.16
[15:30:37] <icinga-miraheze>	 RECOVERY - os162 ferm_active on os162 is OK: OK ferm input default policy is set
[15:30:37] <icinga-miraheze>	 RECOVERY - cp36 SSH on cp36 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:30:38] <icinga-miraheze>	 RECOVERY - swiftproxy161 SSH on swiftproxy161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:30:38] <icinga-miraheze>	 RECOVERY - swiftproxy161 PowerDNS Recursor on swiftproxy161 is OK: DNS OK: 0.054 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:30:38] <icinga-miraheze>	 RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 0.63, 0.28, 0.11
[15:30:38] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.314 second response time
[15:30:39] <icinga-miraheze>	 RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:30:39] <icinga-miraheze>	 RECOVERY - swiftproxy161 ferm_active on swiftproxy161 is OK: OK ferm input default policy is set
[15:30:40] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.600 second response time
[15:30:40] <icinga-miraheze>	 RECOVERY - ping on swiftproxy161 is OK: PING OK - Packet loss = 0%, RTA = 0.40 ms
[15:30:40] <icinga-miraheze>	 RECOVERY - Host mem161 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms
[15:30:41] <icinga-miraheze>	 RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.065 second response time
[15:30:42] <icinga-miraheze>	 RECOVERY - cp36 APT on cp36 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[15:30:43] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[15:30:45] <icinga-miraheze>	 RECOVERY - mw161 ferm_active on mw161 is OK: OK ferm input default policy is set
[15:30:45] <icinga-miraheze>	 RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[15:30:47] <icinga-miraheze>	 RECOVERY - os161 PowerDNS Recursor on os161 is OK: DNS OK: 0.091 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:30:47] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for reports171 on cp36 is CRITICAL: connect to address localhost and port 8205: Connection refused
[15:30:47] <icinga-miraheze>	 RECOVERY - os162 PowerDNS Recursor on os162 is OK: DNS OK: 0.036 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:30:59] <icinga-miraheze>	 RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[15:31:01] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.024 second response time
[15:31:02] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[15:31:06] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[15:31:07] <icinga-miraheze>	 RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 55 packages available for upgrade (0 critical updates).
[15:31:08] <icinga-miraheze>	 RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[15:31:14] <icinga-miraheze>	 RECOVERY - es.countryhumans.polandball.wiki - Cloudflare on sslhost is OK: OK - Certificate 'es.countryhumans.polandball.wiki' will expire on Thu 28 Nov 2024 01:27:58 PM GMT +0000.
[15:31:15] <icinga-miraheze>	 RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4032 bytes in 1.066 second response time
[15:31:16] <icinga-miraheze>	 RECOVERY - www.rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rosettacode.org' will expire on Fri 29 Nov 2024 05:09:15 PM GMT +0000.
[15:31:19] <icinga-miraheze>	 RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mh142.com' will expire on Thu 28 Nov 2024 03:54:54 PM GMT +0000.
[15:31:21] <icinga-miraheze>	 RECOVERY - wiki.mc.marochuru.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mc.marochuru.net' will expire on Fri 29 Nov 2024 07:58:02 PM GMT +0000.
[15:31:22] <icinga-miraheze>	 RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.360 second response time
[15:31:22] <icinga-miraheze>	 RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.066 second response time
[15:31:22] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.362 second response time
[15:31:23] <icinga-miraheze>	 RECOVERY - swiftproxy161 NTP time on swiftproxy161 is OK: NTP OK: Offset 9.834766388e-06 secs
[15:31:27] <icinga-miraheze>	 RECOVERY - db161 SSH on db161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:31:27] <icinga-miraheze>	 RECOVERY - test2.specialchat.org - Cloudflare on sslhost is OK: OK - Certificate 'test2.specialchat.org' will expire on Thu 28 Nov 2024 01:35:40 PM GMT +0000.
[15:31:30] <icinga-miraheze>	 RECOVERY - equestripedia.org - Cloudflare on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Thu 28 Nov 2024 01:27:15 PM GMT +0000.
[15:31:32] <icinga-miraheze>	 RECOVERY - bast161 APT on bast161 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[15:31:32] <icinga-miraheze>	 RECOVERY - ping on os161 is OK: PING OK - Packet loss = 0%, RTA = 0.30 ms
[15:31:33] <icinga-miraheze>	 RECOVERY - publictestwiki.com - Cloudflare on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Thu 28 Nov 2024 01:32:45 PM GMT +0000.
[15:31:33] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[15:31:34] <icinga-miraheze>	 RECOVERY - wiki.openhatch.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.openhatch.org' will expire on Thu 28 Nov 2024 01:39:42 PM GMT +0000.
[15:31:35] <icinga-miraheze>	 RECOVERY - tintinpedia.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:31:37] <icinga-miraheze>	 RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -0.0002972781658 secs
[15:31:38] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_PyWikiBot]
[15:31:39] <icinga-miraheze>	 RECOVERY - fanwiki.takagi3.cn - Cloudflare on sslhost is OK: OK - Certificate 'fanwiki.takagi3.cn' will expire on Thu 28 Nov 2024 01:28:06 PM GMT +0000.
[15:31:42] <icinga-miraheze>	 RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 74152MiB (33% inode=99%);
[15:31:45] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.andreijiroh.uk.eu.org' will expire on Thu 28 Nov 2024 01:37:48 PM GMT +0000.
[15:31:47] <icinga-miraheze>	 RECOVERY - ping on mem161 is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms
[15:31:47] <icinga-miraheze>	 RECOVERY - ping on graylog161 is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms
[15:31:47] <icinga-miraheze>	 RECOVERY - ping on db161 is OK: PING OK - Packet loss = 0%, RTA = 0.27 ms
[15:31:50] <icinga-miraheze>	 RECOVERY - smashbros.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[15:31:51] <icinga-miraheze>	 RECOVERY - wiki.orvyn.ca - Cloudflare on sslhost is OK: OK - Certificate 'wiki.orvyn.ca' will expire on Fri 29 Nov 2024 07:58:09 PM GMT +0000.
[15:31:53] <icinga-miraheze>	 RECOVERY - wiki.elgeis.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.elgeis.com' will expire on Thu 28 Nov 2024 01:38:30 PM GMT +0000.
[15:32:07] <icinga-miraheze>	 RECOVERY - mem161 memcached on mem161 is OK: TCP OK - 0.000 second response time on 10.0.16.131 port 11211
[15:32:08] <icinga-miraheze>	 RECOVERY - chobots.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'chobots.wiki' will expire on Wed 16 Oct 2024 07:29:39 AM GMT +0000.
[15:32:12] <icinga-miraheze>	 RECOVERY - mem161 APT on mem161 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[15:32:12] <icinga-miraheze>	 RECOVERY - education.gced.in - Cloudflare on sslhost is OK: OK - Certificate 'education.gced.in' will expire on Thu 28 Nov 2024 01:26:22 PM GMT +0000.
[15:32:23] <icinga-miraheze>	 RECOVERY - forecasting.wiki - Cloudflare on sslhost is OK: OK - Certificate 'forecasting.wiki' will expire on Thu 28 Nov 2024 01:29:10 PM GMT +0000.
[15:32:24] <icinga-miraheze>	 RECOVERY - correypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'correypedia.org' will expire on Mon 07 Oct 2024 08:32:43 PM GMT +0000.
[15:32:28] <icinga-miraheze>	 RECOVERY - www.johanloopmans.nl - Cloudflare on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Thu 28 Nov 2024 03:57:51 PM GMT +0000.
[15:32:29] <icinga-miraheze>	 RECOVERY - wiki.misslessfighters.club - Cloudflare on sslhost is OK: OK - Certificate 'wiki.misslessfighters.club' will expire on Thu 28 Nov 2024 01:39:31 PM GMT +0000.
[15:32:30] <icinga-miraheze>	 RECOVERY - msmhacks.uk - Cloudflare on sslhost is OK: OK - Certificate 'msmhacks.uk' will expire on Thu 28 Nov 2024 01:31:58 PM GMT +0000.
[15:32:32] <icinga-miraheze>	 RECOVERY - mem161 SSH on mem161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[15:32:33] <icinga-miraheze>	 RECOVERY - wiki.elysiamc.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.elysiamc.xyz' will expire on Thu 28 Nov 2024 01:38:31 PM GMT +0000.
[15:32:33] <icinga-miraheze>	 RECOVERY - alpha.sagan4.org - Cloudflare on sslhost is OK: OK - Certificate 'alpha.sagan4.org' will expire on Thu 28 Nov 2024 01:24:50 PM GMT +0000.
[15:32:35] <icinga-miraheze>	 RECOVERY - www.journeytheword.wiki - Cloudflare on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Fri 29 Nov 2024 05:07:52 PM GMT +0000.
[15:32:35] <icinga-miraheze>	 RECOVERY - wiki.starship.digital - Cloudflare on sslhost is OK: OK - Certificate 'wiki.starship.digital' will expire on Thu 28 Nov 2024 01:41:58 PM GMT +0000.
[15:32:35] <icinga-miraheze>	 RECOVERY - wiki.geoparkcorumbatai.com.br - Cloudflare on sslhost is OK: OK - Certificate 'wiki.geoparkcorumbatai.com.br' will expire on Fri 29 Nov 2024 07:57:50 PM GMT +0000.
[15:32:44] <icinga-miraheze>	 RECOVERY - www.thesimswiki.com - Cloudflare on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Thu 28 Nov 2024 01:43:38 PM GMT +0000.
[15:32:49] <icinga-miraheze>	 RECOVERY - www.gengbaike.top - LetsEncrypt on sslhost is OK: OK - Certificate 'gengbaike.top' will expire on Tue 19 Nov 2024 05:07:14 PM GMT +0000.
[15:32:50] <icinga-miraheze>	 RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[15:32:50] <icinga-miraheze>	 RECOVERY - progressbar95.wiki - Cloudflare on sslhost is OK: OK - Certificate 'progressbar95.wiki' will expire on Thu 28 Nov 2024 01:32:38 PM GMT +0000.
[15:32:55] <icinga-miraheze>	 RECOVERY - wiki.artmechanicum.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.artmechanicum.com' will expire on Thu 28 Nov 2024 01:37:51 PM GMT +0000.
[15:33:01] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw161 on cp37 is CRITICAL: connect to address localhost and port 8115: Connection refused
[15:33:07] <icinga-miraheze>	 RECOVERY - wiki.simplifiedcoding.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.simplifiedcoding.org' will expire on Thu 28 Nov 2024 01:41:50 PM GMT +0000.
[15:33:20] <icinga-miraheze>	 RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[15:33:38] <icinga-miraheze>	 RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:35:28] <icinga-miraheze>	 RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:36:13] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:36:46] <icinga-miraheze>	 RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[15:37:21] <icinga-miraheze>	 RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[15:38:27] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[15:38:49] <icinga-miraheze>	 RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:39:32] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:40:39] <icinga-miraheze>	 RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[15:41:55] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:43:10] <icinga-miraheze>	 RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[15:45:11] <icinga-miraheze>	 RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[15:45:58] <icinga-miraheze>	 RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[15:46:20] <icinga-miraheze>	 RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:46:27] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:46:35] <icinga-miraheze>	 RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[15:47:42] <icinga-miraheze>	 PROBLEM - dcmultiverse.wikitide.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:47:43] <icinga-miraheze>	 PROBLEM - pso2ngs.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:47:44] <icinga-miraheze>	 PROBLEM - kingdomdeath.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:47:48] <icinga-miraheze>	 PROBLEM - keitaiwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:47:50] <icinga-miraheze>	 PROBLEM - geeu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:47:56] <icinga-miraheze>	 PROBLEM - wikitide.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:47:57] <icinga-miraheze>	 PROBLEM - wikigenius.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:47:58] <icinga-miraheze>	 PROBLEM - persist.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:47:59] <icinga-miraheze>	 PROBLEM - worldsanskrit.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:47:59] <icinga-miraheze>	 PROBLEM - fashiondreamer.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:47:59] <icinga-miraheze>	 PROBLEM - hoseki.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:02] <icinga-miraheze>	 PROBLEM - wonderfuleveryday.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:05] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000.
[15:48:07] <icinga-miraheze>	 PROBLEM - beyondtheveilrp.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:08] <icinga-miraheze>	 PROBLEM - volunteerforukraine.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:10] <icinga-miraheze>	 PROBLEM - bonesword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:12] <icinga-miraheze>	 PROBLEM - poupedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:12] <icinga-miraheze>	 PROBLEM - emmytherobot.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:13] <icinga-miraheze>	 PROBLEM - kombinat.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:13] <icinga-miraheze>	 PROBLEM - metroidpedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:13] <icinga-miraheze>	 PROBLEM - tssm.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:15] <icinga-miraheze>	 PROBLEM - theenglishlakes.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:21] <icinga-miraheze>	 PROBLEM - baharna.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:22] <icinga-miraheze>	 PROBLEM - mh142.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:23] <icinga-miraheze>	 PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:23] <icinga-miraheze>	 PROBLEM - *.wikitide.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:24] <icinga-miraheze>	 PROBLEM - osfirsttimer.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:26] <icinga-miraheze>	 PROBLEM - dragdown.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:27] <icinga-miraheze>	 PROBLEM - maxcapacity.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:27] <icinga-miraheze>	 PROBLEM - schizoidnightmares.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:30] <icinga-miraheze>	 PROBLEM - journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:30] <icinga-miraheze>	 PROBLEM - nintendowiki.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:31] <icinga-miraheze>	 PROBLEM - spiceandwolf.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:31] <icinga-miraheze>	 PROBLEM - heavyironmodding.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:31] <icinga-miraheze>	 PROBLEM - gimkit.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:33] <icinga-miraheze>	 PROBLEM - landofliberos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:33] <icinga-miraheze>	 PROBLEM - dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:35] <icinga-miraheze>	 PROBLEM - fanonpedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:36] <icinga-miraheze>	 PROBLEM - annapolishistorywiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:36] <icinga-miraheze>	 PROBLEM - podpedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:37] <icinga-miraheze>	 PROBLEM - dc.wikitide.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:39] <icinga-miraheze>	 PROBLEM - comprehensibleinputwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:40] <icinga-miraheze>	 PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:40] <icinga-miraheze>	 PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:42] <icinga-miraheze>	 PROBLEM - eosdev.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:44] <icinga-miraheze>	 PROBLEM - royal-wiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:45] <icinga-miraheze>	 PROBLEM - rodzinka.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:45] <icinga-miraheze>	 PROBLEM - lostmediawiki.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:46] <icinga-miraheze>	 PROBLEM - partopedia.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:46] <icinga-miraheze>	 PROBLEM - issue-tracker.wikitide.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:47] <icinga-miraheze>	 PROBLEM - ecole.science - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:47] <icinga-miraheze>	 PROBLEM - kalons-reverie.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:50] <icinga-miraheze>	 PROBLEM - starveil.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:51] <icinga-miraheze>	 PROBLEM - aryavratpedia.co - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:52] <icinga-miraheze>	 PROBLEM - archivesofhavnor.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:53] <icinga-miraheze>	 PROBLEM - urbanshade.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:54] <icinga-miraheze>	 PROBLEM - knifepointhorror.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:48:55] <icinga-miraheze>	 PROBLEM - project-patterns.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:01] <icinga-miraheze>	 PROBLEM - lads.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:01] <icinga-miraheze>	 PROBLEM - rippaverse.wikitide.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:02] <icinga-miraheze>	 PROBLEM - johanloopmans.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:03] <icinga-miraheze>	 PROBLEM - rosettacode.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:04] <icinga-miraheze>	 PROBLEM - quadball.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:06] <icinga-miraheze>	 PROBLEM - portalsofphereon.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:07] <icinga-miraheze>	 PROBLEM - thegreatwar.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:08] <icinga-miraheze>	 PROBLEM - phighting.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:08] <icinga-miraheze>	 PROBLEM - patternarchive.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:08] <icinga-miraheze>	 PROBLEM - aryavartpedia.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:09] <icinga-miraheze>	 PROBLEM - mockgovernments.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:12] <icinga-miraheze>	 PROBLEM - dragonquestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:13] <icinga-miraheze>	 PROBLEM - kunwok.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:13] <icinga-miraheze>	 PROBLEM - evilgeniuswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:14] <icinga-miraheze>	 PROBLEM - baligawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:17] <icinga-miraheze>	 PROBLEM - wiki.encyclopbd.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:17] <icinga-miraheze>	 PROBLEM - pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:19] <icinga-miraheze>	 PROBLEM - squareenixwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:20] <icinga-miraheze>	 PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:22] <icinga-miraheze>	 PROBLEM - rothwell-leeds.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:24] <icinga-miraheze>	 PROBLEM - icclopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:27] <icinga-miraheze>	 PROBLEM - hardcore.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:27] <icinga-miraheze>	 PROBLEM - hololive.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:28] <icinga-miraheze>	 PROBLEM - neurologiahpm.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:29] <icinga-miraheze>	 PROBLEM - solaswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:29] <icinga-miraheze>	 PROBLEM - yokaiwatchwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:34] <icinga-miraheze>	 PROBLEM - stablestate.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:35] <icinga-miraheze>	 PROBLEM - vesc.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:39] <icinga-miraheze>	 PROBLEM - freedomplanetwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:45] <icinga-miraheze>	 PROBLEM - cpmusicwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:49] <icinga-miraheze>	 PROBLEM - removededm.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:51] <icinga-miraheze>	 PROBLEM - worldtriggerwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:52] <icinga-miraheze>	 PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:55] <icinga-miraheze>	 PROBLEM - kingdomway.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:02] <icinga-miraheze>	 PROBLEM - trollpasta.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:12] <icinga-miraheze>	 PROBLEM - pokemonwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:14] <icinga-miraheze>	 PROBLEM - fanojo.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:15] <icinga-miraheze>	 PROBLEM - terraforming.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:19] <icinga-miraheze>	 PROBLEM - christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:29] <icinga-miraheze>	 PROBLEM - kodiak.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:32] <icinga-miraheze>	 PROBLEM - rainverse.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:33] <icinga-miraheze>	 PROBLEM - dkpedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:33] <icinga-miraheze>	 PROBLEM - xomnipedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:41] <icinga-miraheze>	 PROBLEM - encyclopediarobotica.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:42] <icinga-miraheze>	 PROBLEM - wikimoma.art - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:42] <icinga-miraheze>	 PROBLEM - crashspyrowiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:50] <icinga-miraheze>	 PROBLEM - familiacorsi.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:51] <icinga-miraheze>	 PROBLEM - beidipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:50:56] <icinga-miraheze>	 PROBLEM - coffeewiki.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:51:19] <icinga-miraheze>	 PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:51:26] <icinga-miraheze>	 PROBLEM - civwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:51:37] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:51:59] <icinga-miraheze>	 PROBLEM - clubpenguinfanon.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:52:09] <icinga-miraheze>	 PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:52:10] <icinga-miraheze>	 PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:52:14] <icinga-miraheze>	 PROBLEM - cheeseepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:52:34] <icinga-miraheze>	 PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:52:44] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:52:58] <icinga-miraheze>	 PROBLEM - puzzles.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:53:10] <icinga-miraheze>	 PROBLEM - junxions.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:53:14] <icinga-miraheze>	 PROBLEM - iol.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:53:26] <icinga-miraheze>	 PROBLEM - grayzonewarfare.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:53:27] <icinga-miraheze>	 PROBLEM - rarewarewiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:53:27] <icinga-miraheze>	 PROBLEM - rippaversewiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:53:34] <icinga-miraheze>	 PROBLEM - crocwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:53:35] <icinga-miraheze>	 PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:53:40] <icinga-miraheze>	 PROBLEM - oxboxtra.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:53:55] <icinga-miraheze>	 PROBLEM - osrwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:54:02] <icinga-miraheze>	 PROBLEM - otherkin.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:54:14] <icinga-miraheze>	 PROBLEM - isocasg.org - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:54:34] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[15:54:45] <icinga-miraheze>	 RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:54:48] <icinga-miraheze>	 PROBLEM - bobobay.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:54:49] <icinga-miraheze>	 PROBLEM - electowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:54:50] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[15:54:58] <icinga-miraheze>	 PROBLEM - allthetropes.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:55:01] <icinga-miraheze>	 PROBLEM - jackbox.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:55:17] <icinga-miraheze>	 PROBLEM - buildabearwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:55:22] <icinga-miraheze>	 PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:55:32] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:14:33] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:46:45] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:15:06] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:44:12] <icinga-miraheze>	 PROBLEM - cp36 NTP time on cp36 is CRITICAL: connect to address 10.0.16.137 port 5666: No route to hostconnect to host 10.0.16.137 port 5666: No route to host
[17:44:17] <icinga-miraheze>	 RECOVERY - cp36 PowerDNS Recursor on cp36 is OK: DNS OK: 2.367 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:44:30] <icinga-miraheze>	 RECOVERY - bonesword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bonesword.wiki' will expire on Tue 12 Nov 2024 06:49:37 AM GMT +0000.
[17:44:33] <icinga-miraheze>	 RECOVERY - emmytherobot.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'emmytherobot.wiki' will expire on Fri 01 Nov 2024 01:07:57 PM GMT +0000.
[17:44:34] <icinga-miraheze>	 RECOVERY - journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Sun 20 Oct 2024 04:49:12 PM GMT +0000.
[17:44:35] <icinga-miraheze>	 RECOVERY - worldtriggerwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'worldtriggerwiki.com' will expire on Fri 18 Oct 2024 01:12:55 AM GMT +0000.
[17:44:41] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for phorge171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8202
[17:44:43] <icinga-miraheze>	 RECOVERY - eosdev.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'eosdev.xyz' will expire on Sat 05 Oct 2024 08:57:15 PM GMT +0000.
[17:44:47] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8113
[17:44:50] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy
[17:44:50] <icinga-miraheze>	 RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4008 bytes in 0.146 second response time
[17:44:55] <icinga-miraheze>	 RECOVERY - kombinat.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kombinat.eu.org' will expire on Fri 22 Nov 2024 10:09:37 AM GMT +0000.
[17:44:56] <icinga-miraheze>	 RECOVERY - maxcapacity.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'maxcapacity.wiki' will expire on Sun 22 Sep 2024 04:44:31 PM GMT +0000.
[17:45:00] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mwtask171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8161
[17:45:02] <icinga-miraheze>	 RECOVERY - annapolishistorywiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'annapolishistorywiki.org' will expire on Tue 19 Nov 2024 09:58:55 AM GMT +0000.
[17:45:06] <icinga-miraheze>	 RECOVERY - lostmediawiki.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'lostmediawiki.ru' will expire on Sat 09 Nov 2024 03:14:43 PM GMT +0000.
[17:45:09] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw152 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8114
[17:45:09] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for swiftproxy161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8206
[17:45:13] <icinga-miraheze>	 RECOVERY - urbanshade.org - LetsEncrypt on sslhost is OK: OK - Certificate 'urbanshade.org' will expire on Sat 26 Oct 2024 03:54:50 PM GMT +0000.
[17:45:14] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for reports171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8205
[17:45:14] <icinga-miraheze>	 RECOVERY - *.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[17:45:18] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for matomo151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8203
[17:45:18] <icinga-miraheze>	 RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 11%
[17:45:19] <icinga-miraheze>	 RECOVERY - issue-tracker.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[17:45:19] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw172 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8118
[17:45:24] <icinga-miraheze>	 RECOVERY - nintendowiki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nintendowiki.wiki' will expire on Fri 11 Oct 2024 05:10:12 PM GMT +0000.
[17:45:30] <icinga-miraheze>	 RECOVERY - baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Wed 09 Oct 2024 06:50:45 PM GMT +0000.
[17:45:32] <icinga-miraheze>	 RECOVERY - kalons-reverie.com - LetsEncrypt on sslhost is OK: OK - Certificate 'kalons-reverie.com' will expire on Mon 28 Oct 2024 10:29:48 AM GMT +0000.
[17:45:32] <icinga-miraheze>	 RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Thu 10 Oct 2024 12:01:50 AM GMT +0000.
[17:45:38] <icinga-miraheze>	 RECOVERY - spiceandwolf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiceandwolf.wiki' will expire on Fri 22 Nov 2024 04:12:57 AM GMT +0000.
[17:45:46] <icinga-miraheze>	 RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Wed 09 Oct 2024 12:29:12 PM GMT +0000.
[17:45:53] <icinga-miraheze>	 RECOVERY - christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sun 20 Oct 2024 04:51:22 PM GMT +0000.
[17:45:58] <icinga-miraheze>	 RECOVERY - rippaverse.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[17:46:03] <icinga-miraheze>	 RECOVERY - beidipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beidipedia.com' will expire on Sat 09 Nov 2024 03:43:42 PM GMT +0000.
[17:46:03] <icinga-miraheze>	 RECOVERY - ecole.science - LetsEncrypt on sslhost is OK: OK - Certificate 'ecole.science' will expire on Thu 10 Oct 2024 01:22:18 AM GMT +0000.
[17:46:04] <icinga-miraheze>	 RECOVERY - knifepointhorror.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'knifepointhorror.wiki' will expire on Fri 11 Oct 2024 12:38:56 AM GMT +0000.
[17:46:06] <icinga-miraheze>	 RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Wed 09 Oct 2024 10:29:33 AM GMT +0000.
[17:46:08] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8119
[17:46:09] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8117
[17:46:09] <icinga-miraheze>	 RECOVERY - cp36 NTP time on cp36 is OK: NTP OK: Offset -4.598498344e-05 secs
[17:46:10] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8115
[17:46:13] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw182 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8120
[17:46:15] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[17:46:15] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for swiftproxy171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8207
[17:46:16] <icinga-miraheze>	 RECOVERY - patternarchive.online - LetsEncrypt on sslhost is OK: OK - Certificate 'patternarchive.online' will expire on Sun 20 Oct 2024 04:49:30 PM GMT +0000.
[17:46:16] <icinga-miraheze>	 RECOVERY - pokemonwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'pokemonwiki.info' will expire on Sun 20 Oct 2024 04:47:56 PM GMT +0000.
[17:46:16] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for puppet181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8204
[17:46:16] <icinga-miraheze>	 RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Sun 20 Oct 2024 04:52:09 PM GMT +0000.
[17:46:20] <icinga-miraheze>	 RECOVERY - dragdown.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'dragdown.wiki' will expire on Sat 19 Oct 2024 01:47:25 PM GMT +0000.
[17:46:21] <icinga-miraheze>	 PROBLEM - cp37 NTP time on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[17:46:22] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mwtask181 on cp36 is OK: TCP OK - 0.001 second response time on localhost port 8160
[17:46:22] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw162 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8116
[17:46:26] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mon181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8201
[17:46:26] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for test151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8181
[17:46:29] <icinga-miraheze>	 RECOVERY - mockgovernments.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mockgovernments.com' will expire on Sun 03 Nov 2024 07:20:56 PM GMT +0000.
[17:46:29] <icinga-miraheze>	 RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:46:35] <icinga-miraheze>	 RECOVERY - terraforming.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'terraforming.wiki' will expire on Wed 25 Sep 2024 08:31:50 PM GMT +0000.
[17:46:36] <icinga-miraheze>	 RECOVERY - encyclopediarobotica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediarobotica.org' will expire on Tue 26 Nov 2024 03:12:49 PM GMT +0000.
[17:46:38] <icinga-miraheze>	 RECOVERY - quadball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'quadball.wiki' will expire on Sat 09 Nov 2024 03:38:36 PM GMT +0000.
[17:46:39] <icinga-miraheze>	 RECOVERY - thegreatwar.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thegreatwar.uk' will expire on Wed 09 Oct 2024 07:05:29 PM GMT +0000.
[17:46:39] <icinga-miraheze>	 RECOVERY - kingdomdeath.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kingdomdeath.wiki' will expire on Wed 16 Oct 2024 10:25:08 AM GMT +0000.
[17:46:45] <icinga-miraheze>	 RECOVERY - removededm.com - LetsEncrypt on sslhost is OK: OK - Certificate 'removededm.com' will expire on Mon 25 Nov 2024 01:47:38 PM GMT +0000.
[17:46:47] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for reports171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8205
[17:46:51] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for swiftproxy161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8206
[17:46:53] <icinga-miraheze>	 RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 08 Oct 2024 10:59:51 PM GMT +0000.
[17:46:53] <icinga-miraheze>	 RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4006 bytes in 0.291 second response time
[17:46:53] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for phorge171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8202
[17:46:53] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8113
[17:46:54] <icinga-miraheze>	 RECOVERY - royal-wiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'royal-wiki.org' will expire on Fri 11 Oct 2024 01:33:31 AM GMT +0000.
[17:46:57] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for matomo151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8203
[17:46:58] <icinga-miraheze>	 RECOVERY - landofliberos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'landofliberos.com' will expire on Fri 11 Oct 2024 10:16:43 AM GMT +0000.
[17:46:58] <icinga-miraheze>	 RECOVERY - thestarsareright.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thestarsareright.org' will expire on Thu 10 Oct 2024 12:17:46 PM GMT +0000.
[17:46:59] <icinga-miraheze>	 RECOVERY - volunteerforukraine.com - LetsEncrypt on sslhost is OK: OK - Certificate 'volunteerforukraine.com' will expire on Thu 21 Nov 2024 04:27:53 PM GMT +0000.
[17:47:00] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw152 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8114
[17:47:01] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8115
[17:47:02] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for test151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8181
[17:47:05] <icinga-miraheze>	 RECOVERY - archivesofhavnor.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archivesofhavnor.org' will expire on Mon 25 Nov 2024 03:33:55 PM GMT +0000.
[17:47:06] <icinga-miraheze>	 RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Wed 09 Oct 2024 08:44:19 AM GMT +0000.
[17:47:13] <icinga-miraheze>	 RECOVERY - cheeseepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cheeseepedia.org' will expire on Fri 04 Oct 2024 09:40:03 AM GMT +0000.
[17:47:16] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw162 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8116
[17:47:18] <icinga-miraheze>	 RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Wed 09 Oct 2024 09:56:34 AM GMT +0000.
[17:47:19] <icinga-miraheze>	 RECOVERY - rodzinka.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rodzinka.wiki' will expire on Fri 29 Nov 2024 08:52:05 PM GMT +0000.
[17:47:20] <icinga-miraheze>	 RECOVERY - phighting.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'phighting.wiki' will expire on Fri 01 Nov 2024 08:04:07 PM GMT +0000.
[17:47:24] <icinga-miraheze>	 RECOVERY - baligawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'baligawiki.org' will expire on Wed 06 Nov 2024 08:35:03 PM GMT +0000.
[17:47:25] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mwtask171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8161
[17:47:27] <icinga-miraheze>	 RECOVERY - rainverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rainverse.wiki' will expire on Fri 22 Nov 2024 01:56:56 AM GMT +0000.
[17:47:28] <icinga-miraheze>	 RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Thu 10 Oct 2024 12:53:31 AM GMT +0000.
[17:47:38] <icinga-miraheze>	 RECOVERY - evilgeniuswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'evilgeniuswiki.com' will expire on Sun 13 Oct 2024 01:35:51 PM GMT +0000.
[17:47:46] <icinga-miraheze>	 RECOVERY - lads.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lads.wiki' will expire on Tue 08 Oct 2024 06:14:12 PM GMT +0000.
[17:47:51] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:47:51] <icinga-miraheze>	 RECOVERY - n64brew.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'n64brew.dev' will expire on Wed 09 Oct 2024 07:43:10 PM GMT +0000.
[17:47:52] <icinga-miraheze>	 RECOVERY - partopedia.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'partopedia.ru' will expire on Wed 25 Sep 2024 07:57:11 PM GMT +0000.
[17:47:58] <icinga-miraheze>	 RECOVERY - crashspyrowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crashspyrowiki.com' will expire on Fri 11 Oct 2024 05:04:46 PM GMT +0000.
[17:48:00] <icinga-miraheze>	 RECOVERY - squareenixwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'squareenixwiki.com' will expire on Fri 11 Oct 2024 01:52:06 AM GMT +0000.
[17:48:01] <icinga-miraheze>	 RECOVERY - osdev.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'osdev.wiki' will expire on Sun 24 Nov 2024 10:39:44 PM GMT +0000.
[17:48:08] <icinga-miraheze>	 RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 09 Oct 2024 10:18:01 AM GMT +0000.
[17:48:10] <icinga-miraheze>	 RECOVERY - starveil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'starveil.wiki' will expire on Tue 26 Nov 2024 04:32:11 PM GMT +0000.
[17:48:13] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw172 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8118
[17:48:14] <icinga-miraheze>	 RECOVERY - johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Sun 20 Oct 2024 05:00:59 PM GMT +0000.
[17:48:14] <icinga-miraheze>	 RECOVERY - wiki.encyclopbd.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.encyclopbd.info' will expire on Thu 31 Oct 2024 03:41:19 PM GMT +0000.
[17:48:15] <icinga-miraheze>	 RECOVERY - aryavratpedia.co - LetsEncrypt on sslhost is OK: OK - Certificate 'aryavratpedia.co' will expire on Fri 15 Nov 2024 03:38:29 AM GMT +0000.
[17:48:16] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8119
[17:48:17] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy
[17:48:17] <icinga-miraheze>	 RECOVERY - familiacorsi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'familiacorsi.com' will expire on Tue 22 Oct 2024 05:16:02 PM GMT +0000.
[17:48:18] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for swiftproxy171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8207
[17:48:18] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mon181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8201
[17:48:18] <icinga-miraheze>	 RECOVERY - cp37 PowerDNS Recursor on cp37 is OK: DNS OK: 0.330 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:48:19] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001507401466 secs
[17:48:20] <icinga-miraheze>	 RECOVERY - cp37 NTP time on cp37 is OK: NTP OK: Offset 8.735060692e-05 secs
[17:48:20] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mwtask181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8160
[17:48:21] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for puppet181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8204
[17:48:24] <icinga-miraheze>	 RECOVERY - rarewarewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewarewiki.com' will expire on Fri 11 Oct 2024 05:28:02 PM GMT +0000.
[17:48:27] <icinga-miraheze>	 RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Mon 21 Oct 2024 03:28:07 AM GMT +0000.
[17:48:28] <icinga-miraheze>	 RECOVERY - ff8.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ff8.wiki' will expire on Mon 02 Dec 2024 01:38:11 AM GMT +0000.
[17:48:29] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[17:48:29] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw182 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8120
[17:48:31] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8117
[17:48:38] <icinga-miraheze>	 RECOVERY - project-patterns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'project-patterns.com' will expire on Sat 23 Nov 2024 02:04:38 AM GMT +0000.
[17:48:41] <icinga-miraheze>	 RECOVERY - rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Wed 09 Oct 2024 11:31:31 PM GMT +0000.
[17:48:53] <icinga-miraheze>	 RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Tue 08 Oct 2024 10:25:55 PM GMT +0000.
[17:48:54] <icinga-miraheze>	 RECOVERY - fanojo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fanojo.wiki' will expire on Tue 22 Oct 2024 01:41:50 AM GMT +0000.
[17:48:56] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Thu 10 Oct 2024 07:27:39 PM GMT +0000.
[17:48:56] <icinga-miraheze>	 RECOVERY - dragonquestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dragonquestwiki.com' will expire on Thu 10 Oct 2024 02:39:49 PM GMT +0000.
[17:48:57] <icinga-miraheze>	 RECOVERY - aryavartpedia.online - LetsEncrypt on sslhost is OK: OK - Certificate 'aryavartpedia.online' will expire on Thu 07 Nov 2024 12:38:18 AM GMT +0000.
[17:49:05] <icinga-miraheze>	 RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Wed 09 Oct 2024 09:43:14 AM GMT +0000.
[17:49:05] <icinga-miraheze>	 RECOVERY - vesc.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vesc.wiki' will expire on Sun 13 Oct 2024 03:12:52 PM GMT +0000.
[17:49:09] <icinga-miraheze>	 RECOVERY - neurologiahpm.com - LetsEncrypt on sslhost is OK: OK - Certificate 'neurologiahpm.com' will expire on Sun 29 Sep 2024 12:37:53 PM GMT +0000.
[17:49:11] <icinga-miraheze>	 RECOVERY - hololive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hololive.wiki' will expire on Wed 09 Oct 2024 07:44:53 PM GMT +0000.
[17:49:12] <icinga-miraheze>	 RECOVERY - yokaiwatchwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yokaiwatchwiki.com' will expire on Fri 11 Oct 2024 06:09:00 PM GMT +0000.
[17:49:14] <icinga-miraheze>	 RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Wed 09 Oct 2024 04:35:15 PM GMT +0000.
[17:49:14] <icinga-miraheze>	 RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Fri 08 Nov 2024 02:38:28 AM GMT +0000.
[17:49:15] <icinga-miraheze>	 RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:49:20] <icinga-miraheze>	 RECOVERY - pyramidgames.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pyramidgames.wiki' will expire on Sun 20 Oct 2024 04:56:18 PM GMT +0000.
[17:49:26] <icinga-miraheze>	 RECOVERY - cpmusicwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'cpmusicwiki.com' will expire on Sat 16 Nov 2024 12:23:53 PM GMT +0000.
[17:49:26] <icinga-miraheze>	 RECOVERY - smashbroswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'smashbroswiki.com' will expire on Thu 10 Oct 2024 02:38:52 PM GMT +0000.
[17:49:27] <icinga-miraheze>	 RECOVERY - apeirology.com - LetsEncrypt on sslhost is OK: OK - Certificate 'apeirology.com' will expire on Fri 11 Oct 2024 05:36:17 PM GMT +0000.
[17:49:40] <icinga-miraheze>	 RECOVERY - hardcore.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hardcore.wiki' will expire on Fri 27 Sep 2024 05:17:46 PM GMT +0000.
[17:49:40] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sun 20 Oct 2024 04:53:18 PM GMT +0000.
[17:49:41] <icinga-miraheze>	 RECOVERY - solaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'solaswiki.org' will expire on Fri 11 Oct 2024 12:58:53 PM GMT +0000.
[17:49:45] <icinga-miraheze>	 RECOVERY - puzzles.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.puzzles.wiki' will expire on Thu 10 Oct 2024 09:52:21 AM GMT +0000.
[17:49:52] <icinga-miraheze>	 RECOVERY - xomnipedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'xomnipedia.org' will expire on Thu 17 Oct 2024 09:33:16 AM GMT +0000.
[17:49:54] <icinga-miraheze>	 RECOVERY - wikimoma.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimoma.art' will expire on Thu 10 Oct 2024 10:39:49 AM GMT +0000.
[17:50:00] <icinga-miraheze>	 RECOVERY - freedomplanetwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'freedomplanetwiki.com' will expire on Fri 11 Oct 2024 06:21:55 PM GMT +0000.
[17:50:00] <icinga-miraheze>	 RECOVERY - rippaversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rippaversewiki.com' will expire on Thu 31 Oct 2024 06:35:20 PM GMT +0000.
[17:50:02] <icinga-miraheze>	 RECOVERY - coffeewiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'coffeewiki.net' will expire on Sun 24 Nov 2024 10:57:49 PM GMT +0000.
[17:50:06] <icinga-miraheze>	 RECOVERY - wikappedia.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'wikappedia.cc' will expire on Fri 25 Oct 2024 03:21:01 PM GMT +0000.
[17:50:09] <icinga-miraheze>	 RECOVERY - kingdomway.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kingdomway.wiki' will expire on Fri 15 Nov 2024 07:42:18 AM GMT +0000.
[17:50:11] <icinga-miraheze>	 RECOVERY - tintinpedia.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'tintinpedia.fr' will expire on Wed 16 Oct 2024 09:40:27 AM GMT +0000.
[17:50:12] <icinga-miraheze>	 RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 27 Sep 2024 11:58:22 PM GMT +0000.
[17:50:12] <icinga-miraheze>	 RECOVERY - trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'trollpasta.com' will expire on Sun 20 Oct 2024 04:49:54 PM GMT +0000.
[17:50:13] <icinga-miraheze>	 RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Fri 11 Oct 2024 12:58:02 AM GMT +0000.
[17:50:13] <icinga-miraheze>	 RECOVERY - oxboxtra.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'oxboxtra.wiki' will expire on Fri 15 Nov 2024 07:42:02 AM GMT +0000.
[17:50:14] <icinga-miraheze>	 RECOVERY - iol.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'iol.wiki' will expire on Fri 11 Oct 2024 01:25:12 AM GMT +0000.
[17:50:15] <icinga-miraheze>	 RECOVERY - bristolstudenthousingcoop.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bristolstudenthousingcoop.org' will expire on Wed 09 Oct 2024 12:01:13 PM GMT +0000.
[17:50:19] <icinga-miraheze>	 RECOVERY - isocasg.org - Cloudflare on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Sat 27 Aug 2039 04:37:00 PM GMT +0000.
[17:50:25] <icinga-miraheze>	 RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Fri 11 Oct 2024 01:04:25 PM GMT +0000.
[17:50:39] <icinga-miraheze>	 RECOVERY - kodiak.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kodiak.wiki' will expire on Mon 11 Nov 2024 04:12:08 AM GMT +0000.
[17:50:55] <icinga-miraheze>	 RECOVERY - clubpenguinfanon.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'clubpenguinfanon.wiki' will expire on Mon 28 Oct 2024 12:43:18 AM GMT +0000.
[17:51:10] <icinga-miraheze>	 RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Sun 13 Oct 2024 01:32:41 PM GMT +0000.
[17:51:30] <icinga-miraheze>	 RECOVERY - farthestfrontier.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'farthestfrontier.wiki' will expire on Fri 29 Nov 2024 08:54:27 PM GMT +0000.
[17:51:35] <icinga-miraheze>	 RECOVERY - corru.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'corru.wiki' will expire on Sat 09 Nov 2024 03:33:02 AM GMT +0000.
[17:51:40] <icinga-miraheze>	 RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Wed 09 Oct 2024 08:01:13 AM GMT +0000.
[17:51:40] <icinga-miraheze>	 RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Tue 08 Oct 2024 09:57:49 PM GMT +0000.
[17:52:00] <icinga-miraheze>	 RECOVERY - looneypyramids.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'looneypyramids.wiki' will expire on Fri 11 Oct 2024 09:34:01 AM GMT +0000.
[17:52:01] <icinga-miraheze>	 RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Wed 09 Oct 2024 09:14:31 AM GMT +0000.
[17:52:42] <icinga-miraheze>	 RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Wed 09 Oct 2024 05:40:21 PM GMT +0000.
[17:52:51] <icinga-miraheze>	 RECOVERY - segawiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'segawiki.com' will expire on Fri 11 Oct 2024 04:01:01 PM GMT +0000.
[17:52:52] <icinga-miraheze>	 RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Wed 09 Oct 2024 12:04:15 PM GMT +0000.
[17:52:53] <icinga-miraheze>	 RECOVERY - burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Sun 20 Oct 2024 05:00:20 PM GMT +0000.
[17:52:55] <icinga-miraheze>	 RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Thu 10 Oct 2024 04:29:46 PM GMT +0000.
[17:52:58] <icinga-miraheze>	 RECOVERY - mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Fri 04 Oct 2024 01:22:23 AM GMT +0000.
[17:53:00] <icinga-miraheze>	 RECOVERY - mariopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mariopedia.org' will expire on Fri 11 Oct 2024 12:06:46 AM GMT +0000.
[17:53:05] <icinga-miraheze>	 RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Fri 04 Oct 2024 04:16:45 PM GMT +0000.
[17:53:21] <icinga-miraheze>	 RECOVERY - junxions.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'junxions.wiki' will expire on Mon 07 Oct 2024 04:04:33 PM GMT +0000.
[17:53:32] <icinga-miraheze>	 RECOVERY - avas.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'avas.wiki' will expire on Thu 10 Oct 2024 11:47:08 AM GMT +0000.
[17:53:39] <icinga-miraheze>	 RECOVERY - grayzonewarfare.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'grayzonewarfare.wiki' will expire on Sat 09 Nov 2024 08:51:50 PM GMT +0000.
[17:53:40] <icinga-miraheze>	 RECOVERY - yepedia.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'yepedia.xyz' will expire on Mon 25 Nov 2024 03:32:36 PM GMT +0000.
[17:53:43] <icinga-miraheze>	 RECOVERY - osrwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'osrwiki.org' will expire on Sat 09 Nov 2024 09:06:39 PM GMT +0000.
[17:53:45] <icinga-miraheze>	 RECOVERY - bobobay.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bobobay.wiki' will expire on Fri 29 Nov 2024 02:57:01 PM GMT +0000.
[17:53:48] <icinga-miraheze>	 RECOVERY - holocron.net - LetsEncrypt on sslhost is OK: OK - Certificate 'holocron.net' will expire on Sat 28 Sep 2024 04:24:05 AM GMT +0000.
[17:53:57] <icinga-miraheze>	 RECOVERY - witchingworld.org - LetsEncrypt on sslhost is OK: OK - Certificate 'witchingworld.org' will expire on Fri 11 Oct 2024 07:17:44 PM GMT +0000.
[17:54:04] <icinga-miraheze>	 RECOVERY - yoshipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yoshipedia.com' will expire on Mon 18 Nov 2024 07:23:13 PM GMT +0000.
[17:54:04] <icinga-miraheze>	 RECOVERY - worldless.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'worldless.wiki' will expire on Tue 24 Sep 2024 08:16:23 AM GMT +0000.
[17:54:12] <icinga-miraheze>	 RECOVERY - otherkin.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'otherkin.wiki' will expire on Sat 09 Nov 2024 03:36:03 PM GMT +0000.
[17:54:12] <icinga-miraheze>	 RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Fri 11 Oct 2024 01:07:26 AM GMT +0000.
[17:54:23] <icinga-miraheze>	 RECOVERY - agesofconflict.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'agesofconflict.wiki' will expire on Sat 12 Oct 2024 05:02:22 PM GMT +0000.
[17:54:32] <icinga-miraheze>	 RECOVERY - grayravens.com - LetsEncrypt on sslhost is OK: OK - Certificate 'grayravens.com' will expire on Thu 10 Oct 2024 05:05:08 PM GMT +0000.
[17:54:36] <icinga-miraheze>	 RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Thu 10 Oct 2024 05:09:32 PM GMT +0000.
[17:54:41] <icinga-miraheze>	 RECOVERY - lostidols.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lostidols.wiki' will expire on Sun 20 Oct 2024 04:54:19 PM GMT +0000.
[17:55:12] <icinga-miraheze>	 RECOVERY - jackbox.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'jackbox.wiki' will expire on Mon 25 Nov 2024 03:33:03 PM GMT +0000.
[17:55:27] <icinga-miraheze>	 RECOVERY - lebork.info.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'lebork.info.pl' will expire on Fri 27 Sep 2024 11:25:33 PM GMT +0000.
[17:55:30] <icinga-miraheze>	 RECOVERY - webkinzguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'webkinzguide.com' will expire on Fri 11 Oct 2024 12:06:14 PM GMT +0000.
[17:55:32] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Wed 09 Oct 2024 07:51:00 AM GMT +0000.
[17:55:33] <icinga-miraheze>	 RECOVERY - inourownwords.online - LetsEncrypt on sslhost is OK: OK - Certificate 'inourownwords.online' will expire on Thu 10 Oct 2024 10:32:38 AM GMT +0000.
[17:55:36] <icinga-miraheze>	 RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Wed 09 Oct 2024 08:07:14 AM GMT +0000.
[17:55:40] <icinga-miraheze>	 RECOVERY - psycho.engineering - LetsEncrypt on sslhost is OK: OK - Certificate 'psycho.engineering' will expire on Sun 13 Oct 2024 01:16:15 PM GMT +0000.
[17:55:47] <icinga-miraheze>	 RECOVERY - hastursnotebook.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hastursnotebook.org' will expire on Sat 12 Oct 2024 09:07:42 AM GMT +0000.
[17:56:01] <icinga-miraheze>	 RECOVERY - exordium.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'exordium.wiki' will expire on Fri 29 Nov 2024 08:53:51 PM GMT +0000.
[17:56:21] <icinga-miraheze>	 RECOVERY - kagaga.jp - LetsEncrypt on sslhost is OK: OK - Certificate 'kagaga.jp' will expire on Tue 22 Oct 2024 06:13:40 PM GMT +0000.
[17:56:39] <icinga-miraheze>	 RECOVERY - echoes-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'echoes-wiki.com' will expire on Tue 22 Oct 2024 05:50:06 PM GMT +0000.
[17:56:50] <icinga-miraheze>	 RECOVERY - creativecommons.id - LetsEncrypt on sslhost is OK: OK - Certificate 'creativecommons.id' will expire on Fri 11 Oct 2024 07:15:53 PM GMT +0000.
[17:57:01] <icinga-miraheze>	 RECOVERY - exmormon.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'exmormon.wiki' will expire on Mon 25 Nov 2024 03:33:23 PM GMT +0000.
[17:57:08] <icinga-miraheze>	 RECOVERY - ncuwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ncuwiki.com' will expire on Fri 11 Oct 2024 06:48:29 PM GMT +0000.
[17:57:09] <icinga-miraheze>	 RECOVERY - polcompball.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[17:57:24] <icinga-miraheze>	 RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Mon 14 Oct 2024 10:25:53 AM GMT +0000.
[17:57:29] <icinga-miraheze>	 RECOVERY - infectowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'infectowiki.com' will expire on Wed 09 Oct 2024 04:22:42 PM GMT +0000.
[17:57:40] <icinga-miraheze>	 RECOVERY - wikimas.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimas.kr' will expire on Thu 10 Oct 2024 11:18:42 AM GMT +0000.
[17:58:05] <icinga-miraheze>	 RECOVERY - dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Wed 30 Oct 2024 11:07:50 PM GMT +0000.
[17:58:26] <icinga-miraheze>	 RECOVERY - pandorastale.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pandorastale.wiki' will expire on Sun 03 Nov 2024 07:10:54 PM GMT +0000.
[17:59:15] <icinga-miraheze>	 RECOVERY - richterian.com - LetsEncrypt on sslhost is OK: OK - Certificate 'richterian.com' will expire on Sun 24 Nov 2024 09:20:13 PM GMT +0000.
[17:59:42] <icinga-miraheze>	 RECOVERY - rct.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rct.wiki' will expire on Thu 10 Oct 2024 11:11:19 AM GMT +0000.
[18:00:30] <icinga-miraheze>	 RECOVERY - baubles.me - LetsEncrypt on sslhost is OK: OK - Certificate 'baubles.me' will expire on Mon 23 Sep 2024 08:39:48 PM GMT +0000.
[18:00:38] <icinga-miraheze>	 RECOVERY - wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[18:11:00] <icinga-miraheze>	 RECOVERY - pso2ngs.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pso2ngs.wiki' will expire on Tue 05 Nov 2024 12:06:00 PM GMT +0000.
[18:11:16] <icinga-miraheze>	 RECOVERY - persist.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'persist.wiki' will expire on Fri 11 Oct 2024 12:21:52 AM GMT +0000.
[18:11:26] <icinga-miraheze>	 RECOVERY - poupedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'poupedia.com' will expire on Fri 29 Nov 2024 02:53:23 PM GMT +0000.
[18:11:27] <icinga-miraheze>	 RECOVERY - tssm.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tssm.wiki' will expire on Fri 27 Sep 2024 04:56:20 PM GMT +0000.
[18:11:38] <icinga-miraheze>	 RECOVERY - dcmultiverse.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[18:11:44] <icinga-miraheze>	 RECOVERY - hoseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hoseki.wiki' will expire on Thu 10 Oct 2024 02:28:28 PM GMT +0000.
[18:12:05] <icinga-miraheze>	 RECOVERY - keitaiwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'keitaiwiki.com' will expire on Sun 13 Oct 2024 03:13:23 PM GMT +0000.
[18:12:05] <icinga-miraheze>	 RECOVERY - osfirsttimer.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'osfirsttimer.wiki' will expire on Wed 25 Sep 2024 08:44:16 PM GMT +0000.
[18:12:06] <icinga-miraheze>	 RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Thu 10 Oct 2024 09:37:26 AM GMT +0000.
[18:12:10] <icinga-miraheze>	 RECOVERY - metroidpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'metroidpedia.com' will expire on Fri 11 Oct 2024 04:46:44 PM GMT +0000.
[18:12:14] <icinga-miraheze>	 RECOVERY - wikitide.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Fri 29 Nov 2024 09:15:37 PM GMT +0000.
[18:12:25] <icinga-miraheze>	 RECOVERY - wonderfuleveryday.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wonderfuleveryday.org' will expire on Tue 03 Dec 2024 02:15:11 AM GMT +0000.
[18:12:29] <icinga-miraheze>	 RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Wed 09 Oct 2024 09:04:11 AM GMT +0000.
[18:12:42] <icinga-miraheze>	 RECOVERY - wikigenius.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikigenius.org' will expire on Sat 09 Nov 2024 11:47:19 AM GMT +0000.
[18:12:44] <icinga-miraheze>	 RECOVERY - fashiondreamer.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fashiondreamer.wiki' will expire on Wed 16 Oct 2024 02:56:55 PM GMT +0000.
[18:12:45] <icinga-miraheze>	 RECOVERY - fanonpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanonpedia.com' will expire on Sat 02 Nov 2024 01:58:33 PM GMT +0000.
[18:13:01] <icinga-miraheze>	 RECOVERY - gimkit.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gimkit.wiki' will expire on Mon 07 Oct 2024 04:03:17 PM GMT +0000.
[18:13:08] <icinga-miraheze>	 RECOVERY - theenglishlakes.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'theenglishlakes.uk' will expire on Mon 18 Nov 2024 07:10:01 PM GMT +0000.
[18:13:11] <icinga-miraheze>	 RECOVERY - comprehensibleinputwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'comprehensibleinputwiki.org' will expire on Thu 28 Nov 2024 03:15:37 PM GMT +0000.
[18:13:13] <icinga-miraheze>	 RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Tue 08 Oct 2024 10:51:41 PM GMT +0000.
[18:13:15] <icinga-miraheze>	 RECOVERY - beyondtheveilrp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beyondtheveilrp.com' will expire on Wed 25 Sep 2024 08:32:11 PM GMT +0000.
[18:13:37] <icinga-miraheze>	 RECOVERY - schizoidnightmares.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'schizoidnightmares.wiki' will expire on Fri 11 Oct 2024 04:55:15 PM GMT +0000.
[18:13:37] <icinga-miraheze>	 RECOVERY - dc.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[18:13:38] <icinga-miraheze>	 RECOVERY - geeu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'geeu.org' will expire on Wed 27 Nov 2024 09:21:16 PM GMT +0000.
[18:14:15] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:18:28] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -9.825825691e-05 secs
[18:22:48] <botifico>	 [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1b73d17ec33d...0ea7db2074c5
[18:22:50] <botifico>	 [02ssl] 07MacFan4000 030ea7db2 - -irad.wiki as domain is parked
[18:23:44] <botifico>	 [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/0ea7db2074c5...93af59ee555d
[18:23:47] <botifico>	 [02ssl] 07MacFan4000 0393af59e - Delete certificates/irad.wiki.crt
[18:25:25] <botifico>	 [02dns] 07MacFan4000 opened pull request 03#607: remove irad.wiki zone - 13https://github.com/miraheze/dns/pull/607
[18:38:35] <icinga-miraheze>	 PROBLEM - db151 MariaDB Connections on db151 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne
[18:38:35] <icinga-miraheze>	  on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66
[18:38:36] <icinga-miraheze>	 PROBLEM - db151 NTP time on db151 is CRITICAL: NTP CRITICAL: No response from NTP server
[18:38:38] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw171
[18:38:41] <icinga-miraheze>	 PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne
[18:38:41] <icinga-miraheze>	  on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66
[18:38:55] <icinga-miraheze>	 PROBLEM - rdb151 PowerDNS Recursor on rdb151 is CRITICAL: Domain 'wikitide.net' was not found by the server
[18:39:14] <icinga-miraheze>	 PROBLEM - cloud17 SSH on cloud17 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:39:22] <icinga-miraheze>	 PROBLEM - jobchron171 NTP time on jobchron171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:39:23] <icinga-miraheze>	 PROBLEM - os162 NTP time on os162 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:39:36] <icinga-miraheze>	 PROBLEM - swiftobject161 PowerDNS Recursor on swiftobject161 is CRITICAL: No response from DNS ::1
[18:39:46] <icinga-miraheze>	 PROBLEM - bast161 PowerDNS Recursor on bast161 is CRITICAL: No response from DNS ::1
[18:39:46] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized
[18:39:46] <icinga-miraheze>	 PROBLEM - reports171 NTP time on reports171 is CRITICAL: NTP CRITICAL: No response from NTP server
[18:39:51] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2407:3641:2161:9774::1/cpweb, 2602:294:0:b13::110/cpweb, 2a02:c206:2161:9253::1/cpweb, 2a02:c206:2162:6187::1/cpweb, 2400:d320:2161:9775::1/cpweb
[18:39:52] <icinga-miraheze>	 PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized
[18:40:01] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 100%
[18:40:02] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTP on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized
[18:40:05] <icinga-miraheze>	 PROBLEM - rdb151 NTP time on rdb151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:40:15] <icinga-miraheze>	 PROBLEM - swiftobject161 NTP time on swiftobject161 is CRITICAL: NTP CRITICAL: No response from NTP server
[18:40:17] <icinga-miraheze>	 PROBLEM - mem161 NTP time on mem161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:40:23] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw171
[18:40:25] <RhinosF1>	 I swear you suck today
[18:40:27] <icinga-miraheze>	 PROBLEM - db171 NTP time on db171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:40:29] <icinga-miraheze>	 PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne
[18:40:29] <icinga-miraheze>	  on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66
[18:40:29] <icinga-miraheze>	 PROBLEM - mw171 NTP time on mw171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:40:31] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw171
[18:40:40] <icinga-miraheze>	 PROBLEM - puppet181 NTP time on puppet181 is CRITICAL: NTP CRITICAL: No response from NTP server
[18:40:41] <icinga-miraheze>	 PROBLEM - phorge171 NTP time on phorge171 is CRITICAL: NTP CRITICAL: No response from NTP server
[18:40:46] <icinga-miraheze>	 PROBLEM - swiftproxy161 HTTP on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized
[18:40:50] <icinga-miraheze>	 PROBLEM - db151 NTP time on db151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:40:57] <icinga-miraheze>	 PROBLEM - swiftac171 NTP time on swiftac171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:40:58] <icinga-miraheze>	 PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne
[18:40:58] <icinga-miraheze>	  on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66
[18:41:04] <icinga-miraheze>	 PROBLEM - Host cloud17 is DOWN: PING CRITICAL - Packet loss = 100%
[18:41:05] <icinga-miraheze>	 PROBLEM - os162 PowerDNS Recursor on os162 is CRITICAL: Domain 'wikitide.net' was not found by the server
[18:41:16] <icinga-miraheze>	 PROBLEM - bots171 NTP time on bots171 is CRITICAL: NTP CRITICAL: No response from NTP server
[18:41:16] <icinga-miraheze>	 PROBLEM - ldap171 NTP time on ldap171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:41:16] <icinga-miraheze>	 PROBLEM - swiftobject151 PowerDNS Recursor on swiftobject151 is CRITICAL: Domain 'wikitide.net' was not found by the server
[18:41:17] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw171
[18:41:22] <icinga-miraheze>	 PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: No response from DNS ::1
[18:41:30] <RhinosF1>	 Bad servers
[18:41:32] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:41:32] <icinga-miraheze>	 PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: No response from DNS ::1
[18:41:33] <icinga-miraheze>	 PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: Domain 'wikitide.net' was not found by the server
[18:41:33] <icinga-miraheze>	 PROBLEM - ldap171 LDAP on ldap171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:41:42] <icinga-miraheze>	 PROBLEM - puppet181 PowerDNS Recursor on puppet181 is CRITICAL: No response from DNS ::1
[18:44:42] <botifico>	 [02puppet] 07Universal-Omega created branch 03mattermost - 13https://github.com/miraheze/puppet
[18:44:43] <botifico>	 [02puppet] 07Universal-Omega created branch 03mattermost 13https://github.com/miraheze/puppet/commit/631c19c15cdcc4b6cfe721f585ff7bdcd655c8a9
[18:47:42] <MacFan4000>	 Ugh again
[18:48:08] <RhinosF1>	 Yup
[18:48:16] <RhinosF1>	 I pinged void and Alex
[18:51:44] <RhinosF1>	 Void is aware and looking
[18:55:17] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03mattermost [+69/-0/±0] 13https://github.com/miraheze/puppet/compare/631c19c15cdc...3512cdee9dec
[18:55:19] <botifico>	 [02puppet] 07Universal-Omega 033512cde - Add mattermost module
[18:56:03] <botifico>	 [02puppet] 07Universal-Omega opened pull request 03#3913: Add mattermost module - 13https://github.com/miraheze/puppet/pull/3913
[18:56:09] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3913: Add mattermost module - 13https://github.com/miraheze/puppet/pull/3913#issuecomment-2329765109
[18:56:46] <botifico>	 [02puppet] 07github-actions[bot] pushed 031 commit to 03mattermost [+0/-0/±5] 13https://github.com/miraheze/puppet/compare/3512cdee9dec...c10dc2a9d774
[18:56:49] <botifico>	 [02puppet] 07github-actions 03c10dc2a - CI: lint puppet code to standards
[18:56:51] <botifico>	 [02puppet] 07github-actions[bot] synchronize pull request 03#3913: Add mattermost module - 13https://github.com/miraheze/puppet/pull/3913
[20:20:37] <icinga-miraheze>	 RECOVERY - db161 NTP time on db161 is OK: NTP OK: Offset 1.4513731e-05 secs
[20:20:37] <icinga-miraheze>	 RECOVERY - db181 MariaDB on db181 is OK: Uptime: 18272 Threads: 12 Questions: 8379463 Slow queries: 0 Opens: 40951 Open tables: 40881 Queries per second avg: 458.595
[20:20:37] <icinga-miraheze>	 RECOVERY - Host cp26 is UP: PING OK - Packet loss = 0%, RTA = 149.59 ms
[20:20:37] <icinga-miraheze>	 RECOVERY - wiki.consid.vn - Cloudflare on sslhost is OK: OK - Certificate 'wiki.consid.vn' will expire on Fri 29 Nov 2024 07:57:33 PM GMT +0000.
[20:20:37] <icinga-miraheze>	 RECOVERY - os151 NTP time on os151 is OK: NTP OK: Offset -0.005529403687 secs
[20:20:39] <icinga-miraheze>	 RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:20:40] <icinga-miraheze>	 RECOVERY - ldap171 LDAP on ldap171 is OK: LDAP OK - 0.146 seconds response time
[20:20:41] <icinga-miraheze>	 RECOVERY - wiki.cdntennis.ca - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cdntennis.ca' will expire on Thu 28 Nov 2024 01:38:03 PM GMT +0000.
[20:20:41] <icinga-miraheze>	 RECOVERY - db151 NTP time on db151 is OK: NTP OK: Offset -0.004268378019 secs
[20:20:41] <icinga-miraheze>	 RECOVERY - cp51 Nginx Backend for mw181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8119
[20:20:41] <icinga-miraheze>	 RECOVERY - zomvilles.pixiemeta.com - Cloudflare on sslhost is OK: OK - Certificate 'zomvilles.pixiemeta.com' will expire on Thu 28 Nov 2024 01:43:45 PM GMT +0000.
[20:20:43] <icinga-miraheze>	 RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[20:20:45] <icinga-miraheze>	 RECOVERY - www.hoolehistoryheritagesociety.org.uk - Cloudflare on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk' will expire on Thu 28 Nov 2024 01:43:26 PM GMT +0000.
[20:21:06] <icinga-miraheze>	 RECOVERY - Host cloud16 is UP: PING WARNING - Packet loss = 96%, RTA = 0.16 ms
[20:22:32] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:23:42] <icinga-miraheze>	 RECOVERY - bluearchive.wiki - Cloudflare on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Thu 28 Nov 2024 01:59:07 PM GMT +0000.
[20:23:43] <icinga-miraheze>	 RECOVERY - cp51 NTP time on cp51 is OK: NTP OK: Offset -0.000141710043 secs
[20:23:44] <icinga-miraheze>	 RECOVERY - theenglishlakes.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'theenglishlakes.uk' will expire on Mon 18 Nov 2024 07:10:01 PM GMT +0000.
[20:23:45] <icinga-miraheze>	 RECOVERY - cp51 Nginx Backend for phorge171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8202
[20:23:45] <icinga-miraheze>	 RECOVERY - wiki.controlistes.fr - Cloudflare on sslhost is OK: OK - Certificate 'wiki.controlistes.fr' will expire on Fri 29 Nov 2024 06:59:16 PM GMT +0000.
[20:23:46] <icinga-miraheze>	 RECOVERY - www.samafia.org - Cloudflare on sslhost is OK: OK - Certificate 'www.samafia.org' will expire on Thu 28 Nov 2024 01:43:35 PM GMT +0000.
[20:23:46] <icinga-miraheze>	 RECOVERY - beyondtheveilrp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beyondtheveilrp.com' will expire on Wed 25 Sep 2024 08:32:11 PM GMT +0000.
[20:23:46] <icinga-miraheze>	 RECOVERY - wiki.routedoodle.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.routedoodle.com' will expire on Thu 28 Nov 2024 01:41:39 PM GMT +0000.
[20:23:46] <icinga-miraheze>	 RECOVERY - bonesword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bonesword.wiki' will expire on Tue 12 Nov 2024 06:49:37 AM GMT +0000.
[20:23:47] <icinga-miraheze>	 RECOVERY - wiki.cogsci.live - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cogsci.live' will expire on Thu 28 Nov 2024 01:38:08 PM GMT +0000.
[20:23:47] <icinga-miraheze>	 RECOVERY - wiki.pixandbit.gay - Cloudflare on sslhost is OK: OK - Certificate 'wiki.pixandbit.gay' will expire on Thu 28 Nov 2024 01:41:28 PM GMT +0000.
[20:23:48] <icinga-miraheze>	 RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'warframestat.us' will expire on Tue 19 Nov 2024 12:19:13 AM GMT +0000.
[20:23:49] <icinga-miraheze>	 RECOVERY - cp51 APT on cp51 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[20:23:50] <icinga-miraheze>	 RECOVERY - ldap171 NTP time on ldap171 is OK: NTP OK: Offset -7.292628288e-05 secs
[20:23:50] <icinga-miraheze>	 RECOVERY - cloud15 NTP time on cloud15 is OK: NTP OK: Offset -0.002416044474 secs
[20:23:50] <icinga-miraheze>	 RECOVERY - wiki.climatechange.ai - Cloudflare on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Thu 28 Nov 2024 01:38:08 PM GMT +0000.
[20:23:50] <icinga-miraheze>	 RECOVERY - polcompball.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:23:52] <icinga-miraheze>	 RECOVERY - ethene.wiki - Cloudflare on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Thu 28 Nov 2024 01:27:58 PM GMT +0000.
[20:23:52] <icinga-miraheze>	 RECOVERY - revi.wiki - Cloudflare on sslhost is OK: OK - Certificate 'revi.wiki' will expire on Thu 28 Nov 2024 01:32:55 PM GMT +0000.
[20:23:53] <icinga-miraheze>	 RECOVERY - cp26 Nginx Backend for puppet181 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8204
[20:23:55] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 149.10 ms
[20:23:55] <icinga-miraheze>	 RECOVERY - cp51 Nginx Backend for mw162 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8116
[20:23:56] <icinga-miraheze>	 PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:24:07] <icinga-miraheze>	 RECOVERY - junxions.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:24:07] <icinga-miraheze>	 RECOVERY - vise.dayid.org - Cloudflare on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Thu 28 Nov 2024 12:48:31 PM GMT +0000.
[20:24:08] <icinga-miraheze>	 RECOVERY - mw171 NTP time on mw171 is OK: NTP OK: Offset -1.692771912e-05 secs
[20:24:10] <icinga-miraheze>	 RECOVERY - cp26 conntrack_table_size on cp26 is OK: OK: nf_conntrack is 0 % full
[20:24:12] <icinga-miraheze>	 RECOVERY - strinova.org - Cloudflare on sslhost is OK: OK - Certificate 'strinova.org' will expire on Thu 28 Nov 2024 01:34:25 PM GMT +0000.
[20:24:12] <icinga-miraheze>	 RECOVERY - eosdev.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'eosdev.xyz' will expire on Sat 05 Oct 2024 08:57:15 PM GMT +0000.
[20:24:12] <icinga-miraheze>	 RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.orain.org' will expire on Fri 29 Nov 2024 04:30:28 PM GMT +0000.
[20:24:12] <icinga-miraheze>	 RECOVERY - wiki.mastodon.kr - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Thu 28 Nov 2024 01:39:22 PM GMT +0000.
[20:24:12] <icinga-miraheze>	 RECOVERY - wiki.hacklabsr.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.hacklabsr.org' will expire on Thu 28 Nov 2024 01:38:51 PM GMT +0000.
[20:24:13] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 1.031 second response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[20:24:14] <icinga-miraheze>	 RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:24:14] <icinga-miraheze>	 RECOVERY - www.ngnl.wiki - Cloudflare on sslhost is OK: OK - Certificate 'www.ngnl.wiki' will expire on Thu 28 Nov 2024 01:43:30 PM GMT +0000.
[20:24:15] <icinga-miraheze>	 RECOVERY - cloud15 IPMI Sensors on cloud15 is OK: IPMI Status: OK
[20:24:17] <icinga-miraheze>	 RECOVERY - mem151 PowerDNS Recursor on mem151 is OK: DNS OK: 2.034 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[20:24:17] <icinga-miraheze>	 RECOVERY - Host ns2 is UP: PING OK - Packet loss = 0%, RTA = 134.53 ms
[20:24:18] <icinga-miraheze>	 RECOVERY - tdr.wiki - Cloudflare on sslhost is OK: OK - Certificate 'tdr.wiki' will expire on Thu 28 Nov 2024 01:34:35 PM GMT +0000.
[20:24:18] <icinga-miraheze>	 RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.dariawiki.org' will expire on Thu 28 Nov 2024 04:21:50 PM GMT +0000.
[20:24:19] <icinga-miraheze>	 RECOVERY - kafka181 PowerDNS Recursor on kafka181 is OK: DNS OK: 0.028 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[20:24:19] <icinga-miraheze>	 RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[20:24:19] <icinga-miraheze>	 RECOVERY - squareenix.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:24:19] <icinga-miraheze>	 RECOVERY - wiki.lifesprogress.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.lifesprogress.com' will expire on Thu 28 Nov 2024 01:39:18 PM GMT +0000.
[20:24:32] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.gab.pt.eu.org' will expire on Thu 28 Nov 2024 01:38:42 PM GMT +0000.
[20:24:32] <icinga-miraheze>	 RECOVERY - cp26 Nginx Backend for mw151 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8113
[20:24:33] <icinga-miraheze>	 RECOVERY - wiki.ssij.se - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ssij.se' will expire on Thu 28 Nov 2024 01:41:56 PM GMT +0000.
[20:24:35] <icinga-miraheze>	 PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 84 packages available for upgrade (15 critical updates).
[20:24:35] <icinga-miraheze>	 RECOVERY - tgtd.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:24:36] <icinga-miraheze>	 RECOVERY - journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Sun 20 Oct 2024 04:49:12 PM GMT +0000.
[20:24:36] <icinga-miraheze>	 RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Thu 10 Oct 2024 09:37:26 AM GMT +0000.
[20:24:37] <icinga-miraheze>	 RECOVERY - mw181 PowerDNS Recursor on mw181 is OK: DNS OK: 0.035 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[20:24:37] <icinga-miraheze>	 RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.034 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[20:24:37] <icinga-miraheze>	 RECOVERY - wiki.worldbuilding.network - Cloudflare on sslhost is OK: OK - Certificate 'wiki.worldbuilding.network' will expire on Thu 28 Nov 2024 01:42:55 PM GMT +0000.
[20:24:38] <icinga-miraheze>	 RECOVERY - cp26 SSH on cp26 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:24:40] <icinga-miraheze>	 RECOVERY - cp26 Nginx Backend for test151 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8181
[20:24:40] <icinga-miraheze>	 RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:24:40] <icinga-miraheze>	 PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:24:40] <icinga-miraheze>	 RECOVERY - cp26 Nginx Backend for mon181 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8201
[20:24:41] <icinga-miraheze>	 RECOVERY - cp51 PowerDNS Recursor on cp51 is OK: DNS OK: 0.506 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[20:24:41] <icinga-miraheze>	 RECOVERY - graphite151 NTP time on graphite151 is OK: NTP OK: Offset -0.00257691741 secs
[20:24:41] <icinga-miraheze>	 RECOVERY - bots171 NTP time on bots171 is OK: NTP OK: Offset -2.288818359e-05 secs
[20:24:48] <icinga-miraheze>	 RECOVERY - wiki.sequoiafabrica.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.sequoiafabrica.org' will expire on Thu 28 Nov 2024 01:41:45 PM GMT +0000.
[20:24:52] <icinga-miraheze>	 RECOVERY - wiki.oasismp.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.oasismp.com' will expire on Thu 28 Nov 2024 01:39:40 PM GMT +0000.
[20:24:54] <icinga-miraheze>	 RECOVERY - wiki.msnld.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.msnld.com' will expire on Thu 28 Nov 2024 01:39:33 PM GMT +0000.
[20:25:12] <icinga-miraheze>	 RECOVERY - www.dccomicswiki.com - Cloudflare on sslhost is OK: OK - Certificate 'www.dccomicswiki.com' will expire on Fri 29 Nov 2024 04:51:43 PM GMT +0000.
[20:25:17] <icinga-miraheze>	 RECOVERY - kaylaandrachel.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:25:21] <icinga-miraheze>	 RECOVERY - zenith.nsmbu.net - Cloudflare on sslhost is OK: OK - Certificate 'zenith.nsmbu.net' will expire on Thu 28 Nov 2024 01:43:42 PM GMT +0000.
[20:25:22] <icinga-miraheze>	 RECOVERY - alternatewiki.tombricks.com - Cloudflare on sslhost is OK: OK - Certificate 'alternatewiki.tombricks.com' will expire on Thu 28 Nov 2024 01:24:51 PM GMT +0000.
[20:25:25] <icinga-miraheze>	 RECOVERY - largedu.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'largedu.eu.org' will expire on Fri 08 Nov 2024 12:52:49 AM GMT +0000.
[20:25:32] <icinga-miraheze>	 RECOVERY - nl.xliving.org - Cloudflare on sslhost is OK: OK - Certificate 'nl.xliving.org' will expire on Thu 28 Nov 2024 01:31:58 PM GMT +0000.
[20:25:34] <icinga-miraheze>	 RECOVERY - pandorastale.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:25:36] <icinga-miraheze>	 RECOVERY - inext.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:25:40] <icinga-miraheze>	 RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:25:41] <icinga-miraheze>	 RECOVERY - lostmediawiki.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'lostmediawiki.ru' will expire on Sat 09 Nov 2024 03:14:43 PM GMT +0000.
[20:25:44] <icinga-miraheze>	 RECOVERY - wiki.sadboyzpod.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.sadboyzpod.com' will expire on Thu 28 Nov 2024 01:41:42 PM GMT +0000.
[20:25:44] <icinga-miraheze>	 PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[20:25:47] <icinga-miraheze>	 PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne
[20:25:47] <icinga-miraheze>	  on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66
[20:25:48] <icinga-miraheze>	 RECOVERY - urbanshade.org - LetsEncrypt on sslhost is OK: OK - Certificate 'urbanshade.org' will expire on Sat 26 Oct 2024 03:54:50 PM GMT +0000.
[20:25:52] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 3.213 second response time
[20:25:52] <icinga-miraheze>	 RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[20:25:54] <icinga-miraheze>	 PROBLEM - mw161 ferm_active on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[20:25:54] <icinga-miraheze>	 PROBLEM - mw161 php-fpm on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[20:25:55] <icinga-miraheze>	 PROBLEM - mw161 NTP time on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[20:25:55] <icinga-miraheze>	 RECOVERY - graphite151 PowerDNS Recursor on graphite151 is OK: DNS OK: 0.310 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[20:25:55] <icinga-miraheze>	 PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[20:25:59] <icinga-miraheze>	 PROBLEM - mw162 SSH on mw162 is CRITICAL: connect to address 10.0.16.133 and port 22: No route to host
[20:25:59] <icinga-miraheze>	 PROBLEM - ping on mw162 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.133)
[20:26:02] <icinga-miraheze>	 PROBLEM - mw162 NTP time on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[20:26:02] <icinga-miraheze>	 PROBLEM - mw162 PowerDNS Recursor on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[20:26:02] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[20:26:02] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[20:26:03] <icinga-miraheze>	 RECOVERY - wiki.lefrenchmelee.fr - Cloudflare on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Thu 28 Nov 2024 01:39:13 PM GMT +0000.
[20:26:03] <icinga-miraheze>	 RECOVERY - kombinat.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kombinat.eu.org' will expire on Fri 22 Nov 2024 10:09:37 AM GMT +0000.
[20:26:04] <icinga-miraheze>	 PROBLEM - mem161 NTP time on mem161 is CRITICAL: connect to address 10.0.16.131 port 5666: No route to hostconnect to host 10.0.16.131 port 5666: No route to host
[20:26:04] <icinga-miraheze>	 PROBLEM - mem161 ferm_active on mem161 is CRITICAL: connect to address 10.0.16.131 port 5666: No route to hostconnect to host 10.0.16.131 port 5666: No route to host
[20:26:04] <icinga-miraheze>	 PROBLEM - mem161 SSH on mem161 is CRITICAL: connect to address 10.0.16.131 and port 22: No route to host
[20:26:05] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:26:05] <icinga-miraheze>	 PROBLEM - mw162 ferm_active on mw162 is CRITICAL: connect to address 10.0.16.133 port 5666: No route to hostconnect to host 10.0.16.133 port 5666: No route to host
[20:26:07] <icinga-miraheze>	 PROBLEM - mem161 memcached on mem161 is CRITICAL: connect to address 10.0.16.131 and port 11211: No route to host
[20:26:09] <icinga-miraheze>	 PROBLEM - ping on db161 is CRITICAL: CRITICAL - Host Unreachable (10.0.16.128)
[20:26:09] <icinga-miraheze>	 PROBLEM - db161 Backups SQL mhglobal on db161 is CRITICAL: connect to address 10.0.16.128 port 5666: No route to hostconnect to host 10.0.16.128 port 5666: No route to host
[20:26:10] <icinga-miraheze>	 RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.307 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[20:26:10] <icinga-miraheze>	 PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[20:26:10] <icinga-miraheze>	 PROBLEM - mem161 Current Load on mem161 is CRITICAL: connect to address 10.0.16.131 port 5666: No route to hostconnect to host 10.0.16.131 port 5666: No route to host
[20:26:12] <icinga-miraheze>	 PROBLEM - Host db161 is DOWN: CRITICAL - Host Unreachable (10.0.16.128)
[20:26:13] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.15 ms
[20:26:14] <icinga-miraheze>	 RECOVERY - christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sun 20 Oct 2024 04:51:22 PM GMT +0000.
[20:26:15] <icinga-miraheze>	 RECOVERY - ns2 SSH on ns2 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:26:15] <icinga-miraheze>	 PROBLEM - mw161 APT on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[20:26:28] <icinga-miraheze>	 PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[20:26:30] <icinga-miraheze>	 PROBLEM - cp27 HTTPS on cp27 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[20:26:31] <icinga-miraheze>	 PROBLEM - mw161 SSH on mw161 is CRITICAL: connect to address 10.0.16.132 and port 22: No route to host
[20:26:32] <icinga-miraheze>	 PROBLEM - mw161 Disk Space on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[20:26:32] <icinga-miraheze>	 PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time
[20:26:34] <icinga-miraheze>	 PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[20:26:35] <icinga-miraheze>	 RECOVERY - dc.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[20:26:35] <icinga-miraheze>	 PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[20:26:39] <icinga-miraheze>	 PROBLEM - Host mw161 is DOWN: CRITICAL - Host Unreachable (10.0.16.132)
[20:26:39] <icinga-miraheze>	 PROBLEM - mw161 conntrack_table_size on mw161 is CRITICAL: connect to address 10.0.16.132 port 5666: No route to hostconnect to host 10.0.16.132 port 5666: No route to host
[20:26:40] <icinga-miraheze>	 RECOVERY - schizoidnightmares.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'schizoidnightmares.wiki' will expire on Fri 11 Oct 2024 04:55:15 PM GMT +0000.
[20:26:40] <icinga-miraheze>	 RECOVERY - geeu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'geeu.org' will expire on Wed 27 Nov 2024 09:21:16 PM GMT +0000.
[20:26:42] <icinga-miraheze>	 PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:26:46] <icinga-miraheze>	 RECOVERY - www.dovearchives.wiki - Cloudflare on sslhost is OK: OK - Certificate 'www.dovearchives.wiki' will expire on Thu 28 Nov 2024 01:43:19 PM GMT +0000.
[20:26:48] <icinga-miraheze>	 RECOVERY - www.tintinpedia.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'www.tintinpedia.fr' will expire on Thu 28 Nov 2024 03:56:15 PM GMT +0000.
[20:26:50] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:26:56] <icinga-miraheze>	 RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'poserdazfreebies.orain.org' will expire on Fri 29 Nov 2024 04:24:24 PM GMT +0000.
[20:27:03] <icinga-miraheze>	 RECOVERY - wiki.dragonsmp.pro - Cloudflare on sslhost is OK: OK - Certificate 'wiki.dragonsmp.pro' will expire on Thu 28 Nov 2024 01:38:25 PM GMT +0000.
[20:27:07] <icinga-miraheze>	 RECOVERY - wiki.queenscourt.games - Cloudflare on sslhost is OK: OK - Certificate 'wiki.queenscourt.games' will expire on Thu 28 Nov 2024 01:41:35 PM GMT +0000.
[20:27:09] <icinga-miraheze>	 RECOVERY - with.cpt-ra.bid - Cloudflare on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Thu 28 Nov 2024 01:43:10 PM GMT +0000.
[20:27:10] <icinga-miraheze>	 RECOVERY - beidipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beidipedia.com' will expire on Sat 09 Nov 2024 03:43:42 PM GMT +0000.
[20:27:17] <icinga-miraheze>	 RECOVERY - wiki.metaiitgn.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.metaiitgn.org' will expire on Thu 28 Nov 2024 01:39:25 PM GMT +0000.
[20:27:18] <icinga-miraheze>	 RECOVERY - wiki.leapit.no - Cloudflare on sslhost is OK: OK - Certificate 'wiki.leapit.no' will expire on Fri 29 Nov 2024 07:57:56 PM GMT +0000.
[20:27:19] <icinga-miraheze>	 RECOVERY - *.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[20:27:28] <icinga-miraheze>	 RECOVERY - pizzatower.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:27:32] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:27:33] <icinga-miraheze>	 RECOVERY - evade.robtospro.com - Cloudflare on sslhost is OK: OK - Certificate 'evade.robtospro.com' will expire on Thu 28 Nov 2024 01:27:59 PM GMT +0000.
[20:27:33] <icinga-miraheze>	 RECOVERY - wiki.kachow.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.kachow.org' will expire on Fri 29 Nov 2024 07:57:55 PM GMT +0000.
[20:27:35] <icinga-miraheze>	 RECOVERY - cyberlaw.ccdcoe.org - Cloudflare on sslhost is OK: OK - Certificate 'cyberlaw.ccdcoe.org' will expire on Thu 28 Nov 2024 01:25:20 PM GMT +0000.
[20:27:42] <icinga-miraheze>	 RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:27:49] <icinga-miraheze>	 RECOVERY - wiki.joust.ro - Cloudflare on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Thu 28 Nov 2024 01:39:03 PM GMT +0000.
[20:27:50] <icinga-miraheze>	 RECOVERY - wiki.macc.nyc - Cloudflare on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Fri 29 Nov 2024 07:57:57 PM GMT +0000.
[20:27:51] <icinga-miraheze>	 RECOVERY - wiki.kscucf.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.kscucf.org' will expire on Thu 28 Nov 2024 12:48:30 PM GMT +0000.
[20:27:53] <icinga-miraheze>	 RECOVERY - www.fossilfighterswiki.com - Cloudflare on sslhost is OK: OK - Certificate 'www.fossilfighterswiki.com' will expire on Thu 28 Nov 2024 01:43:21 PM GMT +0000.
[20:28:08] <icinga-miraheze>	 RECOVERY - www.baharna.org - Cloudflare on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Thu 28 Nov 2024 04:02:31 PM GMT +0000.
[20:28:08] <icinga-miraheze>	 RECOVERY - ru.countryhumans.polandball.wiki - Cloudflare on sslhost is OK: OK - Certificate 'ru.countryhumans.polandball.wiki' will expire on Thu 28 Nov 2024 01:32:59 PM GMT +0000.
[20:28:09] <icinga-miraheze>	 RECOVERY - en.religiononfire.mar.in.ua - Cloudflare on sslhost is OK: OK - Certificate 'en.religiononfire.mar.in.ua' will expire on Thu 28 Nov 2024 01:27:10 PM GMT +0000.
[20:28:16] <icinga-miraheze>	 RECOVERY - nintendowiki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nintendowiki.wiki' will expire on Fri 11 Oct 2024 05:10:12 PM GMT +0000.
[20:28:17] <icinga-miraheze>	 RECOVERY - lophocmatngu.wiki - Cloudflare on sslhost is OK: OK - Certificate 'lophocmatngu.wiki' will expire on Thu 28 Nov 2024 11:52:55 AM GMT +0000.
[20:28:28] <icinga-miraheze>	 RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[20:28:31] <icinga-miraheze>	 RECOVERY - uk.religiononfire.mar.in.ua - Cloudflare on sslhost is OK: OK - Certificate 'uk.religiononfire.mar.in.ua' will expire on Thu 28 Nov 2024 01:36:49 PM GMT +0000.
[20:28:32] <icinga-miraheze>	 RECOVERY - wiki.digitaldesignhq.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.digitaldesignhq.com' will expire on Fri 29 Nov 2024 07:57:33 PM GMT +0000.
[20:28:33] <icinga-miraheze>	 RECOVERY - www.rippaversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rippaversewiki.com' will expire on Thu 28 Nov 2024 04:33:43 PM GMT +0000.
[20:28:38] <icinga-miraheze>	 RECOVERY - wiki.mostly.vet - LetsEncrypt on sslhost is OK: OK - Certificate 'mostly.vet' will expire on Fri 18 Oct 2024 12:05:32 AM GMT +0000.
[20:28:40] <icinga-miraheze>	 RECOVERY - kb.nena.org - Cloudflare on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Thu 28 Nov 2024 01:31:32 PM GMT +0000.
[20:28:47] <icinga-miraheze>	 RECOVERY - resources.africanvision.org - Cloudflare on sslhost is OK: OK - Certificate 'resources.africanvision.org' will expire on Fri 29 Nov 2024 07:56:19 PM GMT +0000.
[20:28:55] <icinga-miraheze>	 RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Wed 09 Oct 2024 12:29:12 PM GMT +0000.
[20:28:55] <icinga-miraheze>	 RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Wed 09 Oct 2024 09:56:34 AM GMT +0000.
[20:29:02] <icinga-miraheze>	 RECOVERY - n64brew.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'n64brew.dev' will expire on Wed 09 Oct 2024 07:43:10 PM GMT +0000.
[20:29:05] <icinga-miraheze>	 RECOVERY - maxcapacity.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'maxcapacity.wiki' will expire on Sun 22 Sep 2024 04:44:31 PM GMT +0000.
[20:29:06] <icinga-miraheze>	 RECOVERY - wiki.ivy.cm - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ivy.cm' will expire on Thu 28 Nov 2024 01:38:58 PM GMT +0000.
[20:29:07] <icinga-miraheze>	 RECOVERY - wiki.bluevertigo.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.bluevertigo.org' will expire on Fri 29 Nov 2024 07:57:29 PM GMT +0000.
[20:29:11] <icinga-miraheze>	 RECOVERY - www.lostidols.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lostidols.wiki' will expire on Fri 29 Nov 2024 07:05:01 PM GMT +0000.
[20:29:11] <icinga-miraheze>	 RECOVERY - www.ebcat.me - Cloudflare on sslhost is OK: OK - Certificate 'www.ebcat.me' will expire on Thu 28 Nov 2024 03:56:46 PM GMT +0000.
[20:29:12] <icinga-miraheze>	 RECOVERY - swests.org - Cloudflare on sslhost is OK: OK - Certificate 'swests.org' will expire on Thu 28 Nov 2024 01:34:28 PM GMT +0000.
[20:29:12] <icinga-miraheze>	 RECOVERY - reviwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Sat 16 Nov 2024 01:16:18 PM GMT +0000.
[20:29:13] <icinga-miraheze>	 RECOVERY - baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Wed 09 Oct 2024 06:50:45 PM GMT +0000.
[20:29:16] <icinga-miraheze>	 RECOVERY - issue-tracker.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[20:29:17] <icinga-miraheze>	 RECOVERY - encyclopediarobotica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediarobotica.org' will expire on Tue 26 Nov 2024 03:12:49 PM GMT +0000.
[20:29:17] <icinga-miraheze>	 RECOVERY - o.kaoyu666.cn - Cloudflare on sslhost is OK: OK - Certificate 'o.kaoyu666.cn' will expire on Thu 28 Nov 2024 01:32:00 PM GMT +0000.
[20:29:17] <icinga-miraheze>	 RECOVERY - pubwiki.lab.co.pl - Cloudflare on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Thu 28 Nov 2024 01:32:46 PM GMT +0000.
[20:29:17] <icinga-miraheze>	 RECOVERY - wiki.closai.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.closai.com' will expire on Fri 29 Nov 2024 07:57:29 PM GMT +0000.
[20:29:19] <icinga-miraheze>	 RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Fri 29 Nov 2024 04:16:23 PM GMT +0000.
[20:29:20] <icinga-miraheze>	 RECOVERY - wiki.dfonline.dev - Cloudflare on sslhost is OK: OK - Certificate 'wiki.dfonline.dev' will expire on Thu 28 Nov 2024 01:38:25 PM GMT +0000.
[20:29:22] <icinga-miraheze>	 RECOVERY - spiceandwolf.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiceandwolf.wiki' will expire on Fri 22 Nov 2024 04:12:57 AM GMT +0000.
[20:29:23] <icinga-miraheze>	 RECOVERY - wiki.triplescripts.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.triplescripts.org' will expire on Thu 28 Nov 2024 01:42:43 PM GMT +0000.
[20:29:24] <icinga-miraheze>	 RECOVERY - wiki.tdrweb.top - Cloudflare on sslhost is OK: OK - Certificate 'wiki.tdrweb.top' will expire on Thu 28 Nov 2024 01:42:06 PM GMT +0000.
[20:29:28] <icinga-miraheze>	 RECOVERY - annapolishistorywiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'annapolishistorywiki.org' will expire on Tue 19 Nov 2024 09:58:55 AM GMT +0000.
[20:29:29] <icinga-miraheze>	 RECOVERY - wiki.blackorchestra.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.blackorchestra.net' will expire on Thu 28 Nov 2024 01:38:00 PM GMT +0000.
[20:29:29] <icinga-miraheze>	 RECOVERY - wiki.mxlinuxusers.de - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Thu 28 Nov 2024 01:39:35 PM GMT +0000.
[20:29:29] <icinga-miraheze>	 RECOVERY - wiki.insideearth.info - Cloudflare on sslhost is OK: OK - Certificate 'wiki.insideearth.info' will expire on Thu 28 Nov 2024 01:38:58 PM GMT +0000.
[20:29:31] <icinga-miraheze>	 RECOVERY - pickaxe.showierdata.xyz - Cloudflare on sslhost is OK: OK - Certificate 'pickaxe.showierdata.xyz' will expire on Thu 28 Nov 2024 01:32:05 PM GMT +0000.
[20:29:31] <icinga-miraheze>	 RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:29:40] <icinga-miraheze>	 RECOVERY - pokemonwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'pokemonwiki.info' will expire on Sun 20 Oct 2024 04:47:56 PM GMT +0000.
[20:29:47] <icinga-miraheze>	 RECOVERY - wiki.digitalcandela.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.digitalcandela.com' will expire on Thu 28 Nov 2024 01:38:25 PM GMT +0000.
[20:29:53] <icinga-miraheze>	 RECOVERY - osdev.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'osdev.wiki' will expire on Sun 24 Nov 2024 10:39:44 PM GMT +0000.
[20:29:53] <icinga-miraheze>	 RECOVERY - wiki.auraxis.co - Cloudflare on sslhost is OK: OK - Certificate 'wiki.auraxis.co' will expire on Thu 28 Nov 2024 01:37:55 PM GMT +0000.
[20:29:54] <icinga-miraheze>	 RECOVERY - rarewarewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewarewiki.com' will expire on Fri 11 Oct 2024 05:28:02 PM GMT +0000.
[20:29:55] <icinga-miraheze>	 RECOVERY - terraforming.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'terraforming.wiki' will expire on Wed 25 Sep 2024 08:31:50 PM GMT +0000.
[20:29:59] <icinga-miraheze>	 RECOVERY - www.langrisser.wiki - Cloudflare on sslhost is OK: OK - Certificate 'www.langrisser.wiki' will expire on Thu 28 Nov 2024 03:55:27 PM GMT +0000.
[20:30:01] <icinga-miraheze>	 RECOVERY - fanpedia.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:30:01] <icinga-miraheze>	 RECOVERY - rippaverse.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[20:30:03] <icinga-miraheze>	 RECOVERY - wiki.corgicam.tv - Cloudflare on sslhost is OK: OK - Certificate 'wiki.corgicam.tv' will expire on Thu 28 Nov 2024 01:38:12 PM GMT +0000.
[20:30:15] <icinga-miraheze>	 RECOVERY - eventgate181 Puppet on eventgate181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:30:20] <icinga-miraheze>	 RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.publictestwiki.com' will expire on Fri 29 Nov 2024 07:59:52 PM GMT +0000.
[20:30:21] <icinga-miraheze>	 RECOVERY - www.puzzles.wiki - Cloudflare on sslhost is OK: OK - Certificate 'www.puzzles.wiki' will expire on Thu 28 Nov 2024 04:27:57 PM GMT +0000.
[20:30:22] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[20:30:22] <icinga-miraheze>	 RECOVERY - crustypedia.org - Cloudflare on sslhost is OK: OK - Certificate 'crustypedia.org' will expire on Thu 28 Nov 2024 01:25:18 PM GMT +0000.
[20:30:23] <icinga-miraheze>	 RECOVERY - wiki.apico.buzz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.apico.buzz' will expire on Thu 28 Nov 2024 01:37:49 PM GMT +0000.
[20:30:24] <icinga-miraheze>	 RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Thu 10 Oct 2024 12:53:31 AM GMT +0000.
[20:30:24] <icinga-miraheze>	 RECOVERY - knifepointhorror.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'knifepointhorror.wiki' will expire on Fri 11 Oct 2024 12:38:56 AM GMT +0000.
[20:30:26] <icinga-miraheze>	 RECOVERY - ff8.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ff8.wiki' will expire on Mon 02 Dec 2024 01:38:11 AM GMT +0000.
[20:30:28] <icinga-miraheze>	 RECOVERY - wiki.esnmilanostatale.it - Cloudflare on sslhost is OK: OK - Certificate 'wiki.esnmilanostatale.it' will expire on Thu 28 Nov 2024 01:38:35 PM GMT +0000.
[20:30:29] <icinga-miraheze>	 RECOVERY - atlas.starworld.zone - Cloudflare on sslhost is OK: OK - Certificate 'atlas.starworld.zone' will expire on Thu 28 Nov 2024 01:24:59 PM GMT +0000.
[20:30:30] <icinga-miraheze>	 RECOVERY - ecole.science - LetsEncrypt on sslhost is OK: OK - Certificate 'ecole.science' will expire on Thu 10 Oct 2024 01:22:18 AM GMT +0000.
[20:30:34] <icinga-miraheze>	 RECOVERY - patternarchive.online - LetsEncrypt on sslhost is OK: OK - Certificate 'patternarchive.online' will expire on Sun 20 Oct 2024 04:49:30 PM GMT +0000.
[20:30:37] <icinga-miraheze>	 RECOVERY - wiki.projectdiablo2.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.projectdiablo2.com' will expire on Thu 28 Nov 2024 01:41:30 PM GMT +0000.
[20:30:38] <icinga-miraheze>	 RECOVERY - guia.esporo.net - Cloudflare on sslhost is OK: OK - Certificate 'guia.esporo.net' will expire on Thu 28 Nov 2024 01:30:52 PM GMT +0000.
[20:30:40] <icinga-miraheze>	 RECOVERY - wiki.irisstation.lol - Cloudflare on sslhost is OK: OK - Certificate 'wiki.irisstation.lol' will expire on Thu 28 Nov 2024 01:38:58 PM GMT +0000.
[20:30:44] <icinga-miraheze>	 RECOVERY - wiki.10key.plus - Cloudflare on sslhost is OK: OK - Certificate 'wiki.10key.plus' will expire on Thu 28 Nov 2024 01:37:40 PM GMT +0000.
[20:30:45] <icinga-miraheze>	 RECOVERY - removededm.com - LetsEncrypt on sslhost is OK: OK - Certificate 'removededm.com' will expire on Mon 25 Nov 2024 01:47:38 PM GMT +0000.
[20:30:45] <icinga-miraheze>	 RECOVERY - www.portalsofphereon.com - Cloudflare on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Fri 29 Nov 2024 05:09:19 PM GMT +0000.
[20:30:47] <icinga-miraheze>	 RECOVERY - crashspyro.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:31:04] <icinga-miraheze>	 RECOVERY - wiki.s23.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Thu 28 Nov 2024 01:41:41 PM GMT +0000.
[20:31:11] <icinga-miraheze>	 RECOVERY - mockgovernments.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mockgovernments.com' will expire on Sun 03 Nov 2024 07:20:56 PM GMT +0000.
[20:31:11] <icinga-miraheze>	 RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'potabi.com' will expire on Thu 17 Oct 2024 04:34:59 PM GMT +0000.
[20:31:11] <icinga-miraheze>	 RECOVERY - mirabeta.org - Cloudflare on sslhost is OK: OK - Certificate 'mirabeta.org' will expire on Fri 29 Nov 2024 06:53:23 AM GMT +0000.
[20:31:12] <icinga-miraheze>	 RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Thu 10 Oct 2024 12:01:50 AM GMT +0000.
[20:31:14] <icinga-miraheze>	 RECOVERY - wiki.shaazzz.ir - Cloudflare on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Fri 29 Nov 2024 07:58:47 PM GMT +0000.
[20:31:16] <icinga-miraheze>	 RECOVERY - kalons-reverie.com - LetsEncrypt on sslhost is OK: OK - Certificate 'kalons-reverie.com' will expire on Mon 28 Oct 2024 10:29:48 AM GMT +0000.
[20:31:16] <icinga-miraheze>	 RECOVERY - wiki.gesamtschule-nordkirchen.de - Cloudflare on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Fri 29 Nov 2024 07:57:49 PM GMT +0000.
[20:31:17] <icinga-miraheze>	 RECOVERY - capixabapedia.vipnunes.com - Cloudflare on sslhost is OK: OK - Certificate 'capixabapedia.vipnunes.com' will expire on Thu 28 Nov 2024 01:25:08 PM GMT +0000.
[20:31:17] <icinga-miraheze>	 RECOVERY - wiki.ff6worldscollide.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ff6worldscollide.com' will expire on Thu 28 Nov 2024 01:38:38 PM GMT +0000.
[20:31:19] <icinga-miraheze>	 RECOVERY - wiki.astralprojections.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.astralprojections.org' will expire on Thu 28 Nov 2024 01:37:52 PM GMT +0000.
[20:31:22] <icinga-miraheze>	 RECOVERY - wiki.meeusen.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Fri 29 Nov 2024 07:58:03 PM GMT +0000.
[20:31:22] <icinga-miraheze>	 RECOVERY - wiki.hitscan.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.hitscan.org' will expire on Thu 28 Nov 2024 01:38:53 PM GMT +0000.
[20:31:23] <icinga-miraheze>	 RECOVERY - wiki.junkstore.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.junkstore.xyz' will expire on Thu 28 Nov 2024 01:39:05 PM GMT +0000.
[20:31:23] <icinga-miraheze>	 RECOVERY - www.istpcomputing.com - Cloudflare on sslhost is OK: OK - Certificate 'www.istpcomputing.com' will expire on Thu 28 Nov 2024 01:43:25 PM GMT +0000.
[20:31:28] <icinga-miraheze>	 RECOVERY - 321nails.crpteam.club - Cloudflare on sslhost is OK: OK - Certificate '321nails.crpteam.club' will expire on Thu 28 Nov 2024 02:04:00 PM GMT +0000.
[20:31:30] <icinga-miraheze>	 RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Wed 09 Oct 2024 10:29:33 AM GMT +0000.
[20:31:31] <icinga-miraheze>	 RECOVERY - wiki.fbpml.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.fbpml.org' will expire on Fri 29 Nov 2024 07:57:35 PM GMT +0000.
[20:31:34] <icinga-miraheze>	 RECOVERY - strinova.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:31:34] <icinga-miraheze>	 RECOVERY - quadball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'quadball.wiki' will expire on Sat 09 Nov 2024 03:38:36 PM GMT +0000.
[20:31:34] <icinga-miraheze>	 RECOVERY - legacygt.wiki - Cloudflare on sslhost is OK: OK - Certificate 'legacygt.wiki' will expire on Thu 28 Nov 2024 02:13:17 PM GMT +0000.
[20:31:34] <icinga-miraheze>	 RECOVERY - cheeseepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cheeseepedia.org' will expire on Fri 04 Oct 2024 09:40:03 AM GMT +0000.
[20:31:36] <icinga-miraheze>	 RECOVERY - rainverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rainverse.wiki' will expire on Fri 22 Nov 2024 01:56:56 AM GMT +0000.
[20:31:37] <icinga-miraheze>	 RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Sun 20 Oct 2024 04:52:09 PM GMT +0000.
[20:31:39] <icinga-miraheze>	 RECOVERY - kingdomdeath.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kingdomdeath.wiki' will expire on Wed 16 Oct 2024 10:25:08 AM GMT +0000.
[20:31:39] <icinga-miraheze>	 RECOVERY - thegreatwar.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thegreatwar.uk' will expire on Wed 09 Oct 2024 07:05:29 PM GMT +0000.
[20:31:40] <icinga-miraheze>	 RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'scvo.org' will expire on Fri 29 Nov 2024 07:51:04 AM GMT +0000.
[20:31:41] <icinga-miraheze>	 RECOVERY - amps.wiki.gd - Cloudflare on sslhost is OK: OK - Certificate 'amps.wiki.gd' will expire on Thu 28 Nov 2024 01:24:55 PM GMT +0000.
[20:31:42] <icinga-miraheze>	 RECOVERY - polcompball.wiki - Cloudflare on sslhost is OK: OK - Certificate 'polcompball.wiki' will expire on Thu 28 Nov 2024 01:32:12 PM GMT +0000.
[20:31:43] <icinga-miraheze>	 RECOVERY - www.bluepageswiki.org - Cloudflare on sslhost is OK: OK - Certificate 'www.bluepageswiki.org' will expire on Thu 28 Nov 2024 01:43:15 PM GMT +0000.
[20:31:44] <icinga-miraheze>	 RECOVERY - history.estill.org - Cloudflare on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Thu 28 Nov 2024 01:30:58 PM GMT +0000.
[20:31:45] <icinga-miraheze>	 RECOVERY - wiki.zergmafia.top - LetsEncrypt on sslhost is OK: OK - Certificate 'zergmafia.top' will expire on Sat 30 Nov 2024 06:49:15 PM GMT +0000.
[20:31:45] <icinga-miraheze>	 RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'www.programming.red' will expire on Thu 28 Nov 2024 03:58:43 PM GMT +0000.
[20:31:46] <icinga-miraheze>	 RECOVERY - nintendo.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:31:47] <icinga-miraheze>	 RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Tue 08 Oct 2024 10:25:55 PM GMT +0000.
[20:31:49] <icinga-miraheze>	 RECOVERY - luxuryelevator.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:31:49] <icinga-miraheze>	 RECOVERY - dragdown.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'dragdown.wiki' will expire on Sat 19 Oct 2024 01:47:25 PM GMT +0000.
[20:31:53] <icinga-miraheze>	 RECOVERY - www.dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.dcmultiversewiki.com' will expire on Fri 29 Nov 2024 05:08:41 PM GMT +0000.
[20:31:55] <icinga-miraheze>	 RECOVERY - wiki.gestaltonline.ca - Cloudflare on sslhost is OK: OK - Certificate 'wiki.gestaltonline.ca' will expire on Fri 29 Nov 2024 07:57:51 PM GMT +0000.
[20:31:56] <icinga-miraheze>	 RECOVERY - wiki.yumeka.icu - Cloudflare on sslhost is OK: OK - Certificate 'wiki.yumeka.icu' will expire on Thu 28 Nov 2024 01:43:00 PM GMT +0000.
[20:31:57] <icinga-miraheze>	 RECOVERY - wiki.lesunk.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.lesunk.com' will expire on Thu 28 Nov 2024 01:39:15 PM GMT +0000.
[20:31:59] <icinga-miraheze>	 RECOVERY - www.ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Fri 25 Oct 2024 04:43:10 PM GMT +0000.
[20:31:59] <icinga-miraheze>	 RECOVERY - piwik.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:32:01] <icinga-miraheze>	 RECOVERY - wiki.gowashtenaw.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.gowashtenaw.org' will expire on Thu 28 Nov 2024 01:38:45 PM GMT +0000.
[20:32:01] <icinga-miraheze>	 RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:32:02] <icinga-miraheze>	 RECOVERY - talumn.neptune.wiki - Cloudflare on sslhost is OK: OK - Certificate 'talumn.neptune.wiki' will expire on Thu 28 Nov 2024 01:34:28 PM GMT +0000.
[20:32:03] <icinga-miraheze>	 RECOVERY - fc.songcontests.eu - Cloudflare on sslhost is OK: OK - Certificate 'fc.songcontests.eu' will expire on Thu 28 Nov 2024 01:28:08 PM GMT +0000.
[20:32:06] <icinga-miraheze>	 RECOVERY - wiki.aoemods.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.aoemods.com' will expire on Fri 29 Nov 2024 07:57:25 PM GMT +0000.
[20:32:08] <icinga-miraheze>	 RECOVERY - wiki.cubestudios.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cubestudios.xyz' will expire on Thu 28 Nov 2024 01:38:18 PM GMT +0000.
[20:32:09] <icinga-miraheze>	 RECOVERY - wiki.buryland.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.buryland.net' will expire on Thu 28 Nov 2024 01:38:01 PM GMT +0000.
[20:32:13] <icinga-miraheze>	 RECOVERY - thestarsareright.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thestarsareright.org' will expire on Thu 10 Oct 2024 12:17:46 PM GMT +0000.
[20:32:16] <icinga-miraheze>	 RECOVERY - croc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:32:17] <icinga-miraheze>	 RECOVERY - gufengcheng.top - Cloudflare on sslhost is OK: OK - Certificate 'gufengcheng.top' will expire on Fri 29 Nov 2024 08:05:47 PM GMT +0000.
[20:32:18] <icinga-miraheze>	 RECOVERY - celeste.ink - Cloudflare on sslhost is OK: OK - Certificate 'celeste.ink' will expire on Thu 28 Nov 2024 01:25:15 PM GMT +0000.
[20:32:18] <icinga-miraheze>	 RECOVERY - wiki.susqu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'susqu.org' will expire on Thu 07 Nov 2024 03:32:03 AM GMT +0000.
[20:32:19] <icinga-miraheze>	 RECOVERY - www.documentopublico.com.br - Cloudflare on sslhost is OK: OK - Certificate 'www.documentopublico.com.br' will expire on Thu 28 Nov 2024 01:43:19 PM GMT +0000.
[20:32:19] <icinga-miraheze>	 RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:32:22] <icinga-miraheze>	 RECOVERY - small.polandballwiki.com - Cloudflare on sslhost is OK: OK - Certificate 'small.polandballwiki.com' will expire on Thu 28 Nov 2024 01:34:19 PM GMT +0000.
[20:32:25] <icinga-miraheze>	 RECOVERY - dlfm-wiki.top - Cloudflare on sslhost is OK: OK - Certificate 'dlfm-wiki.top' will expire on Thu 28 Nov 2024 02:15:47 PM GMT +0000.
[20:32:28] <icinga-miraheze>	 RECOVERY - moddedminecraft.wiki - Cloudflare on sslhost is OK: OK - Certificate 'moddedminecraft.wiki' will expire on Thu 28 Nov 2024 01:31:58 PM GMT +0000.
[20:32:29] <icinga-miraheze>	 RECOVERY - rockrevolution.wiki - Cloudflare on sslhost is OK: OK - Certificate 'rockrevolution.wiki' will expire on Thu 28 Nov 2024 01:32:58 PM GMT +0000.
[20:32:35] <icinga-miraheze>	 RECOVERY - wiki.manos.pt - Cloudflare on sslhost is OK: OK - Certificate 'wiki.manos.pt' will expire on Thu 28 Nov 2024 01:39:20 PM GMT +0000.
[20:32:40] <icinga-miraheze>	 RECOVERY - wiki.k2cho.me - Cloudflare on sslhost is OK: OK - Certificate 'wiki.k2cho.me' will expire on Thu 28 Nov 2024 01:39:08 PM GMT +0000.
[20:32:43] <icinga-miraheze>	 RECOVERY - steem.wiki - Cloudflare on sslhost is OK: OK - Certificate 'steem.wiki' will expire on Thu 28 Nov 2024 01:34:25 PM GMT +0000.
[20:32:44] <icinga-miraheze>	 RECOVERY - wiki.seamly.io - Cloudflare on sslhost is OK: OK - Certificate 'wiki.seamly.io' will expire on Thu 28 Nov 2024 01:41:43 PM GMT +0000.
[20:32:49] <icinga-miraheze>	 RECOVERY - wiki.eastsidegamers.no - Cloudflare on sslhost is OK: OK - Certificate 'wiki.eastsidegamers.no' will expire on Thu 28 Nov 2024 01:38:28 PM GMT +0000.
[20:32:59] <icinga-miraheze>	 RECOVERY - wiki.ate42.ru - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ate42.ru' will expire on Thu 28 Nov 2024 01:37:54 PM GMT +0000.
[20:33:00] <icinga-miraheze>	 RECOVERY - wiki.cowgl.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cowgl.xyz' will expire on Thu 28 Nov 2024 01:38:15 PM GMT +0000.
[20:33:00] <icinga-miraheze>	 RECOVERY - wiki.gtnewhorizons.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.gtnewhorizons.com' will expire on Thu 28 Nov 2024 01:38:49 PM GMT +0000.
[20:33:07] <icinga-miraheze>	 RECOVERY - wiki.case-clicker.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.case-clicker.com' will expire on Thu 28 Nov 2024 01:38:02 PM GMT +0000.
[20:33:12] <icinga-miraheze>	 RECOVERY - wiki.endiversum.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.endiversum.com' will expire on Sat 30 Nov 2024 08:01:39 PM GMT +0000.
[20:33:13] <icinga-miraheze>	 RECOVERY - mason.sagan4.org - Cloudflare on sslhost is OK: OK - Certificate 'mason.sagan4.org' will expire on Thu 28 Nov 2024 01:31:45 PM GMT +0000.
[20:33:15] <icinga-miraheze>	 RECOVERY - wiki.wubbygame.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.wubbygame.com' will expire on Thu 28 Nov 2024 01:42:55 PM GMT +0000.
[20:33:15] <icinga-miraheze>	 RECOVERY - wiki.ooer.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'ooer.ooo' will expire on Mon 25 Nov 2024 05:56:18 AM GMT +0000.
[20:33:20] <icinga-miraheze>	 RECOVERY - wiki.chevrine.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.chevrine.com' will expire on Thu 28 Nov 2024 01:38:05 PM GMT +0000.
[20:33:25] <icinga-miraheze>	 RECOVERY - wiki.stag.lol - Cloudflare on sslhost is OK: OK - Certificate 'wiki.stag.lol' will expire on Thu 28 Nov 2024 01:41:58 PM GMT +0000.
[20:33:27] <icinga-miraheze>	 RECOVERY - wiki.18t.rip - Cloudflare on sslhost is OK: OK - Certificate 'wiki.18t.rip' will expire on Thu 28 Nov 2024 01:37:41 PM GMT +0000.
[20:33:28] <icinga-miraheze>	 RECOVERY - repository.archiopedia.org - Cloudflare on sslhost is OK: OK - Certificate 'repository.archiopedia.org' will expire on Fri 29 Nov 2024 07:56:16 PM GMT +0000.
[20:33:31] <icinga-miraheze>	 RECOVERY - wiki.site97.fr - Cloudflare on sslhost is OK: OK - Certificate 'wiki.site97.fr' will expire on Thu 28 Nov 2024 01:41:50 PM GMT +0000.
[20:33:32] <icinga-miraheze>	 RECOVERY - wiki.puucraft.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.puucraft.net' will expire on Thu 28 Nov 2024 01:41:32 PM GMT +0000.
[20:33:33] <icinga-miraheze>	 RECOVERY - siam.info.gf - Cloudflare on sslhost is OK: OK - Certificate 'siam.info.gf' will expire on Thu 28 Nov 2024 01:34:19 PM GMT +0000.
[20:33:36] <icinga-miraheze>	 RECOVERY - wiki.tgtdrblx.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.tgtdrblx.com' will expire on Thu 28 Nov 2024 01:42:05 PM GMT +0000.
[20:33:36] <icinga-miraheze>	 RECOVERY - en.omniversalis.org - Cloudflare on sslhost is OK: OK - Certificate 'en.omniversalis.org' will expire on Thu 28 Nov 2024 01:26:28 PM GMT +0000.
[20:33:36] <icinga-miraheze>	 RECOVERY - wiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:33:48] <icinga-miraheze>	 RECOVERY - wiki.aridia.space - Cloudflare on sslhost is OK: OK - Certificate 'wiki.aridia.space' will expire on Fri 29 Nov 2024 07:57:26 PM GMT +0000.
[20:33:49] <icinga-miraheze>	 RECOVERY - wiki.yunachannel.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.yunachannel.com' will expire on Thu 28 Nov 2024 01:43:00 PM GMT +0000.
[20:33:50] <icinga-miraheze>	 RECOVERY - wiki.pixlies.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.pixlies.net' will expire on Thu 28 Nov 2024 01:41:28 PM GMT +0000.
[20:33:52] <icinga-miraheze>	 RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:33:53] <icinga-miraheze>	 RECOVERY - tl.awiki.org - Cloudflare on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Thu 28 Nov 2024 01:35:48 PM GMT +0000.
[20:33:58] <icinga-miraheze>	 RECOVERY - sona.pona.la - Cloudflare on sslhost is OK: OK - Certificate 'sona.pona.la' will expire on Thu 28 Nov 2024 01:34:20 PM GMT +0000.
[20:34:03] <icinga-miraheze>	 RECOVERY - you.r-fit.cc - Cloudflare on sslhost is OK: OK - Certificate 'you.r-fit.cc' will expire on Thu 28 Nov 2024 01:43:40 PM GMT +0000.
[20:34:03] <icinga-miraheze>	 RECOVERY - wiki.infomedia.co.uk - Cloudflare on sslhost is OK: OK - Certificate 'wiki.infomedia.co.uk' will expire on Thu 28 Nov 2024 01:38:55 PM GMT +0000.
[20:34:03] <icinga-miraheze>	 RECOVERY - lpfteam.nl - Cloudflare on sslhost is OK: OK - Certificate 'lpfteam.nl' will expire on Thu 28 Nov 2024 01:31:41 PM GMT +0000.
[20:34:06] <icinga-miraheze>	 RECOVERY - wiki.secondrenaissance.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.secondrenaissance.net' will expire on Thu 28 Nov 2024 01:41:45 PM GMT +0000.
[20:34:11] <icinga-miraheze>	 RECOVERY - wiki.theluxuryelevator.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.theluxuryelevator.com' will expire on Thu 28 Nov 2024 01:42:39 PM GMT +0000.
[20:34:13] <icinga-miraheze>	 RECOVERY - www.lab612.at - Cloudflare on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Thu 28 Nov 2024 01:43:29 PM GMT +0000.
[20:34:14] <icinga-miraheze>	 RECOVERY - meijinversus.com - Cloudflare on sslhost is OK: OK - Certificate 'meijinversus.com' will expire on Thu 28 Nov 2024 01:31:48 PM GMT +0000.
[20:34:15] <icinga-miraheze>	 PROBLEM - db171 APT on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host
[20:34:15] <icinga-miraheze>	 PROBLEM - db171 NTP time on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host
[20:34:16] <icinga-miraheze>	 RECOVERY - Host cp36 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms
[20:34:16] <icinga-miraheze>	 PROBLEM - jobchron171 NTP time on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[20:34:20] <icinga-miraheze>	 PROBLEM - ping on swiftproxy171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.108)
[20:34:22] <icinga-miraheze>	 PROBLEM - ping on reports171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.139)
[20:34:24] <icinga-miraheze>	 PROBLEM - ping on swiftac171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.125)
[20:34:25] <icinga-miraheze>	 RECOVERY - history.sdtef.org - Cloudflare on sslhost is OK: OK - Certificate 'history.sdtef.org' will expire on Thu 28 Nov 2024 01:30:58 PM GMT +0000.
[20:34:26] <icinga-miraheze>	 RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:34:27] <icinga-miraheze>	 PROBLEM - phorge171 ferm_active on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host
[20:34:27] <icinga-miraheze>	 PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host
[20:34:27] <icinga-miraheze>	 PROBLEM - phorge171 NTP time on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host
[20:34:27] <icinga-miraheze>	 PROBLEM - phorge171 Disk Space on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host
[20:34:27] <icinga-miraheze>	 RECOVERY - lostmediaru.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:34:30] <icinga-miraheze>	 RECOVERY - cp36 PowerDNS Recursor on cp36 is OK: DNS OK: 0.330 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[20:34:31] <icinga-miraheze>	 PROBLEM - cp36 NTP time on cp36 is WARNING: NTP WARNING: Offset 0.4913442433 secs
[20:34:33] <icinga-miraheze>	 PROBLEM - ping on jobchron171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.120)
[20:34:34] <icinga-miraheze>	 PROBLEM - ping on db171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.119)
[20:34:35] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host
[20:34:35] <icinga-miraheze>	 PROBLEM - reports171 NTP time on reports171 is CRITICAL: connect to address 10.0.17.139 port 5666: No route to hostconnect to host 10.0.17.139 port 5666: No route to host
[20:34:35] <icinga-miraheze>	 PROBLEM - Host reports171 is DOWN: CRITICAL - Host Unreachable (10.0.17.139)
[20:34:36] <icinga-miraheze>	 PROBLEM - Host mw171 is DOWN: CRITICAL - Host Unreachable (10.0.17.122)
[20:34:37] <icinga-miraheze>	 RECOVERY - wiki.nevillepedia.eu - Cloudflare on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Thu 28 Nov 2024 01:39:39 PM GMT +0000.
[20:34:37] <icinga-miraheze>	 PROBLEM - Host swiftproxy171 is DOWN: CRITICAL - Host Unreachable (10.0.17.108)
[20:34:38] <icinga-miraheze>	 PROBLEM - bots171 Current Load on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[20:34:39] <icinga-miraheze>	 RECOVERY - hcw.tomat.dev - Cloudflare on sslhost is OK: OK - Certificate 'hcw.tomat.dev' will expire on Thu 28 Nov 2024 01:30:55 PM GMT +0000.
[20:34:39] <icinga-miraheze>	 PROBLEM - ping on ldap171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.121)
[20:34:40] <icinga-miraheze>	 RECOVERY - wiki.make717.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Fri 29 Nov 2024 07:58:02 PM GMT +0000.
[20:34:41] <icinga-miraheze>	 PROBLEM - bots171 PowerDNS Recursor on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[20:34:42] <icinga-miraheze>	 PROBLEM - jobchron171 Redis Process on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[20:34:42] <icinga-miraheze>	 PROBLEM - Host mw172 is DOWN: CRITICAL - Host Unreachable (10.0.17.123)
[20:34:43] <icinga-miraheze>	 RECOVERY - www.widedata.org - Cloudflare on sslhost is OK: OK - Certificate 'www.widedata.org' will expire on Thu 28 Nov 2024 04:20:57 PM GMT +0000.
[20:34:44] <icinga-miraheze>	 PROBLEM - Host swiftac171 is DOWN: CRITICAL - Host Unreachable (10.0.17.125)
[20:34:44] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mwtask171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host PROBLEM - cp37 Nginx Backend for mw151 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[20:34:44] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for matomo151 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[20:34:56] <icinga-miraheze>	 RECOVERY - wiki.kingmingshrimp.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.kingmingshrimp.com' will expire on Thu 28 Nov 2024 01:39:08 PM GMT +0000.
[20:34:57] <icinga-miraheze>	 PROBLEM - jobchron171 PowerDNS Recursor on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[20:34:57] <icinga-miraheze>	 PROBLEM - jobchron171 Disk Space on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[20:34:59] <icinga-miraheze>	 RECOVERY - wiki.knowledgerevolution.eu - Cloudflare on sslhost is OK: OK - Certificate 'wiki.knowledgerevolution.eu' will expire on Thu 28 Nov 2024 01:39:11 PM GMT +0000.
[20:35:00] <icinga-miraheze>	 PROBLEM - bots171 IRC-Discord Relay Bot relaybot2 on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[20:35:00] <icinga-miraheze>	 RECOVERY - rare.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:35:02] <icinga-miraheze>	 RECOVERY - rippaversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rippaversewiki.com' will expire on Thu 31 Oct 2024 06:35:20 PM GMT +0000.
[20:35:04] <icinga-miraheze>	 RECOVERY - wiki.aclevo.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.aclevo.com' will expire on Thu 28 Nov 2024 01:37:42 PM GMT +0000.
[20:35:05] <icinga-miraheze>	 RECOVERY - lads.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lads.wiki' will expire on Tue 08 Oct 2024 06:14:12 PM GMT +0000.
[20:35:05] <icinga-miraheze>	 RECOVERY - wiki.penguinmod.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.penguinmod.com' will expire on Thu 28 Nov 2024 01:41:25 PM GMT +0000.
[20:35:08] <icinga-miraheze>	 RECOVERY - wiki.zahira.me - Cloudflare on sslhost is OK: OK - Certificate 'wiki.zahira.me' will expire on Fri 29 Nov 2024 07:59:09 PM GMT +0000.
[20:35:11] <icinga-miraheze>	 RECOVERY - evil.wiki - Cloudflare on sslhost is OK: OK - Certificate 'evil.wiki' will expire on Thu 28 Nov 2024 01:28:00 PM GMT +0000.
[20:35:15] <icinga-miraheze>	 RECOVERY - oxboxtra.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'oxboxtra.wiki' will expire on Fri 15 Nov 2024 07:42:02 AM GMT +0000.
[20:35:16] <icinga-miraheze>	 RECOVERY - squareenixwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'squareenixwiki.com' will expire on Fri 11 Oct 2024 01:52:06 AM GMT +0000.
[20:35:16] <icinga-miraheze>	 RECOVERY - wiki.taes.com.br - Cloudflare on sslhost is OK: OK - Certificate 'wiki.taes.com.br' will expire on Thu 28 Nov 2024 01:42:02 PM GMT +0000.
[20:35:17] <icinga-miraheze>	 RECOVERY - archive.stellurgists.wiki - Cloudflare on sslhost is OK: OK - Certificate 'archive.stellurgists.wiki' will expire on Thu 28 Nov 2024 01:24:58 PM GMT +0000.
[20:35:22] <icinga-miraheze>	 PROBLEM - ns1 Current Load on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host
[20:35:22] <icinga-miraheze>	 RECOVERY - chinafake.wiki - Cloudflare on sslhost is OK: OK - Certificate 'chinafake.wiki' will expire on Thu 28 Nov 2024 01:25:15 PM GMT +0000.
[20:35:25] <icinga-miraheze>	 RECOVERY - wiki.col6.de - Cloudflare on sslhost is OK: OK - Certificate 'wiki.col6.de' will expire on Thu 28 Nov 2024 01:38:09 PM GMT +0000.
[20:35:26] <icinga-miraheze>	 RECOVERY - wiki.jagtek.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.jagtek.org' will expire on Thu 28 Nov 2024 01:38:59 PM GMT +0000.
[20:35:26] <icinga-miraheze>	 RECOVERY - tintinpedia.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'tintinpedia.fr' will expire on Wed 16 Oct 2024 09:40:27 AM GMT +0000.
[20:35:27] <icinga-miraheze>	 PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: connect to address 10.0.17.124 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket
[20:35:28] <icinga-miraheze>	 RECOVERY - wiki.tmyt105.leyhp.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.tmyt105.leyhp.com' will expire on Thu 28 Nov 2024 01:42:43 PM GMT +0000.
[20:35:29] <icinga-miraheze>	 RECOVERY - wiki.creaturathegame.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.creaturathegame.com' will expire on Thu 28 Nov 2024 01:38:15 PM GMT +0000.
[20:35:29] <icinga-miraheze>	 RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'thesimswiki.com' will expire on Tue 03 Dec 2024 05:30:24 AM GMT +0000.
[20:35:30] <icinga-miraheze>	 PROBLEM - bots171 CVT Bot on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[20:35:30] <icinga-miraheze>	 PROBLEM - bots171 IRC Log Server Bot on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[20:35:30] <icinga-miraheze>	 RECOVERY - wikien.wildterra2.com - Cloudflare on sslhost is OK: OK - Certificate 'wikien.wildterra2.com' will expire on Thu 28 Nov 2024 01:43:06 PM GMT +0000.
[20:35:30] <icinga-miraheze>	 PROBLEM - phorge171 PowerDNS Recursor on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host
[20:35:30] <icinga-miraheze>	 PROBLEM - phorge171 conntrack_table_size on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host
[20:35:30] <icinga-miraheze>	 RECOVERY - wiki.mcjones.gay - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mcjones.gay' will expire on Thu 28 Nov 2024 01:39:22 PM GMT +0000.
[20:35:33] <icinga-miraheze>	 PROBLEM - jobchron171 SSH on jobchron171 is CRITICAL: connect to address 10.0.17.120 and port 22: No route to host
[20:35:34] <icinga-miraheze>	 PROBLEM - db171 SSH on db171 is CRITICAL: connect to address 10.0.17.119 and port 22: No route to host
[20:35:34] <icinga-miraheze>	 PROBLEM - db171 Puppet on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host
[20:35:35] <icinga-miraheze>	 RECOVERY - wiki.iurevar.world - LetsEncrypt on sslhost is OK: OK - Certificate 'iurevar.world' will expire on Sat 23 Nov 2024 05:34:13 AM GMT +0000.
[20:35:38] <icinga-miraheze>	 RECOVERY - unhingedwiki.paralleldv.org - Cloudflare on sslhost is OK: OK - Certificate 'unhingedwiki.paralleldv.org' will expire on Thu 28 Nov 2024 01:36:49 PM GMT +0000.
[20:35:38] <icinga-miraheze>	 RECOVERY - nexttide.org - Cloudflare on sslhost is OK: OK - Certificate 'nexttide.org' will expire on Fri 29 Nov 2024 04:52:29 PM GMT +0000.
[20:35:38] <icinga-miraheze>	 PROBLEM - bots171 IRC RC Bot on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[20:35:38] <icinga-miraheze>	 PROBLEM - bots171 APT on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[20:35:39] <icinga-miraheze>	 PROBLEM - ns1 SSH on ns1 is CRITICAL: connect to address 10.0.17.136 and port 22: No route to host
[20:35:39] <icinga-miraheze>	 PROBLEM - ns1 NTP time on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host
[20:35:40] <icinga-miraheze>	 RECOVERY - wiki.drycleaninglobby.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.drycleaninglobby.net' will expire on Thu 28 Nov 2024 01:38:28 PM GMT +0000.
[20:35:41] <icinga-miraheze>	 PROBLEM - bots171 SSH on bots171 is CRITICAL: connect to address 10.0.17.143 and port 22: No route to host
[20:35:41] <icinga-miraheze>	 PROBLEM - bots171 IRC-Discord Relay Bot relaybot on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host
[20:35:53] <icinga-miraheze>	 PROBLEM - db171 ferm_active on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host
[20:35:53] <icinga-miraheze>	 RECOVERY - cp36 conntrack_table_size on cp36 is OK: OK: nf_conntrack is 1 % full
[20:35:54] <icinga-miraheze>	 RECOVERY - meijinversus.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:35:54] <icinga-miraheze>	 RECOVERY - wiki.maribelhearn.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.maribelhearn.com' will expire on Thu 28 Nov 2024 01:39:21 PM GMT +0000.
[20:35:55] <icinga-miraheze>	 RECOVERY - cp36 SSH on cp36 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:35:55] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw172 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8118
[20:35:56] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8113
[20:35:57] <icinga-miraheze>	 RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Sun 13 Oct 2024 01:32:41 PM GMT +0000.
[20:35:58] <icinga-miraheze>	 PROBLEM - jobchron171 Current Load on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[20:35:59] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw162 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8116
[20:36:01] <icinga-miraheze>	 PROBLEM - jobchron171 conntrack_table_size on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host
[20:36:01] <icinga-miraheze>	 PROBLEM - Host bots171 is DOWN: CRITICAL - Host Unreachable (10.0.17.143)
[20:36:02] <icinga-miraheze>	 RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:36:02] <icinga-miraheze>	 RECOVERY - wiki.encyclopbd.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.encyclopbd.info' will expire on Thu 31 Oct 2024 03:41:19 PM GMT +0000.
[20:36:02] <icinga-miraheze>	 RECOVERY - wiki.beergeeks.co.il - Cloudflare on sslhost is OK: OK - Certificate 'wiki.beergeeks.co.il' will expire on Fri 29 Nov 2024 07:00:17 PM GMT +0000.
[20:36:06] <icinga-miraheze>	 PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne
[20:36:06] <icinga-miraheze>	  on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66
[20:36:06] <icinga-miraheze>	 PROBLEM - Host db171 is DOWN: CRITICAL - Host Unreachable (10.0.17.119)
[20:36:08] <icinga-miraheze>	 RECOVERY - mariopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mariopedia.org' will expire on Fri 11 Oct 2024 12:06:46 AM GMT +0000.
[20:36:09] <icinga-miraheze>	 RECOVERY - wiki.gamergeeked.us - Cloudflare on sslhost is OK: OK - Certificate 'wiki.gamergeeked.us' will expire on Thu 28 Nov 2024 12:48:21 PM GMT +0000.
[20:36:10] <icinga-miraheze>	 RECOVERY - www.hebammenwiki.de - Cloudflare on sslhost is OK: OK - Certificate 'www.hebammenwiki.de' will expire on Thu 28 Nov 2024 01:43:26 PM GMT +0000.
[20:36:11] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for test151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8181
[20:36:11] <icinga-miraheze>	 RECOVERY - partopedia.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'partopedia.ru' will expire on Wed 25 Sep 2024 07:57:11 PM GMT +0000.
[20:36:12] <icinga-miraheze>	 RECOVERY - wiki.surf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.surf' will expire on Sun 01 Dec 2024 03:10:28 AM GMT +0000.
[20:36:14] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000.
[20:36:14] <icinga-miraheze>	 RECOVERY - johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Sun 20 Oct 2024 05:00:59 PM GMT +0000.
[20:36:15] <icinga-miraheze>	 RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 19197MiB (21% inode=98%);
[20:36:15] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mon181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8201
[20:36:16] <icinga-miraheze>	 RECOVERY - farthestfrontier.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'farthestfrontier.wiki' will expire on Fri 29 Nov 2024 08:54:27 PM GMT +0000.
[20:36:16] <icinga-miraheze>	 RECOVERY - fanojo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fanojo.wiki' will expire on Tue 22 Oct 2024 01:41:50 AM GMT +0000.
[20:36:17] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8117
[20:36:18] <icinga-miraheze>	 PROBLEM - Host jobchron171 is DOWN: CRITICAL - Host Unreachable (10.0.17.120)
[20:36:18] <icinga-miraheze>	 RECOVERY - wiki.solarconquest.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.solarconquest.net' will expire on Thu 28 Nov 2024 01:41:55 PM GMT +0000.
[20:36:19] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for swiftproxy171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8207
[20:36:19] <icinga-miraheze>	 RECOVERY - www.christipedia.nl - Cloudflare on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Fri 29 Nov 2024 05:09:33 PM GMT +0000.
[20:36:20] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for swiftproxy161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8206
[20:36:21] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 100%
[20:36:32] <icinga-miraheze>	 RECOVERY - aryavratpedia.co - LetsEncrypt on sslhost is OK: OK - Certificate 'aryavratpedia.co' will expire on Fri 15 Nov 2024 03:38:29 AM GMT +0000.
[20:36:33] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mwtask171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8161
[20:36:40] <icinga-miraheze>	 RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Tue 08 Oct 2024 09:57:49 PM GMT +0000.
[20:36:41] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mw181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8119
[20:36:41] <icinga-miraheze>	 RECOVERY - zh.tardis.wiki - Cloudflare on sslhost is OK: OK - Certificate 'zh.tardis.wiki' will expire on Thu 28 Nov 2024 01:43:45 PM GMT +0000.
[20:36:44] <icinga-miraheze>	 RECOVERY - wiki.gtnhdemocracy.site - Cloudflare on sslhost is OK: OK - Certificate 'wiki.gtnhdemocracy.site' will expire on Thu 28 Nov 2024 01:38:50 PM GMT +0000.
[20:36:44] <icinga-miraheze>	 RECOVERY - www.iceria.org - Cloudflare on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Thu 28 Nov 2024 03:57:06 PM GMT +0000.
[20:36:48] <icinga-miraheze>	 RECOVERY - robloxapi.wiki - Cloudflare on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Thu 28 Nov 2024 01:58:02 PM GMT +0000.
[20:36:48] <icinga-miraheze>	 RECOVERY - wiki.yonicstudios.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.yonicstudios.com' will expire on Thu 28 Nov 2024 01:42:58 PM GMT +0000.
[20:36:49] <icinga-miraheze>	 RECOVERY - corru.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'corru.wiki' will expire on Sat 09 Nov 2024 03:33:02 AM GMT +0000.
[20:36:51] <icinga-miraheze>	 RECOVERY - status.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:36:53] <icinga-miraheze>	 RECOVERY - wiki.meregos.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.meregos.com' will expire on Thu 28 Nov 2024 01:39:25 PM GMT +0000.
[20:36:54] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Sun 20 Oct 2024 04:53:18 PM GMT +0000.
[20:36:54] <icinga-miraheze>	 RECOVERY - sega.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:36:57] <icinga-miraheze>	 RECOVERY - wiki.dwarvesrp.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.dwarvesrp.com' will expire on Thu 28 Nov 2024 01:38:28 PM GMT +0000.
[20:36:58] <icinga-miraheze>	 RECOVERY - www.lebork.info.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lebork.info.pl' will expire on Thu 28 Nov 2024 04:02:44 PM GMT +0000.
[20:36:59] <icinga-miraheze>	 RECOVERY - ao90.pinho.org - Cloudflare on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Fri 29 Nov 2024 07:42:23 PM GMT +0000.
[20:37:03] <icinga-miraheze>	 RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 27 Sep 2024 11:58:22 PM GMT +0000.
[20:37:07] <icinga-miraheze>	 RECOVERY - segawiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'segawiki.com' will expire on Fri 11 Oct 2024 04:01:01 PM GMT +0000.
[20:37:08] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - Cloudflare on sslhost is OK: OK - Certificate 'lcn.zfc.id.lv' will expire on Fri 29 Nov 2024 07:55:16 PM GMT +0000.
[20:37:09] <icinga-miraheze>	 RECOVERY - rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Wed 09 Oct 2024 11:31:31 PM GMT +0000.
[20:37:09] <icinga-miraheze>	 RECOVERY - alpha.sagan4.org - Cloudflare on sslhost is OK: OK - Certificate 'alpha.sagan4.org' will expire on Thu 28 Nov 2024 01:24:50 PM GMT +0000.
[20:37:15] <icinga-miraheze>	 RECOVERY - wiki.pulsus.cc - Cloudflare on sslhost is OK: OK - Certificate 'wiki.pulsus.cc' will expire on Thu 28 Nov 2024 01:41:31 PM GMT +0000.
[20:37:16] <icinga-miraheze>	 RECOVERY - wiki.jill-jimmy.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.jill-jimmy.com' will expire on Thu 28 Nov 2024 01:39:01 PM GMT +0000.
[20:37:21] <icinga-miraheze>	 RECOVERY - archives.nsgov.org - Cloudflare on sslhost is OK: OK - Certificate 'archives.nsgov.org' will expire on Thu 28 Nov 2024 01:24:58 PM GMT +0000.
[20:37:25] <icinga-miraheze>	 RECOVERY - grk.repository.archiopedia.org - Cloudflare on sslhost is OK: OK - Certificate 'grk.repository.archiopedia.org' will expire on Fri 29 Nov 2024 07:55:09 PM GMT +0000.
[20:37:26] <icinga-miraheze>	 RECOVERY - beta.sagan4.org - Cloudflare on sslhost is OK: OK - Certificate 'beta.sagan4.org' will expire on Thu 28 Nov 2024 01:25:01 PM GMT +0000.
[20:37:31] <icinga-miraheze>	 RECOVERY - www.thesimswiki.com - Cloudflare on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Thu 28 Nov 2024 01:43:38 PM GMT +0000.
[20:37:31] <icinga-miraheze>	 RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Wed 09 Oct 2024 09:14:31 AM GMT +0000.
[20:37:32] <icinga-miraheze>	 PROBLEM - cloud17 IPMI Sensors on cloud17 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[20:37:32] <icinga-miraheze>	 RECOVERY - wiki.geoparkcorumbatai.com.br - Cloudflare on sslhost is OK: OK - Certificate 'wiki.geoparkcorumbatai.com.br' will expire on Fri 29 Nov 2024 07:57:50 PM GMT +0000.
[20:37:33] <icinga-miraheze>	 RECOVERY - cities.simulz.kr - Cloudflare on sslhost is OK: OK - Certificate 'cities.simulz.kr' will expire on Thu 28 Nov 2024 01:25:13 PM GMT +0000.
[20:37:35] <icinga-miraheze>	 RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 29 Nov 2024 04:19:24 PM GMT +0000.
[20:37:37] <icinga-miraheze>	 RECOVERY - wiki.misslessfighters.club - Cloudflare on sslhost is OK: OK - Certificate 'wiki.misslessfighters.club' will expire on Thu 28 Nov 2024 01:39:31 PM GMT +0000.
[20:37:37] <icinga-miraheze>	 RECOVERY - project-patterns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'project-patterns.com' will expire on Sat 23 Nov 2024 02:04:38 AM GMT +0000.
[20:37:39] <icinga-miraheze>	 RECOVERY - vesc.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vesc.wiki' will expire on Sun 13 Oct 2024 03:12:52 PM GMT +0000.
[20:37:39] <icinga-miraheze>	 RECOVERY - www.pomologia.org - Cloudflare on sslhost is OK: OK - Certificate 'www.pomologia.org' will expire on Fri 29 Nov 2024 05:09:02 PM GMT +0000.
[20:37:40] <icinga-miraheze>	 RECOVERY - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bikinibottomdiscord.org' will expire on Thu 21 Nov 2024 01:38:48 AM GMT +0000.
[20:37:42] <icinga-miraheze>	 RECOVERY - wiki.ouro.one - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ouro.one' will expire on Thu 28 Nov 2024 01:40:41 PM GMT +0000.
[20:37:44] <icinga-miraheze>	 RECOVERY - wiki.wikimedia.cat - Cloudflare on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Thu 28 Nov 2024 01:42:53 PM GMT +0000.
[20:37:48] <icinga-miraheze>	 RECOVERY - wiki.artmechanicum.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.artmechanicum.com' will expire on Thu 28 Nov 2024 01:37:51 PM GMT +0000.
[20:37:49] <icinga-miraheze>	 RECOVERY - wiki.strangereons.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.strangereons.com' will expire on Thu 28 Nov 2024 01:42:00 PM GMT +0000.
[20:37:49] <icinga-miraheze>	 RECOVERY - Host ns1 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms
[20:37:51] <icinga-miraheze>	 RECOVERY - dragonquestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dragonquestwiki.com' will expire on Thu 10 Oct 2024 02:39:49 PM GMT +0000.
[20:37:52] <icinga-miraheze>	 RECOVERY - removededmsongs.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:37:56] <icinga-miraheze>	 RECOVERY - neurologiahpm.com - LetsEncrypt on sslhost is OK: OK - Certificate 'neurologiahpm.com' will expire on Sun 29 Sep 2024 12:37:53 PM GMT +0000.
[20:37:57] <icinga-miraheze>	 RECOVERY - wiki.nowchess.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Fri 29 Nov 2024 07:58:06 PM GMT +0000.
[20:37:59] <icinga-miraheze>	 RECOVERY - aryavartpedia.online - LetsEncrypt on sslhost is OK: OK - Certificate 'aryavartpedia.online' will expire on Thu 07 Nov 2024 12:38:18 AM GMT +0000.
[20:37:59] <icinga-miraheze>	 RECOVERY - vitriol.spoon.army - Cloudflare on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Thu 28 Nov 2024 01:37:38 PM GMT +0000.
[20:38:02] <icinga-miraheze>	 RECOVERY - wiki.myehs.eu - Cloudflare on sslhost is OK: OK - Certificate 'wiki.myehs.eu' will expire on Thu 28 Nov 2024 01:39:35 PM GMT +0000.
[20:38:03] <icinga-miraheze>	 RECOVERY - mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Fri 04 Oct 2024 01:22:23 AM GMT +0000.
[20:38:04] <icinga-miraheze>	 PROBLEM - cloud17 NTP time on cloud17 is WARNING: NTP WARNING: Offset 0.1002676189 secs
[20:38:07] <icinga-miraheze>	 RECOVERY - hololive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hololive.wiki' will expire on Wed 09 Oct 2024 07:44:53 PM GMT +0000.
[20:38:07] <icinga-miraheze>	 RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Wed 09 Oct 2024 09:43:14 AM GMT +0000.
[20:38:07] <icinga-miraheze>	 RECOVERY - yokaiwatchwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yokaiwatchwiki.com' will expire on Fri 11 Oct 2024 06:09:00 PM GMT +0000.
[20:38:08] <icinga-miraheze>	 RECOVERY - yepedia.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'yepedia.xyz' will expire on Mon 25 Nov 2024 03:32:36 PM GMT +0000.
[20:38:11] <icinga-miraheze>	 RECOVERY - coffeewiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'coffeewiki.net' will expire on Sun 24 Nov 2024 10:57:49 PM GMT +0000.
[20:38:12] <icinga-miraheze>	 RECOVERY - smashbros.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:38:13] <icinga-miraheze>	 RECOVERY - wikimoma.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimoma.art' will expire on Thu 10 Oct 2024 10:39:49 AM GMT +0000.
[20:38:14] <icinga-miraheze>	 RECOVERY - wiki.alathramc.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.alathramc.com' will expire on Thu 28 Nov 2024 01:37:45 PM GMT +0000.
[20:38:14] <icinga-miraheze>	 RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Wed 09 Oct 2024 08:01:13 AM GMT +0000.
[20:38:16] <icinga-miraheze>	 RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Wed 09 Oct 2024 04:35:15 PM GMT +0000.
[20:38:17] <icinga-miraheze>	 RECOVERY - cpmusicwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'cpmusicwiki.com' will expire on Sat 16 Nov 2024 12:23:53 PM GMT +0000.
[20:38:17] <icinga-miraheze>	 RECOVERY - wiki.ciptamedia.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Thu 28 Nov 2024 01:38:08 PM GMT +0000.
[20:38:17] <icinga-miraheze>	 RECOVERY - donkeykong.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:38:19] <icinga-miraheze>	 RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Wed 09 Oct 2024 12:04:15 PM GMT +0000.
[20:38:22] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.16 ms
[20:38:25] <icinga-miraheze>	 RECOVERY - fanwiki.takagi3.cn - Cloudflare on sslhost is OK: OK - Certificate 'fanwiki.takagi3.cn' will expire on Thu 28 Nov 2024 01:28:06 PM GMT +0000.
[20:38:25] <icinga-miraheze>	 RECOVERY - cloud17 SSH on cloud17 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:38:29] <icinga-miraheze>	 RECOVERY - xomnipedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'xomnipedia.org' will expire on Thu 17 Oct 2024 09:33:16 AM GMT +0000.
[20:38:29] <icinga-miraheze>	 RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mikrodev.com' will expire on Fri 08 Nov 2024 05:10:40 AM GMT +0000.
[20:38:38] <icinga-miraheze>	 RECOVERY - tno.wiki - Cloudflare on sslhost is OK: OK - Certificate 'tno.wiki' will expire on Thu 28 Nov 2024 02:03:20 PM GMT +0000.
[20:38:39] <icinga-miraheze>	 RECOVERY - clubpenguinfanon.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'clubpenguinfanon.wiki' will expire on Mon 28 Oct 2024 12:43:18 AM GMT +0000.
[20:38:40] <icinga-miraheze>	 RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberfurs.org' will expire on Thu 14 Nov 2024 07:49:33 PM GMT +0000.
[20:38:44] <icinga-miraheze>	 RECOVERY - burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Sun 20 Oct 2024 05:00:20 PM GMT +0000.
[20:38:45] <icinga-miraheze>	 RECOVERY - wiki.cosmicdeathfungus.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cosmicdeathfungus.com' will expire on Sun 24 Nov 2024 05:06:01 PM GMT +0000.
[20:38:45] <icinga-miraheze>	 RECOVERY - Host mwtask171 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms
[20:38:46] <icinga-miraheze>	 RECOVERY - wiki.virtualjet.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.virtualjet.net' will expire on Thu 28 Nov 2024 01:42:48 PM GMT +0000.
[20:38:50] <icinga-miraheze>	 RECOVERY - mario.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:38:51] <icinga-miraheze>	 RECOVERY - ns1 APT on ns1 is OK: APT OK: 64 packages available for upgrade (0 critical updates).
[20:38:51] <icinga-miraheze>	 RECOVERY - Host cp37 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms
[20:38:51] <icinga-miraheze>	 RECOVERY - ns1 conntrack_table_size on ns1 is OK: OK: nf_conntrack is 0 % full
[20:38:52] <icinga-miraheze>	 RECOVERY - pyramidgames.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pyramidgames.wiki' will expire on Sun 20 Oct 2024 04:56:18 PM GMT +0000.
[20:38:53] <icinga-miraheze>	 RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Fri 08 Nov 2024 02:38:28 AM GMT +0000.
[20:38:55] <icinga-miraheze>	 RECOVERY - wiki.milkywayidle.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.milkywayidle.com' will expire on Thu 28 Nov 2024 01:39:28 PM GMT +0000.
[20:38:55] <icinga-miraheze>	 RECOVERY - wiki.esufranchise.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.esufranchise.com' will expire on Thu 28 Nov 2024 01:38:35 PM GMT +0000.
[20:38:58] <icinga-miraheze>	 RECOVERY - equestripedia.org - Cloudflare on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Thu 28 Nov 2024 01:27:15 PM GMT +0000.
[20:39:00] <icinga-miraheze>	 RECOVERY - wiki.danvs.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.danvs.net' will expire on Thu 28 Nov 2024 12:47:17 PM GMT +0000.
[20:39:04] <icinga-miraheze>	 RECOVERY - cg.songngu.xyz - Cloudflare on sslhost is OK: OK - Certificate 'cg.songngu.xyz' will expire on Thu 28 Nov 2024 11:52:53 AM GMT +0000.
[20:39:10] <icinga-miraheze>	 RECOVERY - lab.wikimimo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimimo.com' will expire on Tue 08 Oct 2024 10:24:23 AM GMT +0000.
[20:39:11] <icinga-miraheze>	 RECOVERY - witchingworld.org - LetsEncrypt on sslhost is OK: OK - Certificate 'witchingworld.org' will expire on Fri 11 Oct 2024 07:17:44 PM GMT +0000.
[20:39:12] <icinga-miraheze>	 RECOVERY - wiki.thunis.eu - Cloudflare on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Thu 28 Nov 2024 01:42:41 PM GMT +0000.
[20:39:12] <icinga-miraheze>	 RECOVERY - hardcore.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hardcore.wiki' will expire on Fri 27 Sep 2024 05:17:46 PM GMT +0000.
[20:39:12] <icinga-miraheze>	 RECOVERY - wiki.harlonserver.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.harlonserver.net' will expire on Thu 28 Nov 2024 01:38:52 PM GMT +0000.
[20:39:14] <icinga-miraheze>	 RECOVERY - ns1 Current Load on ns1 is OK: LOAD OK - total load average: 0.30, 0.16, 0.06
[20:39:14] <icinga-miraheze>	 RECOVERY - grayravens.com - LetsEncrypt on sslhost is OK: OK - Certificate 'grayravens.com' will expire on Thu 10 Oct 2024 05:05:08 PM GMT +0000.
[20:39:15] <icinga-miraheze>	 RECOVERY - sonicpedia.org - Cloudflare on sslhost is OK: OK - Certificate 'sonicpedia.org' will expire on Thu 28 Nov 2024 01:34:21 PM GMT +0000.
[20:39:16] <icinga-miraheze>	 RECOVERY - solaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'solaswiki.org' will expire on Fri 11 Oct 2024 12:58:53 PM GMT +0000.
[20:39:16] <icinga-miraheze>	 RECOVERY - dc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:39:20] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw152 on cp37 is CRITICAL: connect to address localhost and port 8114: Connection refused
[20:39:20] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw181 on cp37 is CRITICAL: connect to address localhost and port 8119: Connection refused
[20:39:20] <icinga-miraheze>	 PROBLEM - cp37 NTP time on cp37 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[20:39:21] <icinga-miraheze>	 RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Fri 11 Oct 2024 01:04:25 PM GMT +0000.
[20:39:26] <icinga-miraheze>	 RECOVERY - wiki.showtapeselection.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.showtapeselection.com' will expire on Thu 28 Nov 2024 01:41:48 PM GMT +0000.
[20:39:30] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mon181 on cp37 is CRITICAL: connect to address localhost and port 8201: Connection refused
[20:39:30] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw162 on cp37 is CRITICAL: connect to address localhost and port 8116: Connection refused
[20:39:30] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for reports171 on cp37 is CRITICAL: connect to address localhost and port 8205: Connection refused
[20:39:30] <icinga-miraheze>	 RECOVERY - 100acg.wiki - Cloudflare on sslhost is OK: OK - Certificate '100acg.wiki' will expire on Thu 28 Nov 2024 02:29:50 PM GMT +0000.
[20:39:31] <icinga-miraheze>	 RECOVERY - wiki.luemir.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.luemir.xyz' will expire on Thu 28 Nov 2024 01:39:18 PM GMT +0000.
[20:39:31] <icinga-miraheze>	 RECOVERY - agesofconflict.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'agesofconflict.wiki' will expire on Sat 12 Oct 2024 05:02:22 PM GMT +0000.
[20:39:31] <icinga-miraheze>	 RECOVERY - trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'trollpasta.com' will expire on Sun 20 Oct 2024 04:49:54 PM GMT +0000.
[20:39:32] <icinga-miraheze>	 PROBLEM - ns1 NTP time on ns1 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[20:39:32] <icinga-miraheze>	 RECOVERY - cloud17 IPMI Sensors on cloud17 is OK: IPMI Status: OK
[20:39:33] <icinga-miraheze>	 RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:39:34] <icinga-miraheze>	 RECOVERY - islegoblin.wiki - Cloudflare on sslhost is OK: OK - Certificate 'islegoblin.wiki' will expire on Thu 28 Nov 2024 01:31:28 PM GMT +0000.
[20:39:34] <icinga-miraheze>	 RECOVERY - kingdomway.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kingdomway.wiki' will expire on Fri 15 Nov 2024 07:42:18 AM GMT +0000.
[20:39:34] <icinga-miraheze>	 RECOVERY - freedomplanetwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'freedomplanetwiki.com' will expire on Fri 11 Oct 2024 06:21:55 PM GMT +0000.
[20:39:38] <icinga-miraheze>	 RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.033 seconds response time. wikitide.net returns 2a02:c206:2161:9253::1,2a02:c206:2162:6187::1,37.60.225.206,84.247.188.12
[20:39:40] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for test151 on cp37 is CRITICAL: connect to address localhost and port 8181: Connection refused
[20:39:40] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for phorge171 on cp37 is CRITICAL: connect to address localhost and port 8202: Connection refused
[20:39:40] <icinga-miraheze>	 RECOVERY - avas.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'avas.wiki' will expire on Thu 10 Oct 2024 11:47:08 AM GMT +0000.
[20:39:40] <icinga-miraheze>	 RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Fri 11 Oct 2024 12:58:02 AM GMT +0000.
[20:39:42] <icinga-miraheze>	 RECOVERY - ndgkb.nena.org - Cloudflare on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Thu 28 Nov 2024 01:31:55 PM GMT +0000.
[20:39:43] <icinga-miraheze>	 RECOVERY - kodiak.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kodiak.wiki' will expire on Mon 11 Nov 2024 04:12:08 AM GMT +0000.
[20:39:43] <icinga-miraheze>	 RECOVERY - psycho.engineering - LetsEncrypt on sslhost is OK: OK - Certificate 'psycho.engineering' will expire on Sun 13 Oct 2024 01:16:15 PM GMT +0000.
[20:39:44] <icinga-miraheze>	 RECOVERY - wiki.cube-conflict.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cube-conflict.com' will expire on Thu 28 Nov 2024 01:38:19 PM GMT +0000.
[20:39:45] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mwtask181 on cp37 is CRITICAL: connect to address localhost and port 8160: Connection refused
[20:39:45] <icinga-miraheze>	 RECOVERY - wiki.starshipsailing.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.starshipsailing.net' will expire on Thu 28 Nov 2024 01:41:59 PM GMT +0000.
[20:39:47] <icinga-miraheze>	 RECOVERY - inourownwords.online - LetsEncrypt on sslhost is OK: OK - Certificate 'inourownwords.online' will expire on Thu 10 Oct 2024 10:32:38 AM GMT +0000.
[20:39:50] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw182 on cp37 is CRITICAL: connect to address localhost and port 8120: Connection refused
[20:39:53] <icinga-miraheze>	 RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Wed 09 Oct 2024 05:40:21 PM GMT +0000.
[20:39:54] <icinga-miraheze>	 RECOVERY - wiki.omegabuild.uk - Cloudflare on sslhost is OK: OK - Certificate 'wiki.omegabuild.uk' will expire on Thu 28 Nov 2024 01:39:41 PM GMT +0000.
[20:39:55] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for swiftproxy171 on cp37 is CRITICAL: connect to address localhost and port 8207: Connection refused
[20:39:55] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for swiftproxy161 on cp37 is CRITICAL: connect to address localhost and port 8206: Connection refused
[20:39:55] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw171 on cp37 is CRITICAL: connect to address localhost and port 8117: Connection refused
[20:39:55] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw172 on cp37 is CRITICAL: connect to address localhost and port 8118: Connection refused
[20:39:56] <icinga-miraheze>	 RECOVERY - Host bots171 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms
[20:40:00] <icinga-miraheze>	 RECOVERY - uncyclopediakr.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:40:00] <icinga-miraheze>	 PROBLEM - cp37 PowerDNS Recursor on cp37 is CRITICAL: Domain 'wikitide.net' was not found by the server
[20:40:01] <icinga-miraheze>	 RECOVERY - Host db171 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms
[20:40:03] <icinga-miraheze>	 RECOVERY - cloud17 NTP time on cloud17 is OK: NTP OK: Offset 0.04786100984 secs
[20:40:03] <icinga-miraheze>	 RECOVERY - hastursnotebook.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hastursnotebook.org' will expire on Sat 12 Oct 2024 09:07:42 AM GMT +0000.
[20:40:04] <icinga-miraheze>	 RECOVERY - wiki.mineplay.nl - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mineplay.nl' will expire on Thu 28 Nov 2024 01:39:28 PM GMT +0000.
[20:40:12] <icinga-miraheze>	 RECOVERY - acgn.sfdev.eu.org - Cloudflare on sslhost is OK: OK - Certificate 'acgn.sfdev.eu.org' will expire on Thu 28 Nov 2024 02:06:59 PM GMT +0000.
[20:40:14] <icinga-miraheze>	 RECOVERY - wiki.enkro.co - Cloudflare on sslhost is OK: OK - Certificate 'wiki.enkro.co' will expire on Thu 28 Nov 2024 01:38:32 PM GMT +0000.
[20:40:14] <icinga-miraheze>	 RECOVERY - polcompball.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[20:40:18] <icinga-miraheze>	 RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Thu 10 Oct 2024 05:09:32 PM GMT +0000.
[20:40:21] <icinga-miraheze>	 RECOVERY - wiki.amplicity.fr - Cloudflare on sslhost is OK: OK - Certificate 'wiki.amplicity.fr' will expire on Thu 28 Nov 2024 01:37:48 PM GMT +0000.
[20:40:21] <icinga-miraheze>	 RECOVERY - tech.exputra.com - Cloudflare on sslhost is OK: OK - Certificate 'tech.exputra.com' will expire on Thu 28 Nov 2024 01:34:29 PM GMT +0000.
[20:40:24] <icinga-miraheze>	 RECOVERY - Host jobchron171 is UP: PING OK - Packet loss = 0%, RTA = 86.89 ms
[20:40:26] <icinga-miraheze>	 RECOVERY - ping on db171 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms
[20:40:28] <icinga-miraheze>	 RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Thu 10 Oct 2024 04:29:46 PM GMT +0000.
[20:40:28] <icinga-miraheze>	 RECOVERY - Host swiftproxy171 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms
[20:40:31] <icinga-miraheze>	 RECOVERY - pwiki.arkcls.com - Cloudflare on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Thu 28 Nov 2024 01:32:48 PM GMT +0000.
[20:40:31] <icinga-miraheze>	 RECOVERY - lostidols.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lostidols.wiki' will expire on Sun 20 Oct 2024 04:54:19 PM GMT +0000.
[20:40:35] <icinga-miraheze>	 RECOVERY - bobobay.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bobobay.wiki' will expire on Fri 29 Nov 2024 02:57:01 PM GMT +0000.
[20:40:35] <icinga-miraheze>	 RECOVERY - Host bast161 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms
[20:40:35] <icinga-miraheze>	 RECOVERY - bots171 IRC Log Bot on bots171 is OK: PROCS OK: 1 process with args 'adminlogbot.py'
[20:40:36] <icinga-miraheze>	 RECOVERY - bots171 Current Load on bots171 is OK: LOAD OK - total load average: 1.76, 0.59, 0.21
[20:40:37] <icinga-miraheze>	 RECOVERY - wiki.evscope.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.evscope.org' will expire on Thu 28 Nov 2024 01:38:38 PM GMT +0000.
[20:40:38] <icinga-miraheze>	 RECOVERY - bots171 NTP time on bots171 is OK: NTP OK: Offset 8.329749107e-05 secs
[20:40:38] <icinga-miraheze>	 RECOVERY - ping on jobchron171 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms
[20:40:39] <icinga-miraheze>	 RECOVERY - wiki.globasa.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.globasa.net' will expire on Thu 28 Nov 2024 01:38:45 PM GMT +0000.
[20:40:39] <icinga-miraheze>	 RECOVERY - wiki.candelabrem.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.candelabrem.com' will expire on Thu 28 Nov 2024 01:38:01 PM GMT +0000.
[20:40:40] <icinga-miraheze>	 RECOVERY - bots171 PowerDNS Recursor on bots171 is OK: DNS OK: 0.322 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:40:42] <icinga-miraheze>	 RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Fri 04 Oct 2024 04:16:45 PM GMT +0000.
[20:40:43] <icinga-miraheze>	 RECOVERY - ncuwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ncuwiki.com' will expire on Fri 11 Oct 2024 06:48:29 PM GMT +0000.
[20:40:43] <icinga-miraheze>	 RECOVERY - jobchron171 Redis Process on jobchron171 is OK: PROCS OK: 1 process with args 'redis-server'
[20:40:48] <icinga-miraheze>	 RECOVERY - jobchron171 PowerDNS Recursor on jobchron171 is OK: DNS OK: 0.324 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:40:49] <icinga-miraheze>	 RECOVERY - wiki.minkyu.kim - Cloudflare on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Thu 28 Nov 2024 01:39:30 PM GMT +0000.
[20:40:49] <icinga-miraheze>	 RECOVERY - db171 Backups SQL mhglobal on db171 is OK: FILE_AGE OK: /var/log/sql-mhglobal-backup-weekly.log is 315538 seconds old and 121 bytes
[20:40:50] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for puppet181 on cp37 is CRITICAL: connect to address localhost and port 8204: Connection refused
[20:40:50] <icinga-miraheze>	 PROBLEM - cp37 Nginx Backend for mw161 on cp37 is CRITICAL: connect to address localhost and port 8115: Connection refused
[20:40:52] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.41, 0.13, 0.05
[20:40:52] <icinga-miraheze>	 RECOVERY - Host swiftac171 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[20:40:54] <icinga-miraheze>	 RECOVERY - bots171 IRC-Discord Relay Bot relaybot2 on bots171 is OK: PROCS OK: 2 processes with args 'relaybot2/'
[20:40:56] <icinga-miraheze>	 RECOVERY - wiki.thelastdimension.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.thelastdimension.net' will expire on Thu 28 Nov 2024 01:42:38 PM GMT +0000.
[20:40:56] <icinga-miraheze>	 RECOVERY - ping on bots171 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms
[20:41:00] <icinga-miraheze>	 RECOVERY - jobchron171 Disk Space on jobchron171 is OK: DISK OK - free space: / 5663MiB (64% inode=86%);
[20:41:03] <icinga-miraheze>	 RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Fri 11 Oct 2024 01:07:26 AM GMT +0000.
[20:41:17] <icinga-miraheze>	 RECOVERY - worldless.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'worldless.wiki' will expire on Tue 24 Sep 2024 08:16:23 AM GMT +0000.
[20:41:17] <icinga-miraheze>	 RECOVERY - www.lgbtqia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lgbtqia.wiki' will expire on Fri 29 Nov 2024 11:15:50 AM GMT +0000.
[20:41:17] <icinga-miraheze>	 RECOVERY - progressbar95.wiki - Cloudflare on sslhost is OK: OK - Certificate 'progressbar95.wiki' will expire on Thu 28 Nov 2024 01:32:38 PM GMT +0000.
[20:41:18] <icinga-miraheze>	 RECOVERY - minescape.wiki - Cloudflare on sslhost is OK: OK - Certificate 'minescape.wiki' will expire on Thu 28 Nov 2024 01:31:55 PM GMT +0000.
[20:41:19] <icinga-miraheze>	 RECOVERY - bots171 IRC Log Server Bot on bots171 is OK: PROCS OK: 1 process with args 'irclogserverbot.py'
[20:41:19] <icinga-miraheze>	 RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[20:41:20] <icinga-miraheze>	 RECOVERY - wiki.simplifiedcoding.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.simplifiedcoding.org' will expire on Thu 28 Nov 2024 01:41:50 PM GMT +0000.
[20:41:20] <icinga-miraheze>	 RECOVERY - wiki.fatcatfablab.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.fatcatfablab.org' will expire on Thu 28 Nov 2024 01:38:38 PM GMT +0000.
[20:41:20] <icinga-miraheze>	 RECOVERY - bots171 CVT Bot on bots171 is OK: PROCS OK: 2 processes with args 'cvtbot'
[20:41:22] <icinga-miraheze>	 RECOVERY - jobchron171 SSH on jobchron171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:41:22] <icinga-miraheze>	 RECOVERY - correypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'correypedia.org' will expire on Mon 07 Oct 2024 08:32:43 PM GMT +0000.
[20:41:24] <icinga-miraheze>	 RECOVERY - www.pyramidgames.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pyramidgames.wiki' will expire on Thu 28 Nov 2024 04:29:09 PM GMT +0000.
[20:41:27] <icinga-miraheze>	 RECOVERY - polandballwiki.com - Cloudflare on sslhost is OK: OK - Certificate 'polandballwiki.com' will expire on Sat 23 Nov 2024 03:25:15 AM GMT +0000.
[20:41:29] <icinga-miraheze>	 RECOVERY - osrwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'osrwiki.org' will expire on Sat 09 Nov 2024 09:06:39 PM GMT +0000.
[20:41:30] <icinga-miraheze>	 RECOVERY - grk.archiopedia.org - Cloudflare on sslhost is OK: OK - Certificate 'grk.archiopedia.org' will expire on Thu 28 Nov 2024 01:29:19 PM GMT +0000.
[20:41:30] <icinga-miraheze>	 RECOVERY - wiki.albrusmusic.ru - Cloudflare on sslhost is OK: OK - Certificate 'wiki.albrusmusic.ru' will expire on Thu 28 Nov 2024 12:47:19 PM GMT +0000.
[20:41:30] <icinga-miraheze>	 RECOVERY - bots171 SSH on bots171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:41:31] <icinga-miraheze>	 RECOVERY - db171 SSH on db171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:41:31] <icinga-miraheze>	 RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[20:41:31] <icinga-miraheze>	 RECOVERY - bots171 IRC-Discord Relay Bot relaybot on bots171 is OK: PROCS OK: 2 processes with args 'relaybot/'
[20:41:32] <icinga-miraheze>	 RECOVERY - bots171 IRC RC Bot on bots171 is OK: PROCS OK: 1 process with args 'ircrcbot.py'
[20:41:33] <icinga-miraheze>	 RECOVERY - bots171 APT on bots171 is OK: APT OK: 81 packages available for upgrade (0 critical updates).
[20:41:34] <icinga-miraheze>	 RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Mon 14 Oct 2024 10:25:53 AM GMT +0000.
[20:41:35] <icinga-miraheze>	 PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:41:35] <icinga-miraheze>	 RECOVERY - Host ldap171 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms
[20:41:35] <icinga-miraheze>	 RECOVERY - www.glitchcity.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.glitchcity.wiki' will expire on Fri 29 Nov 2024 07:59:35 PM GMT +0000.
[20:41:36] <icinga-miraheze>	 RECOVERY - wiki.eggsdstudios.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.eggsdstudios.com' will expire on Thu 28 Nov 2024 01:38:29 PM GMT +0000.
[20:41:37] <icinga-miraheze>	 RECOVERY - holocron.net - LetsEncrypt on sslhost is OK: OK - Certificate 'holocron.net' will expire on Sat 28 Sep 2024 04:24:05 AM GMT +0000.
[20:41:38] <icinga-miraheze>	 RECOVERY - jobchron171 ferm_active on jobchron171 is OK: OK ferm input default policy is set
[20:41:40] <icinga-miraheze>	 RECOVERY - jobchron171 APT on jobchron171 is OK: APT OK: 75 packages available for upgrade (0 critical updates).
[20:41:41] <icinga-miraheze>	 RECOVERY - jobchron171 JobChron Service on jobchron171 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[20:41:41] <icinga-miraheze>	 RECOVERY - Host phorge171 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms
[20:41:41] <icinga-miraheze>	 RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19630 bytes in 0.053 second response time
[20:41:41] <icinga-miraheze>	 RECOVERY - phorge171 Backups Phorge Static on phorge171 is OK: FILE_AGE OK: /var/log/phorge-backup.log is 322969 seconds old and 91 bytes
[20:41:43] <icinga-miraheze>	 RECOVERY - db171 conntrack_table_size on db171 is OK: OK: nf_conntrack is 0 % full
[20:41:46] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.24, 0.08, 0.03
[20:41:46] <icinga-miraheze>	 RECOVERY - www.gengbaike.top - LetsEncrypt on sslhost is OK: OK - Certificate 'gengbaike.top' will expire on Tue 19 Nov 2024 05:07:14 PM GMT +0000.
[20:41:46] <icinga-miraheze>	 RECOVERY - db171 ferm_active on db171 is OK: OK ferm input default policy is set
[20:41:47] <icinga-miraheze>	 RECOVERY - phorge171 phd on phorge171 is OK: PROCS OK: 1 process with args 'phd'
[20:41:47] <icinga-miraheze>	 RECOVERY - jobchron171 Current Load on jobchron171 is OK: LOAD OK - total load average: 0.19, 0.11, 0.04
[20:41:48] <icinga-miraheze>	 RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 67 packages available for upgrade (0 critical updates).
[20:41:49] <icinga-miraheze>	 RECOVERY - itora.bunnies1230.com - Cloudflare on sslhost is OK: OK - Certificate 'itora.bunnies1230.com' will expire on Thu 28 Nov 2024 01:31:29 PM GMT +0000.
[20:42:00] <icinga-miraheze>	 RECOVERY - bast161 Current Load on bast161 is OK: LOAD OK - total load average: 0.10, 0.14, 0.07
[20:42:05] <icinga-miraheze>	 RECOVERY - ping on bast161 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms
[20:42:08] <icinga-miraheze>	 RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[20:42:09] <icinga-miraheze>	 RECOVERY - www.satepedia.com - Cloudflare on sslhost is OK: OK - Certificate 'www.satepedia.com' will expire on Fri 29 Nov 2024 05:09:13 PM GMT +0000.
[20:42:09] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:42:10] <icinga-miraheze>	 RECOVERY - ping on swiftproxy171 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms
[20:42:10] <icinga-miraheze>	 RECOVERY - grayzonewarfare.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'grayzonewarfare.wiki' will expire on Sat 09 Nov 2024 08:51:50 PM GMT +0000.
[20:42:10] <icinga-miraheze>	 RECOVERY - wiki.elgeis.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.elgeis.com' will expire on Thu 28 Nov 2024 01:38:30 PM GMT +0000.
[20:42:12] <icinga-miraheze>	 RECOVERY - db171 NTP time on db171 is OK: NTP OK: Offset 4.142522812e-06 secs
[20:42:13] <icinga-miraheze>	 RECOVERY - jobchron171 NTP time on jobchron171 is OK: NTP OK: Offset -0.0001272559166 secs
[20:42:15] <icinga-miraheze>	 RECOVERY - ajhalili2006.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:42:15] <icinga-miraheze>	 RECOVERY - bast161 APT on bast161 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[20:42:15] <icinga-miraheze>	 RECOVERY - ping on swiftac171 is OK: PING OK - Packet loss = 0%, RTA = 0.21 ms
[20:42:16] <icinga-miraheze>	 RECOVERY - wiki.otir.nl - Cloudflare on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Thu 28 Nov 2024 01:40:41 PM GMT +0000.
[20:42:19] <icinga-miraheze>	 RECOVERY - phorge171 NTP time on phorge171 is OK: NTP OK: Offset -4.36604023e-05 secs
[20:42:19] <icinga-miraheze>	 RECOVERY - phorge171 ferm_active on phorge171 is OK: OK ferm input default policy is set
[20:42:20] <icinga-miraheze>	 RECOVERY - phorge171 Disk Space on phorge171 is OK: DISK OK - free space: / 25076MiB (56% inode=93%);
[20:42:20] <icinga-miraheze>	 RECOVERY - closinglogosgroup.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:42:24] <icinga-miraheze>	 RECOVERY - Host reports171 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms
[20:42:24] <icinga-miraheze>	 RECOVERY - www.johanloopmans.nl - Cloudflare on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Thu 28 Nov 2024 03:57:51 PM GMT +0000.
[20:42:25] <icinga-miraheze>	 RECOVERY - pandorastale.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pandorastale.wiki' will expire on Sun 03 Nov 2024 07:10:54 PM GMT +0000.
[20:42:26] <icinga-miraheze>	 RECOVERY - reports171 NTP time on reports171 is OK: NTP OK: Offset 8.499622345e-05 secs
[20:42:26] <icinga-miraheze>	 RECOVERY - otherkin.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'otherkin.wiki' will expire on Sat 09 Nov 2024 03:36:03 PM GMT +0000.
[20:42:27] <icinga-miraheze>	 RECOVERY - www.durawiki.com - Cloudflare on sslhost is OK: OK - Certificate 'www.durawiki.com' will expire on Fri 29 Nov 2024 05:08:34 PM GMT +0000.
[20:42:28] <icinga-miraheze>	 RECOVERY - exordium.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'exordium.wiki' will expire on Fri 29 Nov 2024 08:53:51 PM GMT +0000.
[20:42:29] <icinga-miraheze>	 RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2'
[20:42:33] <icinga-miraheze>	 RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mh142.com' will expire on Thu 28 Nov 2024 03:54:54 PM GMT +0000.
[20:42:33] <icinga-miraheze>	 RECOVERY - wikimas.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimas.kr' will expire on Thu 10 Oct 2024 11:18:42 AM GMT +0000.
[20:42:34] <icinga-miraheze>	 RECOVERY - wiki.funkey-project.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.funkey-project.com' will expire on Thu 28 Nov 2024 01:38:40 PM GMT +0000.
[20:42:34] <icinga-miraheze>	 RECOVERY - lebork.info.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'lebork.info.pl' will expire on Fri 27 Sep 2024 11:25:33 PM GMT +0000.
[20:42:37] <icinga-miraheze>	 RECOVERY - Host mw172 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms
[20:42:39] <icinga-miraheze>	 RECOVERY - Host mw171 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[20:42:40] <icinga-miraheze>	 RECOVERY - Host swiftobject171 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms
[20:42:40] <icinga-miraheze>	 RECOVERY - ping on ldap171 is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms
[20:42:41] <icinga-miraheze>	 RECOVERY - pwiki.drydraytonvillagehall.org.uk - Cloudflare on sslhost is OK: OK - Certificate 'pwiki.drydraytonvillagehall.org.uk' will expire on Thu 28 Nov 2024 01:32:48 PM GMT +0000.
[20:42:43] <icinga-miraheze>	 RECOVERY - phorge.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:42:50] <icinga-miraheze>	 RECOVERY - wiki.namegames.dev - Cloudflare on sslhost is OK: OK - Certificate 'wiki.namegames.dev' will expire on Thu 28 Nov 2024 01:39:35 PM GMT +0000.
[20:42:51] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Wed 09 Oct 2024 07:51:00 AM GMT +0000.
[20:42:52] <icinga-miraheze>	 RECOVERY - www.journeytheword.wiki - Cloudflare on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Fri 29 Nov 2024 05:07:52 PM GMT +0000.
[20:42:56] <icinga-miraheze>	 RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:42:56] <icinga-miraheze>	 RECOVERY - de.berlinwiki.org - Cloudflare on sslhost is OK: OK - Certificate 'de.berlinwiki.org' will expire on Thu 28 Nov 2024 01:25:21 PM GMT +0000.
[20:42:56] <icinga-miraheze>	 RECOVERY - wiki.neonai.dev - Cloudflare on sslhost is OK: OK - Certificate 'wiki.neonai.dev' will expire on Thu 28 Nov 2024 01:39:38 PM GMT +0000.
[20:42:56] <icinga-miraheze>	 RECOVERY - jackbox.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'jackbox.wiki' will expire on Mon 25 Nov 2024 03:33:03 PM GMT +0000.
[20:43:03] <icinga-miraheze>	 RECOVERY - kagaga.jp - LetsEncrypt on sslhost is OK: OK - Certificate 'kagaga.jp' will expire on Tue 22 Oct 2024 06:13:40 PM GMT +0000.
[20:43:06] <icinga-miraheze>	 RECOVERY - creativecommons.id - LetsEncrypt on sslhost is OK: OK - Certificate 'creativecommons.id' will expire on Fri 11 Oct 2024 07:15:53 PM GMT +0000.
[20:43:08] <icinga-miraheze>	 RECOVERY - wiki.ricochetuniverse.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ricochetuniverse.com' will expire on Thu 28 Nov 2024 01:41:35 PM GMT +0000.
[20:43:10] <icinga-miraheze>	 RECOVERY - phorge171 conntrack_table_size on phorge171 is OK: OK: nf_conntrack is 0 % full
[20:43:10] <icinga-miraheze>	 RECOVERY - abeyancewiki.bumblebee-studios.se - Cloudflare on sslhost is OK: OK - Certificate 'abeyancewiki.bumblebee-studios.se' will expire on Thu 28 Nov 2024 02:03:51 PM GMT +0000.
[20:43:15] <icinga-miraheze>	 RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.027 second response time
[20:43:19] <icinga-miraheze>	 RECOVERY - lgbtqia.wiki - Cloudflare on sslhost is OK: OK - Certificate 'lgbtqia.wiki' will expire on Thu 28 Nov 2024 01:31:48 PM GMT +0000.
[20:43:20] <icinga-miraheze>	 RECOVERY - phorge171 PowerDNS Recursor on phorge171 is OK: DNS OK: 0.048 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:43:22] <icinga-miraheze>	 RECOVERY - wiki.cor.fyi - Cloudflare on sslhost is OK: OK - Certificate 'wiki.cor.fyi' will expire on Thu 28 Nov 2024 01:38:11 PM GMT +0000.
[20:43:24] <icinga-miraheze>	 RECOVERY - www.rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rosettacode.org' will expire on Fri 29 Nov 2024 05:09:15 PM GMT +0000.
[20:43:26] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.andreijiroh.uk.eu.org' will expire on Thu 28 Nov 2024 01:37:48 PM GMT +0000.
[20:43:27] <icinga-miraheze>	 RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Wed 09 Oct 2024 08:07:14 AM GMT +0000.
[20:43:28] <icinga-miraheze>	 RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.955 second response time
[20:43:33] <icinga-miraheze>	 RECOVERY - ie.unigon.net - Cloudflare on sslhost is OK: OK - Certificate 'ie.unigon.net' will expire on Thu 28 Nov 2024 01:31:21 PM GMT +0000.
[20:43:35] <icinga-miraheze>	 RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.permanentfuturelab.wiki' will expire on Fri 29 Nov 2024 05:09:06 PM GMT +0000.
[20:43:36] <icinga-miraheze>	 RECOVERY - infectowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'infectowiki.com' will expire on Wed 09 Oct 2024 04:22:42 PM GMT +0000.
[20:43:37] <icinga-miraheze>	 RECOVERY - echoes-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'echoes-wiki.com' will expire on Tue 22 Oct 2024 05:50:06 PM GMT +0000.
[20:43:39] <icinga-miraheze>	 RECOVERY - Host swiftproxy161 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms
[20:43:44] <icinga-miraheze>	 RECOVERY - Host db161 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms
[20:43:48] <icinga-miraheze>	 RECOVERY - wiki.elysiamc.xyz - Cloudflare on sslhost is OK: OK - Certificate 'wiki.elysiamc.xyz' will expire on Thu 28 Nov 2024 01:38:31 PM GMT +0000.
[20:43:49] <icinga-miraheze>	 RECOVERY - Host mw162 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms
[20:43:49] <icinga-miraheze>	 RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 3.165 second response time
[20:43:50] <icinga-miraheze>	 RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.285 second response time
[20:43:51] <icinga-miraheze>	 RECOVERY - dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Wed 30 Oct 2024 11:07:50 PM GMT +0000.
[20:43:52] <icinga-miraheze>	 RECOVERY - Host mem161 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms
[20:43:52] <icinga-miraheze>	 RECOVERY - Host os162 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms
[20:43:54] <icinga-miraheze>	 RECOVERY - rct.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rct.wiki' will expire on Thu 10 Oct 2024 11:11:19 AM GMT +0000.
[20:43:57] <icinga-miraheze>	 RECOVERY - mem161 SSH on mem161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:43:58] <icinga-miraheze>	 RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.068 second response time
[20:44:00] <icinga-miraheze>	 RECOVERY - wiki.villagecollaborative.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Fri 29 Nov 2024 07:59:06 PM GMT +0000.
[20:44:02] <icinga-miraheze>	 RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.067 second response time
[20:44:02] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.040 second response time
[20:44:03] <icinga-miraheze>	 RECOVERY - db161 Backups SQL mhglobal on db161 is OK: FILE_AGE OK: /var/log/sql-mhglobal-backup-weekly.log is 315828 seconds old and 209 bytes
[20:44:04] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.587 second response time
[20:44:05] <icinga-miraheze>	 PROBLEM - swiftproxy161 NTP time on swiftproxy161 is CRITICAL: NTP CRITICAL: Offset 0.7274078429 secs
[20:44:05] <icinga-miraheze>	 PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:44:05] <icinga-miraheze>	 RECOVERY - www.sekaipedia.org - Cloudflare on sslhost is OK: OK - Certificate 'www.sekaipedia.org' will expire on Thu 28 Nov 2024 01:43:35 PM GMT +0000.
[20:44:06] <icinga-miraheze>	 RECOVERY - mw162 SSH on mw162 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:44:06] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.326 second response time
[20:44:07] <icinga-miraheze>	 RECOVERY - thelangyalist.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:44:07] <icinga-miraheze>	 RECOVERY - mem161 Current Load on mem161 is OK: LOAD OK - total load average: 0.57, 0.17, 0.06
[20:44:09] <icinga-miraheze>	 RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.067 second response time
[20:44:10] <icinga-miraheze>	 RECOVERY - ping on mw162 is OK: PING OK - Packet loss = 0%, RTA = 0.87 ms
[20:44:10] <icinga-miraheze>	 RECOVERY - meta.sagan4.org - Cloudflare on sslhost is OK: OK - Certificate 'meta.sagan4.org' will expire on Thu 28 Nov 2024 01:31:48 PM GMT +0000.
[20:44:10] <icinga-miraheze>	 RECOVERY - elsterclopedie.marithaime.nl - Cloudflare on sslhost is OK: OK - Certificate 'elsterclopedie.marithaime.nl' will expire on Thu 28 Nov 2024 01:26:26 PM GMT +0000.
[20:44:11] <icinga-miraheze>	 RECOVERY - ping on db161 is OK: PING OK - Packet loss = 0%, RTA = 0.48 ms
[20:44:11] <icinga-miraheze>	 PROBLEM - mem161 NTP time on mem161 is WARNING: NTP WARNING: Offset 0.4116500318 secs
[20:44:11] <icinga-miraheze>	 RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.071 second response time
[20:44:12] <icinga-miraheze>	 RECOVERY - mw162 conntrack_table_size on mw162 is OK: OK: nf_conntrack is 1 % full
[20:44:12] <icinga-miraheze>	 RECOVERY - exmormon.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'exmormon.wiki' will expire on Mon 25 Nov 2024 03:33:23 PM GMT +0000.
[20:44:14] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 3.34, 0.81, 0.27
[20:44:14] <icinga-miraheze>	 RECOVERY - Host os161 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms
[20:44:14] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.751 second response time
[20:44:27] <icinga-miraheze>	 RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[20:44:27] <icinga-miraheze>	 RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 0.318 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:44:27] <icinga-miraheze>	 RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 1.236 second response time
[20:44:30] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 4.01, 1.29, 0.46
[20:44:31] <icinga-miraheze>	 RECOVERY - mw162 php-fpm on mw162 is OK: PROCS OK: 21 processes with command name 'php-fpm8.2'
[20:44:31] <icinga-miraheze>	 RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 0.104 second response time
[20:44:33] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.038 second response time
[20:44:34] <icinga-miraheze>	 RECOVERY - mw162 Disk Space on mw162 is OK: DISK OK - free space: / 20133MiB (37% inode=86%);
[20:44:37] <icinga-miraheze>	 RECOVERY - wiki.whentheycry.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Thu 28 Nov 2024 01:42:50 PM GMT +0000.
[20:44:39] <icinga-miraheze>	 RECOVERY - baubles.me - LetsEncrypt on sslhost is OK: OK - Certificate 'baubles.me' will expire on Mon 23 Sep 2024 08:39:48 PM GMT +0000.
[20:44:56] <icinga-miraheze>	 RECOVERY - wiki.rotary9220.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.rotary9220.net' will expire on Thu 28 Nov 2024 01:41:39 PM GMT +0000.
[20:44:59] <icinga-miraheze>	 RECOVERY - forecasting.wiki - Cloudflare on sslhost is OK: OK - Certificate 'forecasting.wiki' will expire on Thu 28 Nov 2024 01:29:10 PM GMT +0000.
[20:45:00] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.422 second response time
[20:45:05] <icinga-miraheze>	 RECOVERY - Host graylog161 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms
[20:45:18] <icinga-miraheze>	 RECOVERY - os161 SSH on os161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:45:21] <icinga-miraheze>	 RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 0.96, 0.51, 0.20
[20:45:21] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.021 second response time
[20:45:22] <icinga-miraheze>	 RECOVERY - Host swiftobject161 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms
[20:45:23] <icinga-miraheze>	 RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.858 second response time
[20:45:26] <icinga-miraheze>	 RECOVERY - wiki.arsrobotics.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.arsrobotics.org' will expire on Thu 28 Nov 2024 01:37:51 PM GMT +0000.
[20:45:30] <icinga-miraheze>	 RECOVERY - mem161 ferm_active on mem161 is OK: OK ferm input default policy is set
[20:45:32] <icinga-miraheze>	 RECOVERY - tfuk.app - Cloudflare on sslhost is OK: OK - Certificate 'tfuk.app' will expire on Thu 28 Nov 2024 01:35:45 PM GMT +0000.
[20:45:32] <icinga-miraheze>	 RECOVERY - glitchcity.wiki - Cloudflare on sslhost is OK: OK - Certificate 'glitchcity.wiki' will expire on Thu 28 Nov 2024 02:19:32 PM GMT +0000.
[20:45:34] <icinga-miraheze>	 RECOVERY - richterian.com - LetsEncrypt on sslhost is OK: OK - Certificate 'richterian.com' will expire on Sun 24 Nov 2024 09:20:13 PM GMT +0000.
[20:45:35] <icinga-miraheze>	 RECOVERY - os162 PowerDNS Recursor on os162 is OK: DNS OK: 0.040 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:45:35] <icinga-miraheze>	 RECOVERY - mem161 memcached on mem161 is OK: TCP OK - 0.000 second response time on 10.0.16.131 port 11211
[20:45:35] <icinga-miraheze>	 RECOVERY - iewiki.purnamcommunity.in - Cloudflare on sslhost is OK: OK - Certificate 'iewiki.purnamcommunity.in' will expire on Thu 28 Nov 2024 01:31:22 PM GMT +0000.
[20:45:37] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:45:37] <icinga-miraheze>	 RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.677 second response time
[20:45:37] <icinga-miraheze>	 RECOVERY - graylog161 HTTPS on graylog161 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 1.028 second response time
[20:45:37] <icinga-miraheze>	 RECOVERY - ping on graylog161 is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms
[20:45:40] <icinga-miraheze>	 RECOVERY - mw162 ferm_active on mw162 is OK: OK ferm input default policy is set
[20:45:40] <icinga-miraheze>	 RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 9.8%Current connections: 49
[20:45:45] <icinga-miraheze>	 RECOVERY - os161 conntrack_table_size on os161 is OK: OK: nf_conntrack is 0 % full
[20:45:45] <icinga-miraheze>	 RECOVERY - mw162 PowerDNS Recursor on mw162 is OK: DNS OK: 0.033 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:45:45] <icinga-miraheze>	 RECOVERY - wiki.openhatch.org - Cloudflare on sslhost is OK: OK - Certificate 'wiki.openhatch.org' will expire on Thu 28 Nov 2024 01:39:42 PM GMT +0000.
[20:45:46] <icinga-miraheze>	 RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4009 bytes in 3.454 second response time
[20:45:47] <icinga-miraheze>	 RECOVERY - graylog161 SSH on graylog161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:45:48] <icinga-miraheze>	 RECOVERY - tintinpedia.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:45:50] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.778 second response time
[20:45:50] <icinga-miraheze>	 RECOVERY - os161 PowerDNS Recursor on os161 is OK: DNS OK: 0.053 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:45:50] <icinga-miraheze>	 RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 76148MiB (34% inode=99%);
[20:45:50] <icinga-miraheze>	 RECOVERY - os161 ferm_active on os161 is OK: OK ferm input default policy is set
[20:45:50] <icinga-miraheze>	 RECOVERY - wiki.mineland.eu - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Thu 28 Nov 2024 01:39:29 PM GMT +0000.
[20:45:51] <icinga-miraheze>	 RECOVERY - mw162 NTP time on mw162 is OK: NTP OK: Offset -6.765127182e-06 secs
[20:45:51] <icinga-miraheze>	 RECOVERY - storytime.jdstroy.cf - Cloudflare on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Thu 28 Nov 2024 12:50:12 PM GMT +0000.
[20:45:51] <icinga-miraheze>	 RECOVERY - graylog161 Current Load on graylog161 is OK: LOAD OK - total load average: 6.71, 1.88, 0.65
[20:45:52] <icinga-miraheze>	 RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4008 bytes in 3.690 second response time
[20:45:52] <icinga-miraheze>	 RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 69 packages available for upgrade (0 critical updates).
[20:45:53] <icinga-miraheze>	 RECOVERY - wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[20:45:55] <icinga-miraheze>	 RECOVERY - wikislamica.org - Cloudflare on sslhost is OK: OK - Certificate 'wikislamica.org' will expire on Thu 28 Nov 2024 01:46:54 PM GMT +0000.
[20:45:55] <icinga-miraheze>	 RECOVERY - wiki.zamnhacking.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.zamnhacking.net' will expire on Thu 28 Nov 2024 01:43:02 PM GMT +0000.
[20:45:55] <icinga-miraheze>	 RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4032 bytes in 3.045 second response time
[20:46:07] <icinga-miraheze>	 RECOVERY - wiki.ventistudio.fr - Cloudflare on sslhost is OK: OK - Certificate 'wiki.ventistudio.fr' will expire on Thu 28 Nov 2024 01:42:48 PM GMT +0000.
[20:46:08] <icinga-miraheze>	 RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:46:10] <icinga-miraheze>	 RECOVERY - dutki.kro.kr - Cloudflare on sslhost is OK: OK - Certificate 'dutki.kro.kr' will expire on Thu 28 Nov 2024 01:26:21 PM GMT +0000.
[20:46:10] <icinga-miraheze>	 RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:46:15] <icinga-miraheze>	 RECOVERY - os161 APT on os161 is OK: APT OK: 66 packages available for upgrade (0 critical updates).
[20:46:15] <icinga-miraheze>	 RECOVERY - os162 NTP time on os162 is OK: NTP OK: Offset -3.096461296e-05 secs
[20:46:16] <icinga-miraheze>	 RECOVERY - mem161 NTP time on mem161 is OK: NTP OK: Offset -5.099177361e-05 secs
[20:46:17] <icinga-miraheze>	 RECOVERY - wiki.sofurry.games - Cloudflare on sslhost is OK: OK - Certificate 'wiki.sofurry.games' will expire on Thu 28 Nov 2024 01:41:53 PM GMT +0000.
[20:46:20] <icinga-miraheze>	 RECOVERY - swiftobject161 PowerDNS Recursor on swiftobject161 is OK: DNS OK: 0.036 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:46:27] <icinga-miraheze>	 RECOVERY - os161 NTP time on os161 is OK: NTP OK: Offset 8.046627045e-06 secs
[20:46:35] <icinga-miraheze>	 RECOVERY - graylog161 ferm_active on graylog161 is OK: OK ferm input default policy is set
[20:46:41] <icinga-miraheze>	 RECOVERY - www.tagpedia.jp - Cloudflare on sslhost is OK: OK - Certificate 'www.tagpedia.jp' will expire on Mon 02 Dec 2024 05:05:25 PM GMT +0000.
[20:46:42] <icinga-miraheze>	 RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:46:48] <icinga-miraheze>	 RECOVERY - eros.locksteel.net - Cloudflare on sslhost is OK: OK - Certificate 'eros.locksteel.net' will expire on Thu 28 Nov 2024 01:50:40 PM GMT +0000.
[20:46:55] <icinga-miraheze>	 RECOVERY - education.gced.in - Cloudflare on sslhost is OK: OK - Certificate 'education.gced.in' will expire on Thu 28 Nov 2024 01:26:22 PM GMT +0000.
[20:47:02] <icinga-miraheze>	 RECOVERY - wiki.widwwa.co.uk - Cloudflare on sslhost is OK: OK - Certificate 'wiki.widwwa.co.uk' will expire on Thu 28 Nov 2024 01:42:51 PM GMT +0000.
[20:47:02] <icinga-miraheze>	 RECOVERY - wiki.eversd.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.eversd.com' will expire on Thu 28 Nov 2024 01:38:36 PM GMT +0000.
[20:47:03] <icinga-miraheze>	 RECOVERY - wiki.mc.marochuru.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mc.marochuru.net' will expire on Fri 29 Nov 2024 07:58:02 PM GMT +0000.
[20:47:05] <icinga-miraheze>	 RECOVERY - minigamemadness.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:47:07] <icinga-miraheze>	 RECOVERY - test2.specialchat.org - Cloudflare on sslhost is OK: OK - Certificate 'test2.specialchat.org' will expire on Thu 28 Nov 2024 01:35:40 PM GMT +0000.
[20:47:07] <icinga-miraheze>	 RECOVERY - wiki.orvyn.ca - Cloudflare on sslhost is OK: OK - Certificate 'wiki.orvyn.ca' will expire on Fri 29 Nov 2024 07:58:09 PM GMT +0000.
[20:47:10] <icinga-miraheze>	 RECOVERY - www.trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.trollpasta.com' will expire on Fri 29 Nov 2024 05:06:20 PM GMT +0000.
[20:47:11] <icinga-miraheze>	 RECOVERY - chobots.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'chobots.wiki' will expire on Wed 16 Oct 2024 07:29:39 AM GMT +0000.
[20:47:11] <icinga-miraheze>	 RECOVERY - www.mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mwcosmos.com' will expire on Fri 29 Nov 2024 05:06:15 PM GMT +0000.
[20:47:20] <icinga-miraheze>	 RECOVERY - msmhacks.uk - Cloudflare on sslhost is OK: OK - Certificate 'msmhacks.uk' will expire on Thu 28 Nov 2024 01:31:58 PM GMT +0000.
[20:47:25] <icinga-miraheze>	 RECOVERY - bast161 PowerDNS Recursor on bast161 is OK: DNS OK: 0.369 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:47:26] <MirahezeRelay>	 <ellie.rose._> oh that's back
[20:47:27] <icinga-miraheze>	 RECOVERY - wiki.starship.digital - Cloudflare on sslhost is OK: OK - Certificate 'wiki.starship.digital' will expire on Thu 28 Nov 2024 01:41:58 PM GMT +0000.
[20:47:29] <icinga-miraheze>	 RECOVERY - wiki.hwfib.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.hwfib.com' will expire on Thu 28 Nov 2024 12:48:29 PM GMT +0000.
[20:47:31] <MirahezeRelay>	 <pixldev> oh boy
[20:47:31] <icinga-miraheze>	 RECOVERY - wiki.autocountsoft.com - Cloudflare on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Thu 28 Nov 2024 01:37:55 PM GMT +0000.
[20:47:31] <icinga-miraheze>	 RECOVERY - wiki.walkscape.app - Cloudflare on sslhost is OK: OK - Certificate 'wiki.walkscape.app' will expire on Thu 28 Nov 2024 01:42:49 PM GMT +0000.
[20:47:35] <icinga-miraheze>	 RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset -2.038478851e-05 secs
[20:47:38] <MirahezeRelay>	 <pixldev> oh relay is back
[20:47:39] <icinga-miraheze>	 RECOVERY - slymods.info - Cloudflare on sslhost is OK: OK - Certificate 'slymods.info' will expire on Thu 28 Nov 2024 01:34:18 PM GMT +0000.
[20:47:42] <icinga-miraheze>	 RECOVERY - index.pdcommunity.ir - Cloudflare on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Thu 28 Nov 2024 01:31:26 PM GMT +0000.
[20:47:43] <icinga-miraheze>	 RECOVERY - pso2ngs.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pso2ngs.wiki' will expire on Tue 05 Nov 2024 12:06:00 PM GMT +0000.
[20:47:43] <MirahezeRelay>	 <pixldev> oh god the floood
[20:47:45] <MirahezeRelay>	 <orduin> okie, time to move this to #tech
[20:47:48] <icinga-miraheze>	 RECOVERY - metroid.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:47:51] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:47:58] <icinga-miraheze>	 RECOVERY - drachronicles.voxel.gay - Cloudflare on sslhost is OK: OK - Certificate 'drachronicles.voxel.gay' will expire on Thu 28 Nov 2024 01:26:19 PM GMT +0000.
[20:48:02] <icinga-miraheze>	 RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:48:02] <icinga-miraheze>	 RECOVERY - wiki.sheepservermc.net - Cloudflare on sslhost is OK: OK - Certificate 'wiki.sheepservermc.net' will expire on Thu 28 Nov 2024 01:41:48 PM GMT +0000.
[20:48:02] <icinga-miraheze>	 RECOVERY - wiki.newnormal.fans - Cloudflare on sslhost is OK: OK - Certificate 'wiki.newnormal.fans' will expire on Thu 28 Nov 2024 01:39:39 PM GMT +0000.
[20:48:08] <icinga-miraheze>	 RECOVERY - wiki.3point0.science - Cloudflare on sslhost is OK: OK - Certificate 'wiki.3point0.science' will expire on Thu 28 Nov 2024 01:37:39 PM GMT +0000.
[20:48:14] <icinga-miraheze>	 RECOVERY - projectsekai.miraheze.org - Sectigo on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 14 Nov 2024 09:02:47 PM GMT +0000.
[20:48:14] <icinga-miraheze>	 RECOVERY - www.thegreatwar.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thegreatwar.uk' will expire on Fri 29 Nov 2024 05:29:13 PM GMT +0000.
[20:48:52] <icinga-miraheze>	 RECOVERY - Host mw161 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms
[20:49:10] <icinga-miraheze>	 RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:49:15] <icinga-miraheze>	 RECOVERY - mw161 php-fpm on mw161 is OK: PROCS OK: 21 processes with command name 'php-fpm8.2'
[20:49:31] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 4.89, 1.10, 0.36
[20:49:37] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy
[20:49:38] <icinga-miraheze>	 RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3814 bytes in 2.424 second response time
[20:49:48] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[20:49:48] <icinga-miraheze>	 PROBLEM - graylog161 Current Load on graylog161 is CRITICAL: LOAD CRITICAL - total load average: 8.09, 5.71, 2.56
[20:50:09] <icinga-miraheze>	 RECOVERY - mw161 ferm_active on mw161 is OK: OK ferm input default policy is set
[20:50:14] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[20:50:19] <icinga-miraheze>	 RECOVERY - mw161 conntrack_table_size on mw161 is OK: OK: nf_conntrack is 3 % full
[20:50:21] <icinga-miraheze>	 RECOVERY - mw161 PowerDNS Recursor on mw161 is OK: DNS OK: 0.341 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:50:25] <icinga-miraheze>	 RECOVERY - mw161 APT on mw161 is OK: APT OK: 91 packages available for upgrade (0 critical updates).
[20:50:35] <icinga-miraheze>	 RECOVERY - mw161 Disk Space on mw161 is OK: DISK OK - free space: / 20250MiB (38% inode=86%);
[20:50:36] <icinga-miraheze>	 RECOVERY - mw161 SSH on mw161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:50:37] <icinga-miraheze>	 RECOVERY - ping on mw161 is OK: PING OK - Packet loss = 0%, RTA = 0.35 ms
[20:50:49] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[20:51:19] <icinga-miraheze>	 RECOVERY - mw161 NTP time on mw161 is OK: NTP OK: Offset 9.09268856e-05 secs
[20:51:19] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[20:51:47] <icinga-miraheze>	 PROBLEM - graylog161 Current Load on graylog161 is WARNING: LOAD WARNING - total load average: 7.65, 6.37, 3.19
[20:52:58] <icinga-miraheze>	 PROBLEM - ns1 NTP time on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host
[20:53:19] <MirahezeRelay>	 <pixldev> uh oh
[20:53:46] <icinga-miraheze>	 PROBLEM - graylog161 Current Load on graylog161 is CRITICAL: LOAD CRITICAL - total load average: 8.41, 7.10, 3.85
[20:53:50] <icinga-miraheze>	 RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[20:54:14] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[20:54:21] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[20:54:32] <MirahezeRelay>	 <pixldev> is this bad
[20:54:48] <MirahezeRelay>	 <orduin> No
[20:54:53] <icinga-miraheze>	 RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 6.812810898e-05 secs
[20:55:03] <MirahezeRelay>	 <orduin> ns1 wasn't fixing itself, it needed a longer nap
[20:55:21] <icinga-miraheze>	 PROBLEM - cp37 NTP time on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[20:55:26] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:55:31] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[20:55:59] <MirahezeRelay>	 <orduin> same with cp37
[20:56:08] <icinga-miraheze>	 PROBLEM - ping on cp37 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.138)
[20:56:11] <icinga-miraheze>	 PROBLEM - cp37 SSH on cp37 is CRITICAL: connect to address 10.0.17.138 and port 22: No route to host
[20:56:15] <icinga-miraheze>	 PROBLEM - cp37 Current Load on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host
[20:57:18] <icinga-miraheze>	 PROBLEM - cp37 NTP time on cp37 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[20:57:40] <icinga-miraheze>	 RECOVERY - cp37 SSH on cp37 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[20:57:44] <icinga-miraheze>	 PROBLEM - graylog161 Current Load on graylog161 is WARNING: LOAD WARNING - total load average: 7.64, 7.54, 4.78
[20:57:46] <icinga-miraheze>	 RECOVERY - cp37 Current Load on cp37 is OK: LOAD OK - total load average: 0.35, 0.17, 0.06
[20:57:54] <icinga-miraheze>	 RECOVERY - ping on cp37 is OK: PING OK - Packet loss = 0%, RTA = 0.31 ms
[20:59:26] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:00:29] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for puppet181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8204
[21:00:32] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for matomo151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8203
[21:00:37] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8113
[21:01:01] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for swiftproxy171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8207
[21:01:03] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for reports171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8205
[21:01:08] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mon181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8201
[21:01:08] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw182 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8120
[21:01:12] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw152 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8114
[21:01:13] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8119
[21:01:17] <icinga-miraheze>	 RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[21:01:18] <icinga-miraheze>	 PROBLEM - cp37 NTP time on cp37 is WARNING: NTP WARNING: Offset -0.4349020422 secs
[21:01:19] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8117
[21:01:20] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:01:25] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for test151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8181
[21:01:31] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw172 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8118
[21:01:35] <icinga-miraheze>	 RECOVERY - cp37 PowerDNS Recursor on cp37 is OK: DNS OK: 0.057 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[21:01:41] <icinga-miraheze>	 RECOVERY - graylog161 Current Load on graylog161 is OK: LOAD OK - total load average: 2.34, 5.57, 4.68
[21:01:42] <icinga-miraheze>	 RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4007 bytes in 0.080 second response time
[21:01:46] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw162 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8116
[21:01:52] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for swiftproxy161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8206
[21:01:53] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mwtask181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8160
[21:01:55] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy
[21:01:56] <icinga-miraheze>	 RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 26%
[21:02:03] <icinga-miraheze>	 RECOVERY - landofliberos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'landofliberos.com' will expire on Fri 11 Oct 2024 10:16:43 AM GMT +0000.
[21:02:07] <icinga-miraheze>	 RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 08 Oct 2024 10:59:51 PM GMT +0000.
[21:02:11] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Thu 10 Oct 2024 07:27:39 PM GMT +0000.
[21:02:14] <icinga-miraheze>	 RECOVERY - crashspyrowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crashspyrowiki.com' will expire on Fri 11 Oct 2024 05:04:46 PM GMT +0000.
[21:02:15] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for phorge171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8202
[21:02:22] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mw161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8115
[21:02:25] <icinga-miraheze>	 RECOVERY - cp37 Nginx Backend for mwtask171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8161
[21:02:27] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[21:02:30] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:02:35] <icinga-miraheze>	 RECOVERY - baligawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'baligawiki.org' will expire on Wed 06 Nov 2024 08:35:03 PM GMT +0000.
[21:02:36] <icinga-miraheze>	 RECOVERY - royal-wiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'royal-wiki.org' will expire on Fri 11 Oct 2024 01:33:31 AM GMT +0000.
[21:02:52] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[21:02:53] <icinga-miraheze>	 RECOVERY - apeirology.com - LetsEncrypt on sslhost is OK: OK - Certificate 'apeirology.com' will expire on Fri 11 Oct 2024 05:36:17 PM GMT +0000.
[21:02:58] <icinga-miraheze>	 RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Mon 21 Oct 2024 03:28:07 AM GMT +0000.
[21:03:05] <icinga-miraheze>	 RECOVERY - archivesofhavnor.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archivesofhavnor.org' will expire on Mon 25 Nov 2024 03:33:55 PM GMT +0000.
[21:03:17] <icinga-miraheze>	 RECOVERY - wikappedia.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'wikappedia.cc' will expire on Fri 25 Oct 2024 03:21:01 PM GMT +0000.
[21:03:18] <icinga-miraheze>	 RECOVERY - volunteerforukraine.com - LetsEncrypt on sslhost is OK: OK - Certificate 'volunteerforukraine.com' will expire on Thu 21 Nov 2024 04:27:53 PM GMT +0000.
[21:03:18] <icinga-miraheze>	 RECOVERY - cp37 NTP time on cp37 is OK: NTP OK: Offset -0.000156879425 secs
[21:03:21] <icinga-miraheze>	 RECOVERY - rodzinka.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rodzinka.wiki' will expire on Fri 29 Nov 2024 08:52:05 PM GMT +0000.
[21:03:25] <icinga-miraheze>	 RECOVERY - phighting.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'phighting.wiki' will expire on Fri 01 Nov 2024 08:04:07 PM GMT +0000.
[21:03:28] <icinga-miraheze>	 RECOVERY - familiacorsi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'familiacorsi.com' will expire on Tue 22 Oct 2024 05:16:02 PM GMT +0000.
[21:03:36] <icinga-miraheze>	 RECOVERY - looneypyramids.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'looneypyramids.wiki' will expire on Fri 11 Oct 2024 09:34:01 AM GMT +0000.
[21:03:38] <icinga-miraheze>	 RECOVERY - isocasg.org - Cloudflare on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Sat 27 Aug 2039 04:37:00 PM GMT +0000.
[21:03:45] <icinga-miraheze>	 RECOVERY - smashbroswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'smashbroswiki.com' will expire on Thu 10 Oct 2024 02:38:52 PM GMT +0000.
[21:03:49] <icinga-miraheze>	 RECOVERY - bristolstudenthousingcoop.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bristolstudenthousingcoop.org' will expire on Wed 09 Oct 2024 12:01:13 PM GMT +0000.
[21:03:50] <icinga-miraheze>	 RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 09 Oct 2024 10:18:01 AM GMT +0000.
[21:03:56] <icinga-miraheze>	 RECOVERY - puzzles.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.puzzles.wiki' will expire on Thu 10 Oct 2024 09:52:21 AM GMT +0000.
[21:04:01] <icinga-miraheze>	 RECOVERY - evilgeniuswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'evilgeniuswiki.com' will expire on Sun 13 Oct 2024 01:35:51 PM GMT +0000.
[21:04:08] <icinga-miraheze>	 RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Wed 09 Oct 2024 08:44:19 AM GMT +0000.
[21:05:16] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures
[21:06:09] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[21:06:16] <icinga-miraheze>	 RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures
[21:06:27] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[21:06:30] <icinga-miraheze>	 RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures
[21:15:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:21:23] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages libpython3.11, python3.11, libpython3.11-stdlib, python3.11-minimal, libpython3.11-minimal, libgssapi-krb5-2, libkrb5-3, libkrb5support0, libk5crypto3, bind9-dnsutils, bind9-host, bind9-libs, dnsutils, libaom3, and libpq5 on ns2
[21:21:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:22:04] <icinga-miraheze>	 RECOVERY - ns2 APT on ns2 is OK: APT OK: 69 packages available for upgrade (0 critical updates).
[21:30:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:58:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:03:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:48:37] <icinga-miraheze>	 PROBLEM - mem151 Current Load on mem151 is CRITICAL: LOAD CRITICAL - total load average: 4.66, 3.02, 1.41
[22:50:37] <icinga-miraheze>	 RECOVERY - mem151 Current Load on mem151 is OK: LOAD OK - total load average: 0.64, 2.03, 1.24
[22:51:48] <icinga-miraheze>	 PROBLEM - mem151 memcached on mem151 is CRITICAL: connect to address 10.0.15.113 and port 11211: Connection refused
[22:52:46] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw182
[22:53:16] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw162
[22:53:19] <MirahezeRelay>	 <ellie.rose._> is that bad?
[22:53:19] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw152 mw162
[22:53:29] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152
[22:54:13] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 2 backends are down. mw161 mw181
[22:55:15] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy
[22:57:29] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[22:58:46] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy
[22:59:19] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[23:00:13] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[23:01:48] <icinga-miraheze>	 RECOVERY - mem151 memcached on mem151 is OK: TCP OK - 0.000 second response time on 10.0.15.113 port 11211
[23:01:50] <MirahezeLSBot>	 !log [void@mem151] restart memcached
[23:02:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:08:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:43:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1