[01:17:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:47:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:32:20] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:42:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:07:48] <icinga-miraheze>	 PROBLEM - wonderfuleveryday.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wonderfuleveryday.org' expires in 15 day(s) (Mon 08 Jul 2024 03:54:05 AM GMT +0000).
[04:08:03] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bc4613f0b90e...bc2ac3c09802
[04:08:05] <botifico>	 [02ssl] 07WikiTideSSLBot 03bc2ac3c - Bot: Update SSL cert for wonderfuleveryday.org
[04:16:21] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[04:37:18] <icinga-miraheze>	 RECOVERY - wonderfuleveryday.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wonderfuleveryday.org' will expire on Fri 20 Sep 2024 03:07:54 AM GMT +0000.
[04:44:00] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[06:19:07] <icinga-miraheze>	 PROBLEM - wiki.yuanpi.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yuanpi.eu.org could not be found
[07:17:19] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/sysctl.d]
[07:44:34] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[08:47:24] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[08:49:19] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0006404817104 secs
[09:07:25] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:09:19] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0006966292858 secs
[09:47:19] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:15:46] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[10:33:13] <icinga-miraheze>	 PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cpt-ra.bid. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[11:02:47] <icinga-miraheze>	 RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:13:52] <icinga-miraheze>	 PROBLEM - db151 Backups SQL on db151 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864151 seconds old and 133087 bytes
[11:16:20] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:18:15] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001627504826 secs
[13:05:05] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[13:07:10] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:09:24] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[13:11:19] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0002363324165 secs
[14:45:21] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:47:16] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0007333755493 secs
[15:07:09] <icinga-miraheze>	 PROBLEM - thegreatwar.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thegreatwar.uk' expires in 15 day(s) (Mon 08 Jul 2024 03:02:33 PM GMT +0000).
[15:31:28] <icinga-miraheze>	 PROBLEM - www.thegreatwar.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thegreatwar.uk' expires in 15 day(s) (Mon 08 Jul 2024 03:02:33 PM GMT +0000).
[15:37:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:01:06] <icinga-miraheze>	 PROBLEM - www.thegreatwar.uk - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[16:05:31] <icinga-miraheze>	 PROBLEM - wiki.yuanpi.eu.org - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket
[16:12:02] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:16:02] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001121163368 secs
[16:30:34] <icinga-miraheze>	 PROBLEM - www.thegreatwar.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thegreatwar.uk' expires in 15 day(s) (Mon 08 Jul 2024 03:02:33 PM GMT +0000).
[17:12:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:16:37] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:33:36] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03pywikibot-userconfig-ownerro [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b7c30c461dab
[17:33:39] <botifico>	 [02puppet] 07redbluegreenhat 03b7c30c4 - pywikibot: make user-config read only for the irc user
[17:33:42] <botifico>	 [02puppet] 07redbluegreenhat created branch 03pywikibot-userconfig-ownerro - 13https://github.com/miraheze/puppet
[17:34:37] <botifico>	 [02puppet] 07redbluegreenhat opened pull request 03#3873: pywikibot: make user-config read-only for the irc user - 13https://github.com/miraheze/puppet/pull/3873
[17:34:52] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3873: pywikibot: make user-config read-only for the irc user - 13https://github.com/miraheze/puppet/pull/3873#issuecomment-2184123478
[17:35:59] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9e0a684bd8aa...6f333751ac5d
[17:36:00] <botifico>	 [02puppet] 07redbluegreenhat 036f33375 - pywikibot: make user-config read-only for the irc user (#3873)
[17:36:01] <botifico>	 [02puppet] 07redbluegreenhat closed pull request 03#3873: pywikibot: make user-config read-only for the irc user - 13https://github.com/miraheze/puppet/pull/3873
[17:36:04] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03pywikibot-userconfig-ownerro
[17:36:06] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03pywikibot-userconfig-ownerro - 13https://github.com/miraheze/puppet
[17:45:00] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:55:05] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03pywikibot-fix-logging [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9efd95d993dc
[17:55:06] <botifico>	 [02puppet] 07redbluegreenhat 039efd95d - pywikibot: write logs to the /var/log/pwb dir
[17:55:09] <botifico>	 [02puppet] 07redbluegreenhat created branch 03pywikibot-fix-logging - 13https://github.com/miraheze/puppet
[17:55:22] <botifico>	 [02puppet] 07redbluegreenhat opened pull request 03#3874: pywikibot: write logs to the /var/log/pwb dir - 13https://github.com/miraheze/puppet/pull/3874
[17:55:24] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3874: pywikibot: write logs to the /var/log/pwb dir - 13https://github.com/miraheze/puppet/pull/3874#issuecomment-2184131000
[17:55:50] <botifico>	 [02puppet] 07github-actions[bot] pushed 031 commit to 03pywikibot-fix-logging [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9efd95d993dc...99c0f32e826c
[17:55:53] <botifico>	 [02puppet] 07github-actions 0399c0f32 - CI: lint puppet code to standards
[17:55:56] <botifico>	 [02puppet] 07github-actions[bot] synchronize pull request 03#3874: pywikibot: write logs to the /var/log/pwb dir - 13https://github.com/miraheze/puppet/pull/3874
[17:56:35] <botifico>	 [02puppet] 07redbluegreenhat closed pull request 03#3874: pywikibot: write logs to the /var/log/pwb dir - 13https://github.com/miraheze/puppet/pull/3874
[17:56:37] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03pywikibot-fix-logging
[17:56:40] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6f333751ac5d...cb369b5e0e4d
[17:56:41] <botifico>	 [02puppet] 07redbluegreenhat 03cb369b5 - pywikibot: write logs to the /var/log/pwb dir (#3874)
[17:56:44] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03pywikibot-fix-logging - 13https://github.com/miraheze/puppet
[18:08:21] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cb369b5e0e4d...44f491ae8501
[18:08:22] <botifico>	 [02puppet] 07redbluegreenhat 0344f491a - pywikibot: add missing coma
[18:09:30] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:11:31] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[18:22:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:47:01] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:13:53] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[22:22:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1