[00:00:53] <icinga-miraheze>	 RECOVERY - changeprop151 Disk Space on changeprop151 is OK: DISK OK - free space: / 3767MiB (20% inode=89%);
[00:16:04] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 10.34, 18.81, 22.62
[00:20:04] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 10.53, 14.57, 19.99
[01:42:28] <botifico>	 [02mw-config] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/b50c68ded2d8
[01:42:30] <botifico>	 [02mw-config] 07The-Voidwalker 03b50c68d - set $wgCdnServers when cloudflare not in use
[01:42:31] <botifico>	 [02mw-config] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/mw-config
[01:42:47] <botifico>	 [02mw-config] 07The-Voidwalker opened pull request 03#5602: set $wgCdnServers when cloudflare not in use - 13https://github.com/miraheze/mw-config/pull/5602
[01:43:20] <botifico>	 [02mw-config] 07github-actions[bot] pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/b50c68ded2d8...3bcc270d0ac8
[01:43:22] <botifico>	 [02mw-config] 07github-actions 033bcc270 - CI: lint code to MediaWiki standards
[01:43:25] <botifico>	 [02mw-config] 07github-actions[bot] synchronize pull request 03#5602: set $wgCdnServers when cloudflare not in use - 13https://github.com/miraheze/mw-config/pull/5602
[01:43:49] <miraheze-github>	 miraheze/mw-config - The-Voidwalker the build passed.
[03:01:09] <icinga-miraheze>	 PROBLEM - mon181 Backups Grafana on mon181 is CRITICAL: FILE_AGE CRITICAL: /var/log/grafana-backup.log is 1209649 seconds old and 93 bytes
[03:15:10] <botifico>	 [02mw-config] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3bcc270d0ac8...f27c7f4e6bf7
[03:15:13] <botifico>	 [02mw-config] 07The-Voidwalker 03f27c7f4 - fix linter weirdness part 1
[03:15:15] <botifico>	 [02mw-config] 07The-Voidwalker synchronize pull request 03#5602: set $wgCdnServers when cloudflare not in use - 13https://github.com/miraheze/mw-config/pull/5602
[03:16:13] <miraheze-github>	 miraheze/mw-config - The-Voidwalker the build passed.
[03:17:25] <botifico>	 [02mw-config] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f27c7f4e6bf7...a3bef1830ad8
[03:17:27] <botifico>	 [02mw-config] 07The-Voidwalker 03a3bef18 - fix additional spacing issues
[03:17:30] <botifico>	 [02mw-config] 07The-Voidwalker synchronize pull request 03#5602: set $wgCdnServers when cloudflare not in use - 13https://github.com/miraheze/mw-config/pull/5602
[03:18:18] <miraheze-github>	 miraheze/mw-config - The-Voidwalker the build passed.
[04:17:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:22:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[04:46:21] <icinga-miraheze>	 PROBLEM - worldtriggerwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'worldtriggerwiki.com' expires in 7 day(s) (Wed 17 Jul 2024 04:45:38 AM GMT +0000).
[08:45:08] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[09:15:25] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[10:54:39] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[10:54:53] <icinga-miraheze>	 PROBLEM - changeprop151 Disk Space on changeprop151 is WARNING: DISK WARNING - free space: / 1973MiB (10% inode=89%);
[11:23:13] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[11:57:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:54:36] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[13:05:46] <botifico>	 [02mw-config] 07redbluegreenhat commented on pull request 03#5602: set $wgCdnServers when cloudflare not in use - 13https://github.com/miraheze/mw-config/pull/5602#issuecomment-2217693592
[13:52:28] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.34, 16.56, 12.49
[13:52:34] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[13:53:24] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[13:53:25] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[13:53:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:53:31] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[13:53:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:54:26] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.49, 16.22, 12.86
[13:57:48] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:59:44] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0007323920727 secs
[14:02:23] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[14:02:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:02:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:02:34] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 11s
[14:02:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:07:46] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[14:07:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:07:55] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s
[14:07:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:15:30] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[15:14:34] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:43:23] <icinga-miraheze>	 PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.401 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[16:18:52] <icinga-miraheze>	 PROBLEM - changeprop151 Disk Space on changeprop151 is CRITICAL: DISK CRITICAL - free space: / 1079MiB (5% inode=89%);
[16:24:59] <icinga-miraheze>	 PROBLEM - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[16:47:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:03:24] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/4c28ac6debc9...00c1af6d3e72
[17:03:27] <botifico>	 [02ssl] 07WikiTideSSLBot 0300c1af6 - Bot: Add SSL cert for gimkit.wiki
[17:04:39] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/00c1af6d3e72...c683d804c434
[17:04:42] <botifico>	 [02ssl] 07WikiTideSSLBot 03c683d80 - Bot: Add SSL cert for junxions.wiki
[17:11:07] <icinga-miraheze>	 PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - cyberlaw.ccdcoe.org All nameservers failed to answer the query.
[17:41:08] <icinga-miraheze>	 PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[18:10:27] <icinga-miraheze>	 PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - cyberlaw.ccdcoe.org All nameservers failed to answer the query.
[18:51:11] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:53:23] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:55:30] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:57:48] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:59:45] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0003328621387 secs
[19:09:02] <icinga-miraheze>	 RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: SSL OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[19:22:35] <icinga-miraheze>	 RECOVERY - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberlaw.ccdcoe.org' will expire on Sat 27 Jul 2024 10:07:47 PM GMT +0000.
[19:47:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:50:51] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[19:52:49] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001047730446 secs
[20:10:27] <icinga-miraheze>	 PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[20:22:40] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[20:41:17] <icinga-miraheze>	 PROBLEM - archive.a2b2.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.a2b2.org could not be found
[20:51:41] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.60, 17.34, 12.50
[20:51:57] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.68, 15.48, 10.67
[20:52:19] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[20:53:41] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.72, 15.33, 12.38
[20:53:57] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 9.01, 12.69, 10.23
[22:12:39] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': ['ArticleRatings', 'CheckUser'], 'upgrade_skins': ['Metrolook', 'Nimbus']} to all
[22:12:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:13:15] <MirahezeLSBot>	 !log [alex@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': ['ArticleRatings', 'CheckUser'], 'upgrade_skins': ['Metrolook', 'Nimbus']} to all - SUCCESS in 36s
[22:13:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:14:10] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': ['ArticleRatings', 'CheckUser'], 'upgrade_skins': ['Metrolook', 'Nimbus']} to test151
[22:14:12] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': ['ArticleRatings', 'CheckUser'], 'upgrade_skins': ['Metrolook', 'Nimbus']} to test151 - SUCCESS in 2s
[22:14:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:14:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:17:54] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'MediaWikiChat'} to test151
[22:17:55] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'MediaWikiChat'} to test151 - SUCCESS in 1s
[22:17:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:18:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:18:30] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'MediaWikiChat'} to all
[22:18:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:18:41] <MirahezeLSBot>	 !log [alex@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'MediaWikiChat'} to all - SUCCESS in 11s
[22:18:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:53:50] <botifico>	 [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/setuptools-70.3.0 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/7e91ef3b8108
[23:53:54] <botifico>	 [02python-functions] 07dependabot[bot] 037e91ef3 - Bump setuptools from 70.2.0 to 70.3.0 in /.github
[23:53:57] <botifico>	 [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/setuptools-70.3.0 - 13https://github.com/miraheze/python-functions
[23:53:58] <botifico>	 [02python-functions] 07dependabot[bot] labeled pull request 03#37: Bump setuptools from 70.2.0 to 70.3.0 in /.github - 13https://github.com/miraheze/python-functions/pull/37
[23:54:00] <botifico>	 [02python-functions] 07dependabot[bot] opened pull request 03#37: Bump setuptools from 70.2.0 to 70.3.0 in /.github - 13https://github.com/miraheze/python-functions/pull/37
[23:54:02] <botifico>	 [02python-functions] 07dependabot[bot] labeled pull request 03#37: Bump setuptools from 70.2.0 to 70.3.0 in /.github - 13https://github.com/miraheze/python-functions/pull/37
[23:54:04] <botifico>	 [02python-functions] 07coderabbitai[bot] commented on pull request 03#37: Bump setuptools from 70.2.0 to 70.3.0 in /.github - 13https://github.com/miraheze/python-functions/pull/37#issuecomment-2219047950
[23:58:11] <miraheze-github>	 miraheze/python-functions - dependabot[bot] the build passed.