[00:00:16] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN
[00:00:16] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN
[00:02:14] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100%
[00:02:16] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100%
[00:04:19] <icinga-miraheze>	 RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21502MiB (24% inode=98%);
[00:04:29] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21495MiB (24% inode=98%);
[00:19:06] <botifico>	 [02puppet] 07Universal-Omega closed pull request 03#3759: role::db: support daily backups - 13https://github.com/miraheze/puppet/pull/3759
[00:19:09] <botifico>	 [02miraheze/puppet] 07Universal-Omega deleted branch 03db-daily-backups
[00:19:10] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3552f5bbafb7...0cfcc6298753
[00:19:11] <botifico>	 [02miraheze/puppet] 07Universal-Omega 030cfcc62 - role::db: support daily backups (#3759)
[00:19:12] <botifico>	 [02puppet] 07Universal-Omega deleted branch 03db-daily-backups - 13https://github.com/miraheze/puppet
[00:19:49] <botifico>	 [02puppet] 07Universal-Omega closed pull request 03#3753: mariadb: allow enabling slow log - 13https://github.com/miraheze/puppet/pull/3753
[00:19:52] <botifico>	 [02miraheze/puppet] 07Universal-Omega deleted branch 03mariadb-slow_log
[00:19:53] <botifico>	 [02puppet] 07Universal-Omega deleted branch 03mariadb-slow_log - 13https://github.com/miraheze/puppet
[00:19:56] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/0cfcc6298753...05da265a9719
[00:19:57] <botifico>	 [02miraheze/puppet] 07Universal-Omega 0305da265 - mariadb: allow enabling slow log (#3753)
[00:19:59] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[00:20:54] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[00:23:14] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3281: check_reverse_dns: don't force mw-lb.miraheze.org - 13https://github.com/miraheze/puppet/pull/3281
[00:23:43] <botifico>	 [02puppet] 07Universal-Omega closed pull request 03#3281: check_reverse_dns: don't force mw-lb.miraheze.org - 13https://github.com/miraheze/puppet/pull/3281
[00:23:44] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/05da265a9719...1d614f344073
[00:23:46] <botifico>	 [02miraheze/puppet] 07MacFan4000 031d614f3 - check_reverse_dns: don't force mw-lb.miraheze.org (#3281)
[00:24:41] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[00:25:05] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[00:28:12] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03fix-puppetlint [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/881b5a9eebdb
[00:28:13] <botifico>	 [02miraheze/puppet] 07Universal-Omega 03881b5a9 - pywikibot: fix puppet lint
[00:28:14] <botifico>	 [02puppet] 07Universal-Omega created branch 03fix-puppetlint - 13https://github.com/miraheze/puppet
[00:28:17] <botifico>	 [02puppet] 07Universal-Omega opened pull request 03#3770: pywikibot: fix puppet lint - 13https://github.com/miraheze/puppet/pull/3770
[00:28:52] <botifico>	 [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03fix-puppetlint [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/881b5a9eebdb...aa8d1097065a
[00:28:55] <botifico>	 [02miraheze/puppet] 07github-actions 03aa8d109 - CI: lint puppet code to standards
[00:28:57] <botifico>	 [02puppet] 07github-actions[bot] synchronize pull request 03#3770: pywikibot: fix puppet lint - 13https://github.com/miraheze/puppet/pull/3770
[00:29:32] <botifico>	 [02puppet] 07Universal-Omega closed pull request 03#3770: pywikibot: fix puppet lint - 13https://github.com/miraheze/puppet/pull/3770
[00:29:35] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1d614f344073...a61897d5177c
[00:29:37] <botifico>	 [02miraheze/puppet] 07Universal-Omega 03a61897d - pywikibot: fix puppet lint (#3770)
[00:29:38] <botifico>	 [02miraheze/puppet] 07Universal-Omega deleted branch 03fix-puppetlint
[00:29:41] <botifico>	 [02puppet] 07Universal-Omega deleted branch 03fix-puppetlint - 13https://github.com/miraheze/puppet
[00:49:28] <icinga-miraheze>	 PROBLEM - forecasting.wiki - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[00:49:52] <icinga-miraheze>	 PROBLEM - minescape.wiki - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[00:51:45] <icinga-miraheze>	 PROBLEM - lgbtqia.wiki - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[00:51:57] <icinga-miraheze>	 PROBLEM - swests.org - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[00:53:22] <icinga-miraheze>	 PROBLEM - thepolicyhub.org.uk - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[00:53:38] <icinga-miraheze>	 PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[00:57:44] <icinga-miraheze>	 PROBLEM - polcompball.wiki - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[00:59:07] <icinga-miraheze>	 PROBLEM - celeste.ink - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:00:49] <icinga-miraheze>	 PROBLEM - largedu.eu.org - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:01:04] <icinga-miraheze>	 PROBLEM - polandballwiki.com - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:01:09] <icinga-miraheze>	 PROBLEM - revi.wiki - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:01:10] <icinga-miraheze>	 PROBLEM - equestripedia.org - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:01:30] <icinga-miraheze>	 PROBLEM - lpfteam.nl - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:02:41] <icinga-miraheze>	 PROBLEM - replication-ops.com - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:04:03] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[01:04:04] <icinga-miraheze>	 PROBLEM - lophocmatngu.wiki - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:05:08] <icinga-miraheze>	 PROBLEM - slymods.info - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:05:30] <icinga-miraheze>	 PROBLEM - tfuk.app - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:06:05] <icinga-miraheze>	 RECOVERY - wiki.jerryly.top - reverse DNS on sslhost is OK: SSL OK - wiki.jerryly.top reverse DNS resolves to cp37.wikitide.net - CNAME OK
[01:07:04] <icinga-miraheze>	 PROBLEM - thesimswiki.com - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:09:06] <icinga-miraheze>	 PROBLEM - reviwiki.info - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:09:12] <icinga-miraheze>	 PROBLEM - progressbar95.wiki - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:09:23] <icinga-miraheze>	 PROBLEM - chobots.wiki - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:09:58] <icinga-miraheze>	 PROBLEM - correypedia.org - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:10:06] <botifico>	 [02puppet] 07MacFan4000 opened pull request 03#3771: check_reverse_dns CNAME should not be none - 13https://github.com/miraheze/puppet/pull/3771
[01:11:08] <icinga-miraheze>	 PROBLEM - sonicpedia.org - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:11:23] <MacFan4000>	 CosmicAlpha: ^^ that should fix the current problem
[01:11:48] <miraheze-github>	 miraheze/puppet - MacFan4000 the build passed.
[01:13:11] <icinga-miraheze>	 RECOVERY - wiki.rlab.org.uk - reverse DNS on sslhost is OK: SSL OK - wiki.rlab.org.uk reverse DNS resolves to cp36.wikitide.net - CNAME OK
[01:13:20] <MacFan4000>	 I didn't think of the fact that cname could be none, which would cause re.match to error
[01:13:34] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[01:13:56] <icinga-miraheze>	 PROBLEM - robloxapi.wiki - reverse DNS on sslhost is WARNING: TypeError: expected string or bytes-like object, got 'NoneType'
[01:16:07] <icinga-miraheze>	 RECOVERY - wiki.worldx.org - reverse DNS on sslhost is OK: SSL OK - wiki.worldx.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[01:16:18] <icinga-miraheze>	 RECOVERY - wiki.cosmicdeathfungus.com - reverse DNS on sslhost is OK: SSL OK - wiki.cosmicdeathfungus.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[01:20:14] <botifico>	 [02puppet] 07Universal-Omega closed pull request 03#3771: check_reverse_dns CNAME should not be none - 13https://github.com/miraheze/puppet/pull/3771
[01:20:16] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a61897d5177c...fab833822998
[01:20:18] <botifico>	 [02miraheze/puppet] 07MacFan4000 03fab8338 - check_reverse_dns CNAME should not be none (#3771)
[01:21:48] <CosmicAlpha>	 MacFan4000: thanks
[01:21:49] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[01:25:18] <botifico>	 [02dns] 07MacFan4000 opened pull request 03#497: add aryavartpedia.online zone - 13https://github.com/miraheze/dns/pull/497
[01:35:35] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:42:34] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[01:43:33] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#497: add aryavartpedia.online zone - 13https://github.com/miraheze/dns/pull/497
[01:43:34] <botifico>	 [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/d8db0b4733c9...dc2f778f2234
[01:43:36] <botifico>	 [02miraheze/dns] 07MacFan4000 03dc2f778 - add aryavartpedia.online zone (#497)
[01:47:34] <icinga-miraheze>	 RECOVERY - forecasting.wiki - reverse DNS on sslhost is OK: SSL OK - forecasting.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[01:48:26] <icinga-miraheze>	 RECOVERY - minescape.wiki - reverse DNS on sslhost is OK: SSL OK - minescape.wiki reverse DNS resolves to cp37.wikitide.net - CNAME FLAT
[01:49:47] <icinga-miraheze>	 RECOVERY - swests.org - reverse DNS on sslhost is OK: SSL OK - swests.org reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[01:51:32] <icinga-miraheze>	 RECOVERY - lgbtqia.wiki - reverse DNS on sslhost is OK: SSL OK - lgbtqia.wiki reverse DNS resolves to cp37.wikitide.net - CNAME FLAT
[01:51:51] <icinga-miraheze>	 RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[01:53:17] <icinga-miraheze>	 RECOVERY - thepolicyhub.org.uk - reverse DNS on sslhost is OK: SSL OK - thepolicyhub.org.uk reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[01:56:58] <icinga-miraheze>	 RECOVERY - polcompball.wiki - reverse DNS on sslhost is OK: SSL OK - polcompball.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[01:57:07] <icinga-miraheze>	 RECOVERY - celeste.ink - reverse DNS on sslhost is OK: SSL OK - celeste.ink reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[01:57:12] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.51 ms
[01:58:55] <icinga-miraheze>	 RECOVERY - revi.wiki - reverse DNS on sslhost is OK: SSL OK - revi.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[01:59:09] <icinga-miraheze>	 RECOVERY - equestripedia.org - reverse DNS on sslhost is OK: SSL OK - equestripedia.org reverse DNS resolves to cp37.wikitide.net - CNAME FLAT
[01:59:10] <icinga-miraheze>	 RECOVERY - polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - polandballwiki.com reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[01:59:12] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.78 ms
[01:59:19] <icinga-miraheze>	 RECOVERY - largedu.eu.org - reverse DNS on sslhost is OK: SSL OK - largedu.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME FLAT
[02:00:35] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:00:51] <icinga-miraheze>	 RECOVERY - lpfteam.nl - reverse DNS on sslhost is OK: SSL OK - lpfteam.nl reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[02:00:52] <icinga-miraheze>	 RECOVERY - replication-ops.com - reverse DNS on sslhost is OK: SSL OK - replication-ops.com reverse DNS resolves to cp37.wikitide.net - CNAME FLAT
[02:02:13] <icinga-miraheze>	 RECOVERY - lophocmatngu.wiki - reverse DNS on sslhost is OK: SSL OK - lophocmatngu.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[02:02:51] <icinga-miraheze>	 RECOVERY - tfuk.app - reverse DNS on sslhost is OK: SSL OK - tfuk.app reverse DNS resolves to cp37.wikitide.net - CNAME FLAT
[02:04:49] <icinga-miraheze>	 RECOVERY - slymods.info - reverse DNS on sslhost is OK: SSL OK - slymods.info reverse DNS resolves to cp37.wikitide.net - CNAME FLAT
[02:04:49] <icinga-miraheze>	 RECOVERY - thesimswiki.com - reverse DNS on sslhost is OK: SSL OK - thesimswiki.com reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[02:07:16] <icinga-miraheze>	 RECOVERY - progressbar95.wiki - reverse DNS on sslhost is OK: SSL OK - progressbar95.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[02:07:55] <icinga-miraheze>	 RECOVERY - correypedia.org - reverse DNS on sslhost is OK: SSL OK - correypedia.org reverse DNS resolves to cp37.wikitide.net - CNAME FLAT
[02:08:12] <icinga-miraheze>	 RECOVERY - chobots.wiki - reverse DNS on sslhost is OK: SSL OK - chobots.wiki reverse DNS resolves to cp37.wikitide.net - CNAME FLAT
[02:08:22] <icinga-miraheze>	 RECOVERY - reviwiki.info - reverse DNS on sslhost is OK: SSL OK - reviwiki.info reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[02:09:08] <icinga-miraheze>	 RECOVERY - sonicpedia.org - reverse DNS on sslhost is OK: SSL OK - sonicpedia.org reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[02:13:02] <icinga-miraheze>	 RECOVERY - robloxapi.wiki - reverse DNS on sslhost is OK: SSL OK - robloxapi.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT
[02:18:33] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.42 ms
[02:18:53] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09989219904 secs
[02:22:33] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.43 ms
[02:30:53] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.68 ms
[02:34:54] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.00 ms
[02:41:02] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.31 ms
[02:45:42] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/e7a29bbb8aa6...ef94a5322a1c
[02:45:43] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 03ef94a53 - Bot: Add SSL cert for aryavartpedia.online
[02:51:05] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 181.66 ms
[03:04:22] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152
[03:12:24] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 170.35 ms
[03:13:32] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[03:14:24] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.49 ms
[03:17:55] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.76, 1.59, 1.22
[03:19:49] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.96, 1.39, 1.19
[03:34:46] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 169.04 ms
[03:38:49] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 176.54 ms
[04:11:11] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[04:39:51] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[04:40:40] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.27 ms
[04:44:43] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.07 ms
[05:08:32] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK
[05:10:07] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[05:23:54] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1021783948 secs
[05:30:28] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 193.10 ms
[05:32:29] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.36 ms
[05:40:42] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 166.59 ms
[05:43:54] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09690400958 secs
[05:44:45] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.86 ms
[06:08:04] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[06:09:03] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[06:09:12] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.87, 1.44, 1.15
[06:11:12] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.16, 1.29, 1.12
[06:26:15] <botifico>	 [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/dc2f778f2234...1cd5a4b37815
[06:26:18] <botifico>	 [02miraheze/dns] 07Universal-Omega 031cd5a4b - Add dccomicswiki.com zone
[06:35:36] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.17 ms
[06:37:02] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[06:37:37] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.54 ms
[06:54:59] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1019896269 secs
[07:00:59] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09810566902 secs
[07:05:00] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1050067246 secs
[07:13:02] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09589415789 secs
[07:17:02] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.106878221 secs
[07:26:33] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.77 ms
[07:30:36] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.36 ms
[07:34:45] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 172.56 ms
[07:35:04] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[07:37:13] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[07:40:50] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.85 ms
[07:41:03] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.07921862602 secs
[07:45:03] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1032519042 secs
[07:54:05] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.65 ms
[07:58:11] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.12 ms
[07:59:03] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09954738617 secs
[08:03:44] <icinga-miraheze>	 PROBLEM - crustypedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'crustypedia.org' expires in 15 day(s) (Sat 30 Mar 2024 07:56:17 AM GMT +0000).
[08:03:56] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ef94a5322a1c...a0ef620e34dc
[08:03:57] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 03a0ef620 - Bot: Update SSL cert for crustypedia.org
[08:14:29] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.30 ms
[08:20:40] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59%
[08:22:36] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 278.13 ms
[08:26:43] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 177.92 ms
[08:31:33] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[08:33:20] <icinga-miraheze>	 RECOVERY - crustypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'crustypedia.org' will expire on Wed 12 Jun 2024 07:03:50 AM GMT +0000.
[08:34:51] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.97 ms
[08:50:40] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[09:05:13] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[09:12:40] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59%
[09:14:40] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[09:15:54] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.49, 1.78, 1.27
[09:17:48] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.06, 1.45, 1.21
[09:19:37] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 168.11 ms
[09:21:38] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.57 ms
[09:24:40] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59%
[09:58:12] <icinga-miraheze>	 PROBLEM - db171 Backups SQL on db171 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864187 seconds old and 109503 bytes
[10:02:09] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[10:02:36] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK
[10:04:30] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 100%
[10:08:35] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.46 ms
[10:09:11] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1014191806 secs
[10:17:12] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09860566258 secs
[10:21:47] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.54 ms
[10:23:48] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.07 ms
[10:54:40] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[10:56:40] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59%
[11:13:44] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 169.17 ms
[11:15:45] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.32 ms
[11:19:25] <icinga-miraheze>	 PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72]
[11:47:25] <icinga-miraheze>	 RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[11:50:20] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.73 ms
[11:54:23] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.00 ms
[11:59:03] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[12:01:13] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1036457121 secs
[12:01:33] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.73 ms
[12:03:34] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.15 ms
[12:13:11] <botifico>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/CreateWiki/compare/dc9286a2a930...8d3a81fbfadd
[12:13:14] <botifico>	 [02miraheze/CreateWiki] 07translatewiki 038d3a81f - Localisation updates from https://translatewiki.net.
[12:13:15] <botifico>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±20] 13https://github.com/miraheze/MirahezeMagic/compare/4f06569463d5...694d869a3621
[12:13:18] <botifico>	 [02miraheze/MirahezeMagic] 07translatewiki 03694d869 - Localisation updates from https://translatewiki.net.
[12:13:21] <botifico>	 [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/fd42ad688356...38b658f3a3a6
[12:13:24] <botifico>	 [02miraheze/ManageWiki] 07translatewiki 0338b658f - Localisation updates from https://translatewiki.net.
[12:13:27] <botifico>	 [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/4f36fa7af945...e50065ed7d48
[12:13:30] <botifico>	 [02miraheze/DataDump] 07translatewiki 03e50065e - Localisation updates from https://translatewiki.net.
[12:16:47] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.26 ms
[12:17:38] <miraheze-github>	 miraheze/DataDump - translatewiki the build passed.
[12:18:24] <miraheze-github>	 miraheze/ManageWiki - translatewiki the build passed.
[12:20:51] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.79 ms
[12:21:30] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build passed.
[12:23:32] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[12:23:33] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:23:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:41] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[12:23:42] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:23:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:26:29] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[12:32:59] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:33:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:09] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[12:33:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:46:16] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.71 ms
[12:48:19] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.33 ms
[12:58:58] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[13:11:45] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.47 ms
[13:13:46] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.84 ms
[14:01:15] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[14:35:00] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.74 ms
[14:39:04] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.21 ms
[14:59:00] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:11:40] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.43 ms
[15:15:43] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.48 ms
[15:46:20] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.58 ms
[15:50:23] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.54 ms
[16:09:27] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.08456915617 secs
[16:10:42] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.09 ms
[16:13:28] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1104972363 secs
[16:22:56] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.81 ms
[16:27:01] <botifico>	 [02miraheze/dns] 07Universal-Omega created branch 03ns 13https://github.com/miraheze/dns/commit/1cd5a4b37815befd04099e15654ad1cb2a63eb3b
[16:27:02] <botifico>	 [02dns] 07Universal-Omega created branch 03ns - 13https://github.com/miraheze/dns
[16:32:16] <botifico>	 [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03ns [+0/-0/±260] 13https://github.com/miraheze/dns/compare/1cd5a4b37815...5001d9316c70
[16:32:19] <botifico>	 [02miraheze/dns] 07Universal-Omega 035001d93 - Use wikitide.net for nameservers
[16:32:33] <botifico>	 [02dns] 07Universal-Omega opened pull request 03#498: Use wikitide.net for nameservers - 13https://github.com/miraheze/dns/pull/498
[16:39:54] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.44, 1.92, 1.43
[16:48:40] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[16:51:20] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.08, 1.67, 1.65
[16:55:24] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 169.52 ms
[16:55:51] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72]
[16:57:25] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.68 ms
[17:01:33] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 167.48 ms
[17:05:37] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.66 ms
[17:12:46] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.46 ms
[17:13:24] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 19 minutes ago with 0 failures
[17:20:54] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.90 ms
[17:23:51] <icinga-miraheze>	 RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[17:27:33] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.08776134253 secs
[17:33:43] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[17:37:09] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.03 ms
[17:43:16] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.37 ms
[17:44:36] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03pdnsrec [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/fe9e654e1f42
[17:44:37] <botifico>	 [02miraheze/puppet] 07Universal-Omega 03fe9e654 - prometheus: add pdns-recursor metrics exporting
[17:44:40] <botifico>	 [02puppet] 07Universal-Omega created branch 03pdnsrec - 13https://github.com/miraheze/puppet
[17:44:43] <botifico>	 [02puppet] 07Universal-Omega opened pull request 03#3772: prometheus: add pdns-recursor metrics exporting - 13https://github.com/miraheze/puppet/pull/3772
[17:52:25] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.21 ms
[17:54:26] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.12 ms
[17:58:33] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.24 ms
[18:02:13] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03pdnsrec [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fe9e654e1f42...5d666875841b
[18:02:14] <botifico>	 [02miraheze/puppet] 07Universal-Omega 035d66687 - Add webserver config
[18:02:15] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3772: prometheus: add pdns-recursor metrics exporting - 13https://github.com/miraheze/puppet/pull/3772
[18:02:37] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.08 ms
[18:02:41] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[18:10:45] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.07 ms
[18:14:49] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.74 ms
[18:23:04] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 261.01 ms
[18:25:05] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.69 ms
[18:27:19] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.41, 1.83, 1.48
[18:29:13] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.79, 1.40, 1.37
[18:31:47] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[18:43:24] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 166.85 ms
[18:53:34] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.41 ms
[19:13:22] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:29:45] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[19:34:56] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03onboard [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/bdda15eb24d0
[19:34:57] <botifico>	 [02miraheze/puppet] 07Universal-Omega 03bdda15e - Add alex to mediawiki-admins
[19:34:58] <botifico>	 [02puppet] 07Universal-Omega created branch 03onboard - 13https://github.com/miraheze/puppet
[19:35:07] <botifico>	 [02puppet] 07Universal-Omega opened pull request 03#3773: Add alex to mediawiki-admins and icinga - 13https://github.com/miraheze/puppet/pull/3773
[19:37:15] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03onboard [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bdda15eb24d0...ea126914a018
[19:37:18] <botifico>	 [02miraheze/puppet] 07Universal-Omega 03ea12691 - Add alex to icinga
[19:37:21] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3773: Add alex to mediawiki-admins and icinga - 13https://github.com/miraheze/puppet/pull/3773
[19:37:32] <botifico>	 [02puppet] 07Universal-Omega edited pull request 03#3773: T11957: add alex to mediawiki-admins and icinga - 13https://github.com/miraheze/puppet/pull/3773
[19:45:11] <botifico>	 [02puppet] 07Universal-Omega closed pull request 03#3773: T11957: add alex to mediawiki-admins and icinga - 13https://github.com/miraheze/puppet/pull/3773
[19:45:13] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/fab833822998...7e62a3617068
[19:45:14] <botifico>	 [02miraheze/puppet] 07Universal-Omega 037e62a36 - T11957: add alex to mediawiki-admins and icinga (#3773)
[19:45:15] <botifico>	 [02miraheze/puppet] 07Universal-Omega deleted branch 03onboard
[19:45:17] <botifico>	 [02puppet] 07Universal-Omega deleted branch 03onboard - 13https://github.com/miraheze/puppet
[19:50:26] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.95 ms
[19:54:30] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.40 ms
[20:24:18] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6a4c1d42d5c4...d4727d21e19f
[20:24:20] <botifico>	 [02miraheze/mw-config] 07Universal-Omega 03d4727d2 - Add Alex (Miraheze) to staffwiki and srewiki
[20:25:10] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:25:20] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d4727d21e19f...ea9f40da7aaa
[20:25:22] <botifico>	 [02miraheze/mw-config] 07Universal-Omega 03ea9f40d - Remove wgMirahezeStaffAccessIds
[20:26:14] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:32:01] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[20:32:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:32:11] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s
[20:32:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:35:11] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1035562456 secs
[20:37:24] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[20:37:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:37:33] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s
[20:37:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:41:09] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.97, 1.76, 1.49
[20:43:09] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.50, 1.69, 1.51
[20:50:26] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.47 ms
[20:52:27] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.93 ms
[20:53:30] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[20:53:31] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[20:53:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:53:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:10:58] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09997791052 secs
[21:31:26] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.75 ms
[21:37:34] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.50 ms
[22:06:22] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 168.72 ms
[22:12:29] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.59 ms
[22:31:44] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a0ef620e34dc...5010c8d14ab4
[22:31:47] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 035010c8d - Bot: Add SSL cert for dccomicswiki.com
[22:34:54] <botifico>	 [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/d11f511eb581...0e4ce697cd7a
[22:34:56] <botifico>	 [02miraheze/landing] 07Universal-Omega 030e4ce69 - Use full path
[22:35:45] <miraheze-github>	 miraheze/landing - Universal-Omega the build passed.
[22:39:12] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7e62a3617068...f1bfb764f92b
[22:39:15] <botifico>	 [02miraheze/puppet] 07Universal-Omega 03f1bfb76 - csp: add miraheze.org to style-src and img-src
[22:41:54] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.02, 2.10, 1.63
[22:43:51] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.07, 1.74, 1.56
[22:45:48] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.10, 1.52, 1.49
[22:49:06] <botifico>	 [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5010c8d14ab4...00f87107ac09
[22:49:07] <botifico>	 [02miraheze/ssl] 07Universal-Omega 0300f8710 - Add www.dccomicswiki.com redirect
[22:58:30] <icinga-miraheze>	 PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9713MiB (10% inode=98%);
[23:02:14] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1006380022 secs
[23:06:05] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09955716133 secs
[23:06:33] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9702MiB (10% inode=98%);
[23:09:58] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1007002294 secs
[23:10:32] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.34, 1.88, 1.52
[23:12:25] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.54, 1.73, 1.50
[23:13:50] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.0999353528 secs
[23:14:19] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.31, 1.65, 1.50
[23:17:02] <icinga-miraheze>	 PROBLEM - cp41 NTP time on cp41 is WARNING: NTP WARNING: Offset 0.1013971269 secs
[23:22:06] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.98, 1.72, 1.55
[23:24:05] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.44, 1.58, 1.51
[23:24:59] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'extension_list': True, 'versions': '1.41'} to all
[23:25:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:25:09] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'extension_list': True, 'versions': '1.41'} to all - SUCCESS in 10s
[23:25:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:26:24] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[23:37:49] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'world': True, 'extension_list': True, 'versions': '1.41'} to all
[23:37:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:39:07] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'world': True, 'extension_list': True, 'versions': '1.41'} to all - SUCCESS in 77s
[23:39:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:40:59] <icinga-miraheze>	 PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.27, 1.71, 1.61
[23:42:58] <icinga-miraheze>	 RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.04, 1.53, 1.56
[23:45:00] <botifico>	 [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/1292f595e859...5d7390219b5d
[23:45:03] <botifico>	 [02miraheze/mediawiki-repos] 07Universal-Omega 035d73902 - Use _branch_ for UnlinkedWikibase
[23:47:46] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'world': True, 'extension_list': True, 'versions': '1.41'} to all
[23:47:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:48:46] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'world': True, 'extension_list': True, 'versions': '1.41'} to all - SUCCESS in 59s
[23:48:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:49:00] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'lang': 'en', 'versions': '1.41'} to all
[23:49:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:49:11] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'lang': 'en', 'versions': '1.41'} to all - SUCCESS in 11s
[23:49:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:49:52] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41'} to all
[23:49:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:55:13] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[23:56:41] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41'} to all - SUCCESS in 409s
[23:56:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:57:07] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 UnlinkedWikibase]
[23:58:10] <icinga-miraheze>	 RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset 0.09963127971 secs