[00:15:17] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[00:40:44] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[00:46:16] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[01:09:35] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query.
[01:44:09] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[02:21:54] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[03:10:56] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[03:52:05] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.18 ms
[03:54:06] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.17 ms
[03:59:32] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[04:03:43] <MirahezeRelay>	 <honglan233> https://issue-tracker.miraheze.org/T11985
[04:04:28] <MirahezeRelay>	 <honglan233> This problem has not been solved at present.
[04:04:31] <MirahezeRelay>	 <honglan233> [800bce074fa73a9d4bf0c540] Caught exception of type Flow\Exception\InvalidActionException
[04:19:54] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000.
[04:23:34] <MirahezeRelay>	 <elliethepwincess> ^ @Site Reliability Engineers courtesy ping, seems to be unresolved since late March
[04:29:22] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[04:37:12] <MirahezeRelay>	 <solodayheavenofficial, replying to elliethepwincess> yeah we have a lot of these
[04:37:42] <MirahezeRelay>	 <solodayheavenofficial> @Site Reliability Engineers per https://issue-tracker.miraheze.org/T11928, please force a logout of all sessions for the user account `Fersteax_Pasique`
[04:37:47] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[04:37:56] <MirahezeRelay>	 <solodayheavenofficial> wait sorry shouldn't have done the 2x ping
[04:39:11] <MirahezeRelay>	 <solodayheavenofficial, replying to honglan233> [1/2] I really cannot tell what the issue is
[04:39:11] <MirahezeRelay>	 <solodayheavenofficial, replying to honglan233> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1235088103073382460/image.png?ex=663318ee&is=6631c76e&hm=158570224e9b71307c67ae9252833a6d78aa06c5ed30ac5a883a17e06cd13546&
[04:39:19] <MirahezeRelay>	 <solodayheavenofficial> Appears to work fine for me
[04:42:00] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[04:57:49] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.82 ms
[05:01:50] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.87 ms
[05:10:52] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query.
[05:33:44] <MirahezeRelay>	 <reception123> !log sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/invalidateUserSessions.php --wiki=telepediawiki --user "Fersteax Pasique"
[05:33:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:35:41] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.407 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[05:53:33] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.58 ms
[05:55:33] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.03 ms
[06:03:15] <icinga-miraheze>	 PROBLEM - puppet181 Backups SSLKeys on puppet181 is WARNING: FILE_AGE WARNING: /var/log/sslkeys-backup.log is 864182 seconds old and 93 bytes
[06:04:36] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[06:07:43] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.76 ms
[06:09:43] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.73 ms
[06:18:31] <icinga-miraheze>	 PROBLEM - swiftac171 Backups Swift Account Container on swiftac171 is WARNING: FILE_AGE WARNING: /var/log/swift-account-container-backup.log is 864135 seconds old and 108 bytes
[07:02:13] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 151.71 ms
[07:04:29] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[07:04:42] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.85 ms
[07:06:11] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 149.77 ms
[07:08:44] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.73 ms
[07:24:13] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 156.49 ms
[07:30:11] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 145.05 ms
[08:02:20] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[08:08:08] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 158.56 ms
[08:10:10] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 139.88 ms
[08:20:38] <MirahezeRelay>	 <honglan233, replying to solodayheavenofficial> It's not a display problem, it's a problem in the process of deleting the topic.
[08:30:46] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 140.53 ms
[08:38:59] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 139.66 ms
[09:00:36] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 171.00 ms
[09:02:36] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.90 ms
[09:26:53] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.08 ms
[09:29:33] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[09:30:52] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 151.81 ms
[09:59:22] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK
[10:19:06] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.00 ms
[10:21:06] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.17 ms
[10:30:20] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.12 ms
[10:32:20] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.59 ms
[10:34:27] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 140.14 ms
[10:39:36] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 153.25 ms
[10:40:37] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 140.61 ms
[10:41:35] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 154.69 ms
[10:45:48] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.22 ms
[10:48:33] <icinga-miraheze>	 PROBLEM - hastursnotebook.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hastursnotebook.org' expires in 15 day(s) (Fri 17 May 2024 10:34:32 AM GMT +0000).
[10:48:45] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a8faf1494dd5...61f249b84957
[10:48:46] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0361f249b - Bot: Update SSL cert for hastursnotebook.org
[10:49:50] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.96 ms
[11:17:25] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:29:47] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[11:34:05] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 139.98 ms
[11:36:07] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 140.37 ms
[11:37:25] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:46:24] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.73 ms
[11:46:33] <icinga-miraheze>	 RECOVERY - hastursnotebook.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hastursnotebook.org' will expire on Tue 30 Jul 2024 09:48:39 AM GMT +0000.
[11:48:24] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.42 ms
[11:58:40] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[12:38:19] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.77 ms
[12:40:19] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.89 ms
[13:04:50] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.43 ms
[13:06:50] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.84 ms
[13:08:36] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.08 ms
[13:10:35] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 150.76 ms
[13:14:41] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.56 ms
[13:16:40] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 157.18 ms
[13:19:23] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 139.87 ms
[13:20:48] <icinga-miraheze>	 PROBLEM - fallofsanctuary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fallofsanctuary.com' expires in 15 day(s) (Fri 17 May 2024 12:53:07 PM GMT +0000).
[13:21:00] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/61f249b84957...57ece6257073
[13:21:03] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0357ece62 - Bot: Update SSL cert for fallofsanctuary.com
[13:21:25] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 140.59 ms
[13:21:26] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=aitgwiki --summary=Impoted from aitg.fandom.com --search-recursively /home/reception/aitg (START)
[13:21:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:24:38] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=aitgwiki --summary=Impoted from aitg.fandom.com --search-recursively /home/reception/aitg (END - exit=0)
[13:24:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:32:35] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[13:37:39] <icinga-miraheze>	 PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[13:49:42] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 155.32 ms
[13:50:06] <icinga-miraheze>	 RECOVERY - fallofsanctuary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fallofsanctuary.com' will expire on Tue 30 Jul 2024 12:20:54 PM GMT +0000.
[13:51:40] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 154.42 ms
[14:02:33] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:05:39] <icinga-miraheze>	 RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[14:10:39] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 161.94 ms
[14:12:38] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 164.95 ms
[14:25:18] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[14:34:30] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 71%
[14:35:05] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 180.04 ms
[14:36:20] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.57 ms
[14:38:20] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.90 ms
[14:44:48] <icinga-miraheze>	 PROBLEM - files.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'files.petrawiki.org' expires in 15 day(s) (Fri 17 May 2024 02:40:58 PM GMT +0000).
[14:44:58] <icinga-miraheze>	 PROBLEM - hr.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hr.petrawiki.org' expires in 15 day(s) (Fri 17 May 2024 02:27:45 PM GMT +0000).
[14:45:00] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/57ece6257073...82b7f8708f3c
[14:45:03] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0382b7f87 - Bot: Update SSL cert for files.petrawiki.org
[14:45:11] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/82b7f8708f3c...0ba7dbd1b9fe
[14:45:12] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 030ba7dbd - Bot: Update SSL cert for hr.petrawiki.org
[14:49:27] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 138.45 ms
[14:56:29] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 51%
[14:59:44] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.36 ms
[15:00:29] <icinga-miraheze>	 RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 29%
[15:01:44] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.98 ms
[15:09:39] <icinga-miraheze>	 PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[15:10:01] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 155.11 ms
[15:12:04] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 138.87 ms
[15:14:26] <icinga-miraheze>	 RECOVERY - hr.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hr.petrawiki.org' will expire on Tue 30 Jul 2024 01:45:05 PM GMT +0000.
[15:14:42] <icinga-miraheze>	 RECOVERY - files.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'files.petrawiki.org' will expire on Tue 30 Jul 2024 01:44:54 PM GMT +0000.
[15:19:57] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.21 ms
[15:21:57] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.88 ms
[15:23:10] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[15:28:04] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 159.22 ms
[15:34:02] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 161.24 ms
[15:35:39] <icinga-miraheze>	 RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[15:49:54] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 235.31 ms
[15:52:03] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:54:03] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 138.13 ms
[16:15:15] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 177.81 ms
[16:23:14] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03remove-service_worker [+0/-0/±1] 13https://github.com/miraheze/MediaWikiDebugJS/commit/0440547c8a21
[16:23:15] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega 030440547 - Remove unnecessary service_worker
[16:23:18] <botifico>	 [02MediaWikiDebugJS] 07Universal-Omega created branch 03remove-service_worker - 13https://github.com/miraheze/MediaWikiDebugJS
[16:23:19] <botifico>	 [02MediaWikiDebugJS] 07Universal-Omega opened pull request 03#14: Remove unnecessary service_worker - 13https://github.com/miraheze/MediaWikiDebugJS/pull/14
[16:23:42] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03remove-service_worker [+0/-1/±0] 13https://github.com/miraheze/MediaWikiDebugJS/compare/0440547c8a21...4c954a82e9a6
[16:23:43] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega 034c954a8 - Remove
[16:23:46] <botifico>	 [02MediaWikiDebugJS] 07Universal-Omega synchronize pull request 03#14: Remove unnecessary service_worker - 13https://github.com/miraheze/MediaWikiDebugJS/pull/14
[16:25:56] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MediaWikiDebugJS/compare/2ef1261cd9a0...f4129e427d45
[16:25:58] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega 03f4129e4 - Remove unused static.wikitide.net
[16:27:18] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 178.05 ms
[16:29:41] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.88 ms
[16:31:41] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.40 ms
[16:37:49] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MediaWikiDebugJS/compare/f4129e427d45...65894488e5a4
[16:37:51] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega 036589448 - Replace trailing dot from X-Served-By and rename constant
[16:38:34] <botifico>	 [02MediaWikiDebugJS] 07Universal-Omega closed pull request 03#14: Remove unnecessary service_worker - 13https://github.com/miraheze/MediaWikiDebugJS/pull/14
[16:38:37] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega deleted branch 03remove-service_worker
[16:38:38] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/MediaWikiDebugJS/compare/65894488e5a4...29e47df45533
[16:38:41] <botifico>	 [02miraheze/MediaWikiDebugJS] 07Universal-Omega 0329e47df - Remove unnecessary service_worker
[16:38:42] <botifico>	 [02MediaWikiDebugJS] 07Universal-Omega deleted branch 03remove-service_worker - 13https://github.com/miraheze/MediaWikiDebugJS
[16:49:41] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.07 ms
[16:53:40] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 177.27 ms
[17:06:54] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 178.08 ms
[17:08:53] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 177.20 ms
[17:09:39] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 140.07 ms
[17:13:46] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 139.84 ms
[17:24:04] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.10 ms
[17:36:36] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.92 ms
[17:38:18] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 139.32 ms
[17:42:23] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 138.22 ms
[17:42:38] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.79 ms
[17:46:22] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 181.03 ms
[17:47:25] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:50:33] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 180.01 ms
[17:52:32] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 176.88 ms
[18:02:45] <botifico>	 [02CreateWiki] 07redbluegreenhat closed pull request 03#510: Add a config variable to disable the REST API - 13https://github.com/miraheze/CreateWiki/pull/510
[18:02:47] <botifico>	 [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/CreateWiki/compare/3f4004443a0f...b21aa80d873f
[18:02:50] <botifico>	 [02miraheze/CreateWiki] 07redbluegreenhat 03b21aa80 - Add a config variable to disable the REST API (#510)
[18:03:09] <botifico>	 [02mw-config] 07redbluegreenhat closed pull request 03#5549: Add wgCreateWikiDisableRESTAPI - 13https://github.com/miraheze/mw-config/pull/5549
[18:03:11] <botifico>	 [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ad768c379757...1798e53901a2
[18:03:13] <botifico>	 [02miraheze/mw-config] 07redbluegreenhat 031798e53 - Add wgCreateWikiDisableRESTAPI (#5549)
[18:03:46] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 177.64 ms
[18:03:49] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[18:03:50] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s
[18:03:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:03:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:04:03] <miraheze-github>	 miraheze/mw-config - redbluegreenhat the build passed.
[18:04:20] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151
[18:04:22] <MirahezeLSBot>	 !log [alex@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[18:04:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:04:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:04:36] <MirahezeRelay>	 <bluemoon0332> 😭
[18:06:05] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151
[18:06:06] <MirahezeLSBot>	 !log [alex@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[18:06:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:06:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:07:00] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[18:07:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:07:09] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s
[18:07:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:08:55] <icinga-miraheze>	 PROBLEM - reports171 NTP time on reports171 is CRITICAL: NTP CRITICAL: Offset 4.323756129 secs
[18:11:15] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[18:11:45] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 176.77 ms
[18:15:02] <MirahezeRelay>	 <bluemoon0332> looks like someone deployed Pix's PR to test
[18:15:15] <MirahezeRelay>	 <bluemoon0332> that seems to be what's messing things up
[18:18:58] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 180.05 ms
[18:19:00] <MirahezeRelay>	 <pixldev, replying to bluemoon0332> https://tenor.com/view/clown-emoji-clown-pensive-gif-18275597
[18:19:08] <MirahezeRelay>	 <pixldev> I am the bearer of the cutest
[18:19:21] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 75%
[18:20:57] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 173.54 ms
[18:23:29] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151
[18:23:30] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s
[18:23:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:23:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:23:39] <MirahezeRelay>	 <bluemoon0332> fixed
[18:25:11] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'l10n': True, 'versions': '1.42'} to test151
[18:25:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:26:04] <MirahezeRelay>	 <pixldev, replying to bluemoon0332> What part broke it
[18:26:43] <MirahezeRelay>	 <bluemoon0332> the current commit not being from the https://github.com/miraheze/CreateWiki.git remote
[18:26:56] <MirahezeRelay>	 <bluemoon0332> probably
[18:29:23] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.24 ms
[18:29:58] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'l10n': True, 'versions': '1.42'} to test151 - SUCCESS in 286s
[18:30:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:31:25] <icinga-miraheze>	 PROBLEM - agesofconflict.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'agesofconflict.wiki' expires in 15 day(s) (Fri 17 May 2024 06:26:23 PM GMT +0000).
[18:31:25] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 138.47 ms
[18:31:36] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0ba7dbd1b9fe...cb9295779eb4
[18:31:37] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03cb92957 - Bot: Update SSL cert for agesofconflict.wiki
[18:32:22] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[18:32:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:32:32] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s
[18:32:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:39:21] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 58%
[18:43:21] <icinga-miraheze>	 RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 37%
[18:44:09] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.96 ms
[18:46:39] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.95 ms
[18:50:41] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.49 ms
[18:51:42] <MirahezeRelay>	 <pixldev, replying to bluemoon0332> So not my ass code. Good to know!
[19:00:09] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 181.14 ms
[19:01:13] <icinga-miraheze>	 RECOVERY - agesofconflict.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'agesofconflict.wiki' will expire on Tue 30 Jul 2024 05:31:30 PM GMT +0000.
[19:15:32] <icinga-miraheze>	 RECOVERY - reports171 NTP time on reports171 is OK: NTP OK: Offset 0.004968553782 secs
[19:18:04] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.45 ms
[19:20:06] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.95 ms
[19:21:14] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.63 ms
[19:25:13] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 163.04 ms
[20:00:47] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.76 ms
[20:02:25] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:04:48] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.62 ms
[20:14:03] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/commit/28437b3e7643
[20:14:06] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega 0328437b3 - Migrate to the usage of central ids
[20:14:07] <botifico>	 [02GlobalNewFiles] 07Universal-Omega created branch 03centralids - 13https://github.com/miraheze/GlobalNewFiles
[20:14:08] <botifico>	 [02GlobalNewFiles] 07Universal-Omega opened pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56
[20:14:19] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 148.57 ms
[20:16:17] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 151.60 ms
[20:18:46] <miraheze-github>	 miraheze/GlobalNewFiles - Universal-Omega the build passed.
[20:21:23] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/28437b3e7643...822dd61f9eab
[20:21:24] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega 03822dd61 - Update pager to use central ids
[20:21:27] <botifico>	 [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56
[20:22:02] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[20:24:10] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/822dd61f9eab...d74209696ec3
[20:24:12] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega 03d742096 - Add files_uploader
[20:24:15] <botifico>	 [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56
[20:24:37] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/d74209696ec3...ad00c1aabb8c
[20:24:38] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega 03ad00c1a - Move
[20:24:41] <botifico>	 [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56
[20:29:02] <miraheze-github>	 miraheze/GlobalNewFiles - Universal-Omega the build passed.
[20:31:01] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+1/-0/±0] 13https://github.com/miraheze/GlobalNewFiles/compare/ad00c1aabb8c...4ebc6788ed81
[20:31:04] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega 034ebc678 - Add patch
[20:31:07] <botifico>	 [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56
[20:31:53] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+1/-1/±0] 13https://github.com/miraheze/GlobalNewFiles/compare/4ebc6788ed81...f1674e9947fc
[20:31:56] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega 03f1674e9 - Rename
[20:31:59] <botifico>	 [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56
[20:36:21] <miraheze-github>	 miraheze/GlobalNewFiles - Universal-Omega the build passed.
[20:45:05] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03centralids [+1/-0/±0] 13https://github.com/miraheze/GlobalNewFiles/compare/f1674e9947fc...20dcf8d20f64
[20:45:08] <botifico>	 [02miraheze/GlobalNewFiles] 07Universal-Omega 0320dcf8d - Add migration script
[20:45:11] <botifico>	 [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#56: Migrate to the usage of central ids - 13https://github.com/miraheze/GlobalNewFiles/pull/56
[20:48:00] <MirahezeRelay>	 <cosmicalpha> @bluemoon0332 do you want to take a look at https://github.com/miraheze/GlobalNewFiles/pull/56 if you have time?
[20:49:41] <miraheze-github>	 miraheze/GlobalNewFiles - Universal-Omega the build passed.
[20:50:21] <MirahezeRelay>	 <cosmicalpha> Took only 30 minutes to do so I may have messed up somewhere lol
[20:50:57] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[20:58:02] <MirahezeRelay>	 <cosmicalpha> Oh yeah I see two problems lol
[21:27:20] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:41:08] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 166.91 ms
[21:45:10] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.04 ms
[21:49:53] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[22:21:59] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[22:27:20] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:37:27] <MirahezeRelay>	 <cosmicalpha> !log added raidarr to stewards@ email
[22:37:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:50:52] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[23:19:23] <botifico>	 [02GlobalNewFiles] 07redbluegreenhat reviewed pull request 03#56 commit - 13https://github.com/miraheze/GlobalNewFiles/pull/56#discussion_r1586914609
[23:19:33] <MirahezeRelay>	 <bluemoon0332> @cosmicalpha reviewed
[23:19:53] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[23:25:57] <MirahezeRelay>	 <bluemoon0332> I like the PR btw
[23:26:55] <MirahezeRelay>	 <pixldev> among the best inventions of man
[23:27:42] <MirahezeRelay>	 <cosmicalpha, replying to bluemoon0332> Oh good catch thanks!
[23:28:17] <MirahezeRelay>	 <cosmicalpha> files_user will be dropped after initial migration but in the meantine yeah we need to handle it.
[23:30:58] <MirahezeRelay>	 <cosmicalpha> Other issue is uploader also needs to be nullable for initial so it can be populated without breaking things I believe