[01:06:43] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.68 ms [01:08:46] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.22 ms [01:17:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:39:13] PROBLEM - dragdown.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'dragdown.wiki' expires in 7 day(s) (Thu 16 May 2024 01:14:47 AM GMT +0000). [01:56:26] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [02:39:07] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query. [02:47:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:09:09] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [03:26:27] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [04:34:01] !log [universalomega@mwtask181] starting deploy of {'versions': ['1.41', '1.42'], 'upgrade_extensions': 'PortableInfobox'} to all [04:34:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:34:20] !log [universalomega@mwtask181] finished deploy of {'versions': ['1.41', '1.42'], 'upgrade_extensions': 'PortableInfobox'} to all - SUCCESS in 19s [04:34:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:36:10] Thanks for the wiki ping @.labster ^ [04:38:02] <.labster> Welcome? [04:39:08] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query. [04:39:11] ? [04:39:47] <.labster> People usually don’t thank you for pings so I thought 40% chance of sarcasm [04:40:11] <.labster> Also you might want to push a new version tag in that repo [04:46:23] oh no, I like pings when they are important lol no sarcasm [04:49:17] [1/2] I don't use version tags on that repo just branches the version tags from 5 years ago are just there from when I took over maintenance of the extension. I csn bumb version but don't usually tag it. The only one I do that for is DPL3 as my version of DPL3 is used a lot more and widespread, including on Fandom, Wiki.gg, etc... so I just decided to maintain rel [04:49:17] [2/2] eases for it. [05:09:09] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.401 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [05:22:11] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=runwiki --username-prefix=fandom:run /home/reception/run_pages_full.xml --report 1 (END - exit=0) [05:22:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:26:26] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [05:45:42] !log [universalomega@mwtask181] starting deploy of {'versions': ['1.41', '1.42'], 'upgrade_extensions': 'PortableInfobox'} to all [05:45:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:46:03] !log [universalomega@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [05:46:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:02] !log [universalomega@mwtask181] starting deploy of {'force': True, 'force_upgrade': True, 'versions': ['1.41', '1.42'], 'upgrade_extensions': 'PortableInfobox'} to all [05:47:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:41] !log [universalomega@mwtask181] finished deploy of {'force': True, 'force_upgrade': True, 'versions': ['1.41', '1.42'], 'upgrade_extensions': 'PortableInfobox'} to all - SUCCESS in 38s [05:47:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:56:27] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [06:07:14] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:36:16] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [06:51:32] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.83 ms [06:53:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.85 ms [06:56:26] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [07:07:31] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:09:08] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query. [07:26:28] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [07:33:13] [02mw-config] 07AverageHelper opened pull request 03#5555: Add `wgPortableInfoboxUseFileDescriptionPage` - 13https://github.com/miraheze/mw-config/pull/5555 [07:39:34] miraheze/mw-config - AverageHelper the build passed. [07:40:00] [02mw-config] 07Universal-Omega closed pull request 03#5555: Add `wgPortableInfoboxUseFileDescriptionPage` - 13https://github.com/miraheze/mw-config/pull/5555 [07:40:02] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/e057f87d6f05...acc5400531bd [07:40:03] [02mw-config] 07AverageHelper 03acc5400 - Add `wgPortableInfoboxUseFileDescriptionPage` (#5555) [07:40:29] [02mw-config] 07Universal-Omega commented on pull request 03#5555: Add `wgPortableInfoboxUseFileDescriptionPage` - 13https://github.com/miraheze/mw-config/pull/5555#issuecomment-2099947610 [07:40:54] miraheze/mw-config - Universal-Omega the build passed. [07:53:49] !log [@test151] starting deploy of {'config': True} to test151 [07:53:51] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [07:53:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:53:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:26] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [08:02:06] !log [@mwtask181] starting deploy of {'config': True} to all [08:02:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:02:16] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [08:02:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:38] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [08:07:34] !log [@mwtask171] starting deploy of {'config': True} to all [08:07:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:07:43] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [08:07:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:09:09] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [08:39:05] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query. [08:56:27] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [09:26:26] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [09:50:43] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.51 ms [09:52:46] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.96 ms [10:34:05] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/4a3e83cadafc...a1c17e98551c [10:34:09] [02ssl] 07WikiTideSSLBot 03a1c17e9 - Bot: Add SSL cert for wiki.secondrenaissance.net [11:04:11] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:28:33] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.95 ms [11:30:33] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.35 ms [11:32:19] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:26:17] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [12:37:53] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [12:56:16] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [13:26:17] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [13:56:16] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [14:36:23] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query. [14:56:17] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [15:06:01] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [15:27:02] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 51% [15:29:03] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 29% [15:37:10] !log (metawiki) DELETE FROM echo_unread_wikis WHERE euw_user=6758; [15:37:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:37:17] PROBLEM - puritwiki.p-e.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'puritwiki.p-e.kr' expires in 7 day(s) (Thu 16 May 2024 03:15:22 PM GMT +0000). [15:42:28] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a1c17e98551c...3cd4d79e7d2f [15:42:30] [02ssl] 07WikiTideSSLBot 033cd4d79 - Bot: Add SSL cert for wiki.cowgl.xyz [15:43:08] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3cd4d79e7d2f...94324416d99d [15:43:11] [02ssl] 07WikiTideSSLBot 039432441 - Bot: Update SSL cert for puritwiki.p-e.kr [15:44:19] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/94324416d99d...208c167de7ae [15:44:21] [02ssl] 07WikiTideSSLBot 03208c167 - Bot: Update SSL cert for dragdown.wiki [15:44:46] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/208c167de7ae...18f65b49ee64 [15:44:49] [02ssl] 07WikiTideSSLBot 0318f65b4 - Bot: Update SSL cert for wiki.jeremyboyd.org [15:45:09] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/18f65b49ee64...944d16cfd2ea [15:45:11] [02ssl] 07WikiTideSSLBot 03944d16c - Bot: Update SSL cert for with.cpt-ra.bid [15:45:33] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/944d16cfd2ea...cd03aed4279f [15:45:35] [02ssl] 07WikiTideSSLBot 03cd03aed - Bot: Update SSL cert for wiki.fbpml.org [15:45:49] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cd03aed4279f...8bd54efcebc3 [15:45:52] [02ssl] 07WikiTideSSLBot 038bd54ef - Bot: Update SSL cert for wiki.elgeis.com [15:51:50] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8bd54efcebc3...f4fe51aa9c2d [15:51:51] [02ssl] 07WikiTideSSLBot 03f4fe51a - Bot: Update SSL cert for chobots.wiki [15:52:53] PROBLEM - slymods.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'slymods.info' expires in 7 day(s) (Thu 16 May 2024 03:42:31 PM GMT +0000). [15:52:54] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f4fe51aa9c2d...5354b89f1f79 [15:52:56] [02ssl] 07WikiTideSSLBot 035354b89 - Bot: Update SSL cert for wiki.mineplay.nl [15:53:24] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5354b89f1f79...20e5e0016c1a [15:53:25] [02ssl] 07WikiTideSSLBot 0320e5e00 - Bot: Update SSL cert for wiki.drycleaninglobby.net [15:53:41] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/20e5e0016c1a...eb413b461694 [15:53:42] [02ssl] 07WikiTideSSLBot 03eb413b4 - Bot: Update SSL cert for en.pornwiki.org [15:55:03] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/eb413b461694...c004a9b4168d [15:55:06] [02ssl] 07WikiTideSSLBot 03c004a9b - Bot: Update SSL cert for lab.wikimimo.com [16:00:34] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 150.15 ms [16:04:39] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 151.90 ms [16:15:40] PROBLEM - lab.wikimimo.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'lab.wikimimo.com' expires in 7 day(s) (Thu 16 May 2024 04:11:41 PM GMT +0000). [16:21:28] RECOVERY - wiki.fbpml.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fbpml.org' will expire on Tue 06 Aug 2024 02:45:27 PM GMT +0000. [16:26:47] RECOVERY - dragdown.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'dragdown.wiki' will expire on Tue 06 Aug 2024 02:44:12 PM GMT +0000. [16:29:41] RECOVERY - with.cpt-ra.bid - LetsEncrypt on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Tue 06 Aug 2024 02:45:02 PM GMT +0000. [16:33:20] RECOVERY - www.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Tue 06 Aug 2024 02:53:35 PM GMT +0000. [16:37:10] RECOVERY - wiki.jeremyboyd.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jeremyboyd.org' will expire on Tue 06 Aug 2024 02:44:40 PM GMT +0000. [16:37:17] RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Tue 06 Aug 2024 02:43:02 PM GMT +0000. [16:39:59] RECOVERY - wiki.elgeis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.elgeis.com' will expire on Tue 06 Aug 2024 02:45:43 PM GMT +0000. [16:40:38] RECOVERY - chobots.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'chobots.wiki' will expire on Tue 06 Aug 2024 02:51:45 PM GMT +0000. [16:40:40] RECOVERY - en.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Tue 06 Aug 2024 02:53:35 PM GMT +0000. [16:42:34] RECOVERY - wiki.mineplay.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineplay.nl' will expire on Tue 06 Aug 2024 02:52:48 PM GMT +0000. [16:43:25] RECOVERY - wiki.drycleaninglobby.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.drycleaninglobby.net' will expire on Tue 06 Aug 2024 02:53:18 PM GMT +0000. [16:45:40] RECOVERY - lab.wikimimo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lab.wikimimo.com' will expire on Tue 06 Aug 2024 02:54:57 PM GMT +0000. [16:47:39] RECOVERY - pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Tue 06 Aug 2024 02:53:35 PM GMT +0000. [17:20:47] PROBLEM - wiki.surf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.surf' expires in 7 day(s) (Thu 16 May 2024 05:11:43 PM GMT +0000). [17:22:20] Relaybot pings from IRC are no longer case sensitive and work when pinging like `UserName,` but still only work with exact matches... [17:23:07] Cool [17:23:12] Wdym exact match [17:23:27] Since Discord usernames are always lowercase now figured why not lol [17:23:55] Like it wont match CosmicAlph, or CosmicAlpha2, so you don't ping wrong user (I hope) lol [17:24:04] Makes sense [17:24:11] Implied tbh [17:24:26] Wasn't always the case lol [17:24:31] Yea [17:24:38] CosmicAlpha hi [17:24:45] Pixldev hi [17:25:02] @cosmicalpha do you need @ in front [17:25:11] [02dns] 07MacFan4000 opened pull request 03#517: Add ACME challenges for wikitide.org - 13https://github.com/miraheze/dns/pull/517 [17:26:20] Also btw would using account create rights from sysop/crat on test beta for making test accounts on beta be fine? Since normally sre makes accounts via email. [17:26:39] I don't see why not [17:26:45] It's only disabled cause spambots [17:26:59] (and LTAs) [17:27:10] Who wants to spam beta- whatever [17:27:43] Spambots [17:28:24] Since I have crate I could make accounts if anyone wants, it’s a 15 second thing but eh whatever one spec of dust off tech’s plate [17:28:40] ¯\_(ツ)_/¯ [17:28:44] @cosmicalpha [17:28:45] You need @ in front, or : or , at end [17:28:47] Don’t really matter [17:28:51] Cool [17:29:14] Oh it turns into pings with colons neat [17:30:14] This is fine yeah just don't create a bunch of spambots lol or random users who don't need it. [17:30:33] Ofc. Basically, if an SRE would approve, I’m fine to [17:30:39] [02dns] 07Universal-Omega closed pull request 03#517: Add ACME challenges for wikitide.org - 13https://github.com/miraheze/dns/pull/517 [17:30:41] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d1eb7b7ce1ac...99c0a695929c [17:30:41] [02dns] 07MacFan4000 0399c0a69 - Add ACME challenges for wikitide.org (#517) [17:31:40] What is SRE? [17:31:49] https://tenor.com/view/1984-skander-big-brother-gif-21303304 [17:31:53] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=thegreatwarwiki --update (END - exit=0) [17:31:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:12] https://cdn.discordapp.com/attachments/808001911868489748/1237819353303355393/image0.jpg?ex=663d089b&is=663bb71b&hm=87a8b0929f7519d73ccdd25be8ab2867fffb500ba1eeed3f54df709f79a2b3b7& [17:32:44] La resistance [17:33:02] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/refreshLinks.php --wiki=thegreatwarwiki --update (START) [17:33:03] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/refreshLinks.php --wiki=thegreatwarwiki --update (END - exit=256) [17:33:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:11] I haven’t updated my discord in ever because I’m lazy and don’t wanna look for my iCloud password [17:33:13] So uh [17:33:14] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/refreshLinks.php --wiki=thegreatwarwiki (START) [17:33:14] Hmm? Does that still show for you like that lol or is it an old screenshot? [17:33:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:28] trusty refreshLinks to the rescue [17:34:11] https://cdn.discordapp.com/attachments/808001911868489748/1237819856032632892/image0.jpg?ex=663d0913&is=663bb793&hm=9c6238d866485ceef0d3e1e3df5dee6062a628237e5b03bbda1f5db893e082e1& [17:34:32] My discord is a weee bit fucked [17:34:35] How? lol [17:34:36] For example [17:34:56] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query. [17:35:01] [1/2] External app commands who now? [17:35:01] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1237820063529304147/image0.jpg?ex=663d0944&is=663bb7c4&hm=4e34d901ed25e528da41c2116894719f0bde74307fd4c84da8caa0f1439d7fc6& [17:35:39] _I'm confused lol_ [17:35:59] My discord hasn’t been updated in so long [17:36:01] I: [17:36:07] A: have the old UI still [17:36:08] Hmm that doesn't relay lol I didn't think it would though [17:36:18] B: lost use of applications commands [17:36:29] And C: have a ton of weird cacheing issues [17:36:43] I have a ton of caching issues also. [17:36:51] Ie I enabled the red theme when I had nitro [17:36:54] Still have it [17:37:05] App commands is annoying though [17:37:11] I need to use the web version [17:37:16] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/refreshLinks.php --wiki=thegreatwarwiki (END - exit=0) [17:37:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:50] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=thegreatwarwiki --update (END - exit=0) [17:37:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:58] I wonder how I can make slash commands like /me relay... [17:38:15] _is confused_ [17:38:16] _this won't relay_ [17:38:23] It don’t? [17:38:27] lol it doss now [17:38:33] it didnt before [17:38:35] Doesn’t it count as a normal message send [17:38:41] It should [17:38:42] ohhh [17:38:45] I wonder what it would think of user installed apps now [17:38:55] I had the relay muted on accident lmao [17:39:05] They won’t work here cause too bug [17:39:18] Hm. I could try in the wiki creator chat [17:39:47] It works I just had relay ignored on IRC so did not show up for me [17:40:32] PROBLEM - mem161 APT on mem161 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [17:41:01] Lol [17:42:33] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 55 packages available for upgrade (28 critical updates). [17:44:20] PROBLEM - mw182 APT on mw182 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [17:46:19] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (6 critical updates). [17:47:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:47:46] PROBLEM - mw162 APT on mw162 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [17:49:47] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (6 critical updates). [17:54:15] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [17:59:18] [02dns] 07MacFan4000 opened pull request 03#518: update wikitide.org ACME challenges - 13https://github.com/miraheze/dns/pull/518 [18:15:25] [02dns] 07Universal-Omega closed pull request 03#518: update wikitide.org ACME challenges - 13https://github.com/miraheze/dns/pull/518 [18:15:27] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/99c0a695929c...29bb96a5312c [18:15:30] [02dns] 07MacFan4000 0329bb96a - update wikitide.org ACME challenges (#518) [18:33:29] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:52:51] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.407 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [19:22:09] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [19:26:04] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/29bb96a5312c...3aef00981a79 [19:26:06] [02dns] 07Universal-Omega 033aef009 - Remove ACME [19:33:23] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [19:57:18] PROBLEM - wiki.barengreza.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.barengreza.my.id All nameservers failed to answer the query. [20:02:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:05:11] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [20:23:36] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c004a9b4168d...d2ea40b7d959 [20:23:39] [02ssl] 07WikiTideSSLBot 03d2ea40b - Bot: Update SSL cert for wikitide.org [20:27:04] PROBLEM - wiki.barengreza.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.barengreza.my.id could not be found [20:33:48] !log [alex@test151] Upgraded packages libglib2.0-dev, libglib2.0-dev-bin, libglib2.0-data, libglib2.0-bin, libglib2.0-0, linux-image-6.1.0-21-amd64, linux-image-amd64, and linux-libc-dev [20:33:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:33:59] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:34:09] !log rebooted test151 [20:34:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:36:20] RECOVERY - test151 APT on test151 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [20:49:11] @rhinosf1 did you end up doing that cache thing you mentioned btw? [20:49:36] @bluemoon0332 the one for /1.41/* [20:49:38] ? [20:49:40] yes [20:49:48] @bluemoon0332 not yet, why [20:49:54] just curious [20:50:43] @bluemoon0332 I can look on Friday if other tech people feel it's sensible [20:51:03] RECOVERY - polcompball.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 06 Aug 2024 07:23:30 PM GMT +0000. [20:52:15] [02puppet] 07Universal-Omega pushed 031 commit to 03relaybot-dotnet [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5526aa5dc839 [20:52:16] [02puppet] 07Universal-Omega 035526aa5 - relaybot: upgrade to dotnet-sdk-8.0 [20:52:18] [02puppet] 07Universal-Omega created branch 03relaybot-dotnet - 13https://github.com/miraheze/puppet [20:52:21] [02puppet] 07Universal-Omega opened pull request 03#3849: relaybot: upgrade to dotnet-sdk-8.0 - 13https://github.com/miraheze/puppet/pull/3849 [20:52:27] Dr.FLare is an amazing extension to track what cloudflare is caching and not btw if anyone didn't know [20:53:01] @originalauthority I did not [20:53:12] But then I have most of it's logs too [20:53:52] [02puppet] 07Universal-Omega pushed 031 commit to 03relaybot-dotnet [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5526aa5dc839...461b83d7f1a8 [20:53:55] [02puppet] 07Universal-Omega 03461b83d - Use 6.0 for cvtbot [20:53:58] [02puppet] 07Universal-Omega synchronize pull request 03#3849: relaybot: upgrade to dotnet-sdk-8.0 - 13https://github.com/miraheze/puppet/pull/3849 [20:54:06] [1/2] I ie I can see we are missing caching both these on every single pageload [20:54:06] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1237870166432284762/image.png?ex=663d37ee&is=663be66e&hm=a2d0eef9a1e792e3ca8cc80320731e7b6f9a4bad25e6ec1fe36dc84daee88ba3& [20:54:09] RECOVERY - wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 06 Aug 2024 07:23:30 PM GMT +0000. [20:54:21] Is MH using workers or page rules? [20:54:22] [02puppet] 07Universal-Omega pushed 031 commit to 03relaybot-dotnet [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/461b83d7f1a8...e852edc64ae3 [20:54:24] [02puppet] 07Universal-Omega 03e852edc - Move [20:54:25] [02puppet] 07Universal-Omega synchronize pull request 03#3849: relaybot: upgrade to dotnet-sdk-8.0 - 13https://github.com/miraheze/puppet/pull/3849 [20:54:53] @originalauthority cdn-cgi is served at the edge anyway [20:54:54] [02puppet] 07Universal-Omega pushed 031 commit to 03relaybot-dotnet [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/e852edc64ae3...e406424b6a6d [20:54:57] [02puppet] 07Universal-Omega 03e406424 - Move [20:54:58] [02puppet] 07Universal-Omega synchronize pull request 03#3849: relaybot: upgrade to dotnet-sdk-8.0 - 13https://github.com/miraheze/puppet/pull/3849 [20:55:11] I was more on about codex [20:55:17] I can look [20:55:26] that's a pretty funny meta logo [20:55:45] its green to say its cached [20:55:54] 😳 [20:57:09] @originalauthority we have a cache hit on 2.54m/3.15m requests to load.php [20:57:46] @originalauthority do you know what content type is for that? [20:57:54] for what [20:57:55] load.php? [20:58:14] @originalauthority the codex one ye [20:58:22] I assume it's js [20:58:24] ? [20:58:26] text/css [20:58:57] @originalauthority 402/715 for css [20:59:18] So 56% cache hit rate for load.php requests of type text/css [20:59:39] [02puppet] 07Universal-Omega pushed 031 commit to 03relaybot-dotnet [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e406424b6a6d...bff153c42711 [20:59:42] [02puppet] 07Universal-Omega 03bff153c - Use variable [20:59:44] [02puppet] 07Universal-Omega synchronize pull request 03#3849: relaybot: upgrade to dotnet-sdk-8.0 - 13https://github.com/miraheze/puppet/pull/3849 [20:59:47] hmmm [20:59:50] could be better [20:59:58] [02puppet] 07Universal-Omega pushed 031 commit to 03relaybot-dotnet [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/bff153c42711...30448c6e8109 [20:59:59] [02puppet] 07Universal-Omega 0330448c6 - Move [21:00:02] [02puppet] 07Universal-Omega synchronize pull request 03#3849: relaybot: upgrade to dotnet-sdk-8.0 - 13https://github.com/miraheze/puppet/pull/3849 [21:02:56]