[00:35:33] !log created proxmox accounts for void (yes it was somehow missing) and evalprime [00:35:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:36:15] how in tarnation [00:37:19] Also nice to see the random feature i added is being used to effectively (discord side loggin) [00:38:28] Yeah I quite like it lol [00:38:49] no idea how Void's account was missed lol [00:41:42] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.64, 16.31, 8.85 [00:41:53] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:42:49] PROBLEM - db162 MariaDB on db162 is UNKNOWN: [00:43:26] my shitty python skills being useful holy shit:UnicornMH: lmao glad it helps [00:43:42] PROBLEM - db162 SSH on db162 is CRITICAL: connect to address 10.0.16.140 and port 22: No route to host [00:43:49] PROBLEM - db162 PowerDNS Recursor on db162 is CRITICAL: connect to address 10.0.16.140 port 5666: No route to hostconnect to host 10.0.16.140 port 5666: No route to host [00:43:50] PROBLEM - Host db162 is DOWN: CRITICAL - Host Unreachable (10.0.16.140) [00:45:52] he’s only the longest serving sre member [00:46:21] [02puppet] 07MacFan4000 opened pull request 03#3845: rm db162 - 13https://github.com/miraheze/puppet/pull/3845 [00:46:55] Technically that would be reception [00:50:55] True [00:51:03] Continously it’s void [00:51:06] [02dns] 07MacFan4000 opened pull request 03#514: rm db162 - 13https://github.com/miraheze/dns/pull/514 [00:51:42] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 5.65, 17.78, 15.06 [00:59:50] I was actually on SRE before Void but yeah continually it is Void lol [01:01:35] You know using Proxmox on mobile is getting annoying lol. So is using the developer console to debug CF headers on mobile lol [01:12:40] [02dns] 07Universal-Omega closed pull request 03#514: rm db162 - 13https://github.com/miraheze/dns/pull/514 [01:12:42] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9f8c30e2b0dd...8f3cf62ab63f [01:12:45] [02miraheze/dns] 07MacFan4000 038f3cf62 - rm db162 (#514) [01:12:54] [02puppet] 07Universal-Omega closed pull request 03#3845: rm db162 - 13https://github.com/miraheze/puppet/pull/3845 [01:12:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/661da868e5f0...66f0db3518c7 [01:12:57] [02miraheze/puppet] 07MacFan4000 0366f0db3 - rm db162 (#3845) [01:15:46] !log remove db162 from puppet [01:15:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:21] !log destroy db162 VM [01:21:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:41] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.31 ms [01:32:01] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:34:39] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.47 ms [01:51:56] PROBLEM - wiki.danvs.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.danvs.net All nameservers failed to answer the query. [02:20:52] RECOVERY - wiki.danvs.net - reverse DNS on sslhost is OK: SSL OK - wiki.danvs.net reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:30:50] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [02:33:54] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [03:03:38] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [03:03:59] !log bush hid the facts [03:04:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:04:18] huh? [03:04:30] mate [03:04:47] oh noes, i was so bored [03:05:09] can you log that? lol [03:06:30] [1/2] and it appears [03:06:31] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1233615230001745920/Captura_de_Pantalla_2024-04-26_a_las_9.06.08_p.m..png?ex=662dbd36&is=662c6bb6&hm=647d6014d39fa660ac8d5d93277dff0a1c00d8382c4f172bcd7d06d33fb67340& [03:06:36] Hi so @emicraftnoob - that's not happeing [03:07:02] Preferably don't create extra work for volunteers, especially stupid work [03:07:07] i screwed up [03:07:38] I reverted [03:08:04] Now that that's been taken care of - let's not have it happen again. Appreciated @emicraftnoob for admitting fault. [03:08:38] yeah, it was so stupid, was pure boredom magic lol [03:08:49] Its fine. If we really didn't want it we would restrict who can log [03:09:02] Which maybe we should [03:09:07] Probably should tbh [03:09:10] yeah [03:09:16] No reason for non-SREs to be logging that I can think of [03:09:30] it could definitely be misused in wise ways + what brandon said [03:09:36] worse* [03:09:38] I can just think of a spammer here that decides to log spam [03:09:50] yeah, we can get raided with a lot of logs [03:10:09] easy to revert and likely would be caught fast but yeah.. [03:10:18] Oh, dmehus renamed to Doug..... [03:10:23] yes [03:10:25] loving the about-to-appear redlinks [03:10:29] thousands of them [03:10:58] Redirect User:Dmehus [03:11:00] I want to try logging [03:11:01] yeah y'alls decision obv but I'd recommend restricting [03:11:02] I won't though [03:11:04] ah yes [03:11:32] https://en.wikipedia.org/wiki/Bush_hid_the_facts [03:11:52] Doug is prominent enough User:Dmehus should probably never be used again to avoid impersonation tbh [03:12:08] going to check now if it's blacklisted [03:12:24] also should I redirect Agent's userpage to his GDPR username? [03:12:38] no that would violate GDPR [03:12:42] ah yeah [03:12:47] if/when you can move....https://login.miraheze.org/wiki/User:Dmehus?redirect=no [03:12:54] or maybe it auto-does [03:13:05] It will auto when it gets there [03:13:41] I should probably rename on-wiki to CosmicAlpha to normalize my usernames tbh [03:13:58] or rename here to UO :p [03:14:02] that might honestly be easier [03:14:14] I like CosmicAlpha better [03:14:17] easier to type lol [03:14:22] Ah fai [03:14:26] then yeah [03:14:36] yes, I spent so long one time finding your wiki user [03:15:12] what mobile has a dev thing [03:15:34] Kiwi browser [03:15:42] i think i mentioned it when i adde discord side logging [03:15:44] I'm going to blacklist UO too [03:16:06] huh? oh [03:16:17] from ever being created again :p [03:16:22] you replying to something totally different through me off lol [03:16:22] or anything close to it [03:16:26] yeah sorry [03:16:57] Why are we having this discussion in #sre, which I just realized? lol [03:17:13] I dunno [03:17:38] I've heard about this before lol [03:18:29] because I telled to everyone in SRE the USA's most top secret thing ever lmao [03:19:54] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.28 ms [03:21:53] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.36 ms [03:25:59] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.47 ms [03:27:58] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.13 ms [03:37:07] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 180.88 ms [03:39:06] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.11 ms [04:01:07] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [04:04:41] [02CreateWiki] 07pixDeVl opened pull request 03#511: Notify Wiki Creators about requesters with over X(5) request (T12075) - 13https://github.com/miraheze/CreateWiki/pull/511 [04:13:43] PROBLEM - nijigen.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nijigen.wiki could not be found [04:28:26] PROBLEM - nijigen.wiki - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [04:43:28] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 195.96 ms [04:47:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.07 ms [04:59:46] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [05:04:31] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:09:44] PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:33:44] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:33:58] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [05:37:26] PROBLEM - tfuk.app - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tfuk.app' expires in 15 day(s) (Mon 13 May 2024 05:32:02 AM GMT +0000). [05:37:34] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/586f9855f453...f732940bb6a5 [05:37:36] [02miraheze/ssl] 07WikiTideSSLBot 03f732940 - Bot: Update SSL cert for tfuk.app [05:50:47] !log [void@mwtask181] fixStuckGlobalRename.php --wiki=mundoparalelowiki --logwiki=metawiki Dmehus Doug [05:50:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:52:35] Oh thanks Void [05:52:47] I was just looking at that lol [05:53:06] Ah good, was just going to ask someone else to keep an eye on it [05:53:25] I'm off to bed now, and it's possible it might fail again [05:53:31] Yeah I can for about two-3 hours longer [05:53:58] Night, don't stay up too late :) [05:54:04] If not @reception123 could probably monitor after [05:54:15] Hey yeah lol. Night. [05:59:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.62 ms [06:00:38] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:01:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.89 ms [06:01:00] PROBLEM - wiki.kkstudio.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kkstudio.eu.org' expires in 15 day(s) (Mon 13 May 2024 05:36:28 AM GMT +0000). [06:03:47] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:06:52] RECOVERY - tfuk.app - LetsEncrypt on sslhost is OK: OK - Certificate 'tfuk.app' will expire on Fri 26 Jul 2024 04:37:28 AM GMT +0000. [06:08:47] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:30:01] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:30:12] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.05 ms [06:31:45] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:32:11] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.58 ms [06:36:46] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:52:10] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=talenteddeviantswiki --logwiki=metawiki Dmehus Doug (END - exit=0) [06:52:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:56:37] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=theboyswiki --logwiki=metawiki Dmehus Doug (END - exit=0) [06:56:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:19] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=theburningprincessnoteswiki --logwiki=metawiki Dmehus Doug (END - exit=0) [06:57:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:59:09] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=thefanonworldwiki --logwiki=metawiki Dmehus Doug (END - exit=0) [06:59:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:00:31] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=theheroeswiki --logwiki=metawiki Dmehus Doug (END - exit=0) [07:00:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:01:41] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=theloracularwiki --logwiki=metawiki Dmehus Doug (END - exit=0) [07:01:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:11:52] [1/8] This is what is causing failing renames btw [07:11:52] [2/8] ``` [07:11:52] [3/8] 2024-04-27 07:10:48 LocalRenameUserJob Global_rename_job from=Dmehus to=Doug renamer=1108-Kiju movepages=1 suppressredirects= reason=per [[m:Special:Permalink/391707#Username_usurpation_of_User:Doug|requested]] ([[m:Stewards|steward]] action) ignorestatus= reattach=array(12040) promotetoglobal= namespace=0 title=Global_rename_job requestId=8a5795341c391165d06178e5 (uuid=cca5099e6 [07:11:52] https://meta.wikimedia.org/wiki/Special:Permalink/391707#Username_usurpation_of_User:Doug https://meta.wikimedia.org/wiki/Stewards [07:11:53] [4/8] d9a4fd4926dc1a99482916d,timestamp=1714201831) t=505 error=Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'trablangwiki.cu_private_event' doesn't exist [07:11:53] [5/8] Function: MediaWiki\CheckUser\Hooks::onLocalUserCreated [07:11:53] [6/8] Query: INSERT INTO `cu_private_event` (cupe_namespace,cupe_title,cupe_log_type,cupe_log_action,cupe_params,cupe_page,cupe_actor,cupe_timestamp,cupe_ip,cupe_ip_hex,cupe_xff,cupe_xff_hex,cupe_agent,cupe_comment_id) VALUES (2,'','checkuser-private-event','autocreate-account','a:0:{}',0,140,'20240427071048','127.0.0.1','7F000001','',NULL,'',5) [07:11:54] [7/8] in /srv/mediawiki/1.41/includes/libs/rdbms/database/Database.php on line 1236 [07:11:54] [8/8] ``` [07:12:19] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=trablangwiki --logwiki=metawiki Dmehus Doug (END - exit=0) [07:12:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:15:53] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=triassicwiki --logwiki=metawiki Dmehus Doug (END - exit=0) [07:15:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:18:10] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=trimirdiwiki --logwiki=metawiki Dmehus Doug (END - exit=0) [07:18:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:15] I am going now [07:19:29] @reception123 are you able to pick up on it? [07:30:52] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:36:10] Yep [07:37:45] !log sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=tvcensorshipwiki --logwiki=metawiki Dmehus Doug [07:37:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:00:14] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [08:02:59] !log sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=twistedwonderlandwiki --logwiki=metawiki Dmehus Doug [08:03:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:02] golly, again with this? I thought we had seen the last of this by now [08:48:56] hmmm, that's an inactive wiki that was deleted from the database but somehow remains in cw_wikis [08:49:07] the actual error is that the database does not exist [08:53:58] wait, it does actually exist, my bad [08:54:03] the db at least [09:00:11] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=trablangwiki /srv/mediawiki/1.41/extensions/CheckUser/schema/mysql/patch-cu_private_event-def.sql (END - exit=0) [09:00:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:05:57] PROBLEM - ie.unigon.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ie.unigon.net' expires in 15 day(s) (Mon 13 May 2024 08:59:12 AM GMT +0000). [09:36:50] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchising.org.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:00:47] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [10:05:25] Thank you to sre's 🫡 [10:05:40] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp37.wikitide.net - CNAME OK [10:30:11] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [10:59:35] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [11:28:54] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [12:00:24] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:07:38] :thistbh: [12:12:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:29:48] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [13:01:18] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:30:40] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:56:16] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.62 ms [13:58:15] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.97 ms [14:00:04] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [14:06:19] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.47 ms [14:08:18] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.07 ms [14:11:03] Yeah for some reason CU patches didn’t get applied to all wikis [14:16:30] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.08 ms [14:18:29] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.79 ms [14:29:27] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [14:29:36] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.67 ms [14:31:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.33 ms [14:52:52] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.31 ms [14:56:52] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.76 ms [15:00:48] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [15:13:21] RECOVERY - nijigen.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nijigen.wiki' will expire on Fri 26 Jul 2024 03:09:43 AM GMT +0000. [15:30:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [15:47:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:57:16] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.37 ms [15:59:15] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.21 ms [15:59:33] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [16:06:25] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.88 ms [16:08:24] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.28 ms [16:37:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/276da9adbf0d...1677150ffbd2 [16:37:19] [02miraheze/mw-config] 07Universal-Omega 031677150 - Remove Agent Isai from ImportDump notify jobs [16:37:29] !log [@mwtask171] starting deploy of {'config': True} to all [16:37:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:38] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s [16:37:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:10] miraheze/mw-config - Universal-Omega the build passed. [16:43:52] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [16:48:35] [02CreateWiki] 07redbluegreenhat reviewed pull request 03#511 commit - 13https://github.com/miraheze/CreateWiki/pull/511#discussion_r1581860125 [16:48:38] [02CreateWiki] 07redbluegreenhat reviewed pull request 03#511 commit - 13https://github.com/miraheze/CreateWiki/pull/511#discussion_r1581860678 [16:48:39] [02CreateWiki] 07redbluegreenhat reviewed pull request 03#511 commit - 13https://github.com/miraheze/CreateWiki/pull/511#discussion_r1581860706 [16:48:40] [02CreateWiki] 07redbluegreenhat reviewed pull request 03#511 commit - 13https://github.com/miraheze/CreateWiki/pull/511#discussion_r1581860266 [16:51:43] [02CreateWiki] 07pixDeVl reviewed pull request 03#511 commit - 13https://github.com/miraheze/CreateWiki/pull/511#discussion_r1581861436 [16:53:39] !log [@test151] starting deploy of {'config': True} to test151 [16:53:40] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [16:53:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:53:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:37] [02CreateWiki] 07pixDeVl synchronize pull request 03#511: Notify Wiki Creators about requesters with over X(5) request (T12075) - 13https://github.com/miraheze/CreateWiki/pull/511 [17:01:24] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=worldboxwiki --logwiki=metawiki Dmehus Doug (END - exit=0) [17:01:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:00] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=worldofkirbycraftwiki --logwiki=metawiki Dmehus Doug (END - exit=0) [17:02:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:56] !log [@mwtask181] starting deploy of {'config': True} to all [17:03:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:06] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [17:03:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:24] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=xilambosswiki --logwiki=metawiki Dmehus Doug (END - exit=0) [17:05:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:10:22] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=yf2kwiki --logwiki=metawiki Dmehus Doug (END - exit=0) [17:10:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:11:12] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=yfwikipediawiki --logwiki=metawiki Dmehus Doug (END - exit=0) [17:11:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:14:42] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/f732940bb6a5...cbce88862b83 [17:14:44] [02miraheze/ssl] 07WikiTideSSLBot 03cbce888 - Bot: Add SSL cert for wiki.eggsdstudios.com [17:17:10] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=zhgyaanipediawiki --logwiki=metawiki Dmehus Doug (END - exit=0) [17:17:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:22] @kiju1108 it should finally be done! [17:20:36] Famous last words [17:21:55] PROBLEM - wiki.danvs.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.danvs.net All nameservers failed to answer the query. [17:22:00] Only had to run fixStuckGlobalRename on 20 wikis lol felt like more though [17:23:42] Can we make a script that runs the script that needs the script because the first script failed [17:24:07] Well it runs by a job in CentralAuth if the job fails the script is needed [17:24:52] CentralAuth moment [17:25:06] Dmehus was attached to almost 1k wikis no wonder it took a full day to finish lol [17:25:19] You chose the worst extension to unintentionally share initials with [17:25:29] lol [17:26:05] When did you realize you share a name with It [17:26:33] About a day after I changed my username here lol [17:30:18] Instant regret plus a day [17:33:45] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [17:37:48] [02puppet] 07redbluegreenhat opened pull request 03#3846: Dbadmins mysqldump - 13https://github.com/miraheze/puppet/pull/3846 [17:38:08] [02puppet] 07redbluegreenhat edited pull request 03#3846: Dbadmins mysqldump - 13https://github.com/miraheze/puppet/pull/3846 [17:38:22] [02puppet] 07redbluegreenhat edited pull request 03#3846: Allow dbadmins to use mysqldump - 13https://github.com/miraheze/puppet/pull/3846 [17:40:31] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [17:43:36] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [17:50:51] RECOVERY - wiki.danvs.net - reverse DNS on sslhost is OK: SSL OK - wiki.danvs.net reverse DNS resolves to cp37.wikitide.net - CNAME OK [17:54:21] [02puppet] 07Universal-Omega closed pull request 03#3846: Allow dbadmins to use mysqldump - 13https://github.com/miraheze/puppet/pull/3846 [17:54:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/66f0db3518c7...ac99c896596f [17:54:25] [02miraheze/puppet] 07redbluegreenhat 03ac99c89 - Allow dbadmins to use mysqldump (#3846) [18:01:09]