[01:25:58] [1/2] The Hungarian flag on this page is broken: [01:25:58] [2/2] https://leunissendoranproductionswiki.miraheze.org/wiki/Category:Hungarian_characters [01:48:09] [1/2] Same with the American flag on this page: [01:48:09] [2/2] https://leunissendoranproductionswiki.miraheze.org/wiki/Category:American_characters [01:58:44] Seems its supposed to be loading from instant commins, which is enabled on the wiki, but its not loading for some reason. [01:59:18] Can you fix it, though? [01:59:28] I will take a look shortly. [01:59:33] Okay then! [01:59:35] I am not sure why it isnt loading. [01:59:53] Are you a bureaucrat on the wiki? [02:00:09] Well, I did co-found it. [02:01:36] Can you try and disable "Enable Wikimedia Commons Files ($wgUseQuickInstantCommons)" in Special:ManageWiki/settings and then re-enable it if you have correct permissions to? [02:04:17] No, I don't have permission to do that. [02:04:33] It might be an upstream issue with Wikimedia Commons [02:05:52] Although its very odd that it just acts like instant commons is disabled. [02:06:00] Tell me about it! [02:10:14] It's weird, since I've been getting the dreaded ``[[Category:Pages with broken file links]]`` category after crosschecking on my wiki as of late. [02:10:36] Yeah Wikimedia Commons is down it seems globally(?) [02:10:50] I guess its an upstream issue. [02:11:47] I guess so. [05:13:32] commons doesn't load images for me either.. [05:14:19] at least negative caching is only for 5m [07:04:28] We've used Instant Commons on UT/DR up until recently and it really frequently happened that the images just turned into redlinks because the WMF server died [07:05:41] So we switched away from it [07:51:11] I'm not sure whether this is related, but we will need to upgrade QuickInstantCommons soon because WMF will require a UA for requests like this, and block requests without one [07:53:00] [1/2] https://phabricator.wikimedia.org/T400881 [07:53:00] [2/2] https://phabricator.wikimedia.org/T400119 [07:56:01] I think it might be that [08:04:20] It very well might be that [08:05:32] try updating quickinstantcommons then, a change was made there that adds a link to the mainpage of the wiki it is being used at [08:05:42] Will do [08:06:11] unless im logged into bots171 trying to run mwdeploy lol [08:06:40] thought I was on mwtask181 [08:10:47] doesnt seem to have made a difference but done. [08:21:28] maybe our IPs are still under the effect of the rate limit then? [08:21:39] may fix itself magically later then [10:18:37] @cosmicalpha is there anything in the logs by any chance? [10:19:19] the extension does log all HTTP requests and failures according to the source code, should tell us what the deal is [10:22:24] @abaddriverlol or @blankeclair able to look ^ [10:22:37] graylog won't load for me on my laptop for some reason [10:22:39] god damn it, okay [10:25:07] we only log up to error [10:25:23] pump it up to warning [10:25:36] what if i do a manual curl on bast161? [10:25:41] [1/2] that's when it logs HTTP failures [10:25:41] [2/2] https://github.com/wikimedia/mediawiki-extensions-QuickInstantCommons/blob/master/src/MultiHttpClient.php#L334C6-L335C35 [10:27:08] [1/13] ``` [10:27:08] [2/13] [blankeclair@bast161:~]$ curl -v 'https://commons.wikimedia.org/wiki/File:Zumenon_(estradiol_hemihydrate)_tablets_in_Australia,_with_one_obverse_and_one_reverse_blister_packs.jpg' --user-agent 'Miraheze, Claire manually using curl to check responses (https://miraheze.org; blankeclair@wikitide.org)' [10:27:08] [3/13] * Trying [2606:4700::6812:6be]:443... [10:27:09] [4/13] * Connected to commons.wikimedia.org (2606:4700::6812:6be) port 443 (#0) [10:27:09] [5/13] * ALPN: offers h2,http/1.1 [10:27:09] [6/13] * TLSv1.3 (OUT), TLS handshake, Client hello (1): [10:27:10] [7/13] * CAfile: /etc/ssl/certs/ca-certificates.crt [10:27:10] [8/13] * CApath: /etc/ssl/certs [10:27:10] [9/13] * TLSv1.3 (IN), TLS alert, handshake failure (552): [10:27:11] [10/13] * OpenSSL/3.0.17: error:0A000410:SSL routines::sslv3 alert handshake failure [10:27:11] [11/13] * Closing connection 0 [10:27:11] [12/13] curl: (35) OpenSSL/3.0.17: error:0A000410:SSL routines::sslv3 alert handshake failure [10:27:12] [13/13] ``` [10:27:12] ? [10:27:26] works well on my laptop... [10:27:32] Do -vvv @blankeclair [10:27:46] ineffective for curl, but okay [10:27:57] same thing, i'll try the proxy even though it probably wouldn't help [10:29:24] [1/24] ``` [10:29:24] [2/24] [blankeclair@bast161:~]$ curl -v 'http://commons.wikimedia.org/wiki/File:Zumenon_(estradiol_hemihydrate)_tablets_in_Australia,_with_one_obverse_and_one_reverse_blister_packs.jpg' --user-agent 'Miraheze, Claire manually using curl to check responses (https://miraheze.org; blankeclair@wikitide.org)' -vvv [10:29:25] [3/24] * Trying [2606:4700::6812:6be]:80... [10:29:25] [4/24] * Connected to commons.wikimedia.org (2606:4700::6812:6be) port 80 (#0) [10:29:25] [5/24] > GET /wiki/File:Zumenon_(estradiol_hemihydrate)_tablets_in_Australia,_with_one_obverse_and_one_reverse_blister_packs.jpg HTTP/1.1 [10:29:26] [6/24] > Host: commons.wikimedia.org [10:29:26] [7/24] > User-Agent: Miraheze, Claire manually using curl to check responses (https://miraheze.org; blankeclair@wikitide.org) [10:29:26] [8/24] > Accept: / [10:29:27] [9/24] > [10:29:27] [10/24] < HTTP/1.1 409 Conflict [10:29:27] [11/24] < Date: Fri, 22 Aug 2025 10:29:09 GMT [10:29:28] [12/24] < Content-Type: text/plain; charset=UTF-8 [10:29:28] [13/24] < Content-Length: 16 [10:29:29] [14/24] < Connection: close [10:29:29] [15/24] < X-Frame-Options: SAMEORIGIN [10:29:30] [16/24] < Referrer-Policy: same-origin [10:29:30] [17/24] < Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 [10:29:31] [18/24] < Expires: Thu, 01 Jan 1970 00:00:01 GMT [10:29:31] [19/24] < Server: cloudflare [10:29:32] [20/24] < CF-RAY: 9731b001482cd96a-SLC [10:29:32] [21/24] < [10:29:33] [22/24] * Closing connection 0 [10:29:33] [23/24] error code: 1001[blankeclair@bast161:~]$ [10:29:34] [24/24] ``` [10:30:26] ugh, what kinda fucking dns resolver are we using [10:30:32] \* Connected to commons.wikimedia.org.invalid (2606:4700::6812:7be) port 80 (#0) [10:31:09] we do recursive resolving on the servers with powerdns I believe? [10:31:30] how did we convert NXDOMAIN to cloudflare [10:32:21] I don't know, I never really touched DNS, only NTP and databases 🤷 [10:32:27] OH [10:32:31] [1/2] [blankeclair@bast161:~]$ cat /etc/resolv.conf [10:32:31] [2/2] search wikitide.net miraheze.org [10:32:34] god fucking hell [10:32:38] no wonder why [10:33:42] [1/2] search directives [10:33:42] [2/2] [10:34:07] better link: https://github.com/ArchiveTeam/ArchiveBot/issues/318 [10:35:23] https://github.com/miraheze/puppet/blob/main/modules/base/templates/dns/resolv.conf.erb [10:36:31] [1/15] ``` [10:36:31] [2/15] [blankeclair@bast161:~]$ wget https://commons.wikimedia.org [10:36:32] [3/15] --2025-08-22 10:35:49-- https://commons.wikimedia.org/ [10:36:32] [4/15] Resolving commons.wikimedia.org (commons.wikimedia.org)... 2606:4700::6812:7be, 2606:4700::6812:6be, 104.18.6.190, ... [10:36:32] [5/15] Connecting to commons.wikimedia.org (commons.wikimedia.org)|2606:4700::6812:7be|:443... connected. [10:36:33] [6/15] GnuTLS: A TLS fatal alert has been received. [10:36:33] [7/15] GnuTLS: received alert [40]: Handshake failed [10:36:33] [8/15] Unable to establish SSL connection. [10:36:34] [9/15] [blankeclair@bast161:~]$ dig +short commons.wikimedia.org.miraheze.org AAAA [10:36:34] [10/15] 2606:4700::6812:6be [10:36:34] [11/15] 2606:4700::6812:7be [10:36:35] [12/15] [blankeclair@bast161:~]$ dig +short commons.wikimedia.org.miraheze.org A [10:36:35] [13/15] 104.18.6.190 [10:36:36] [14/15] 104.18.7.190 [10:36:36] [15/15] ``` [10:36:37] mmh, how annoying [10:42:31] hmm... it only manifested since all the resolvers failed [10:43:31] wot [10:43:51] [1/15] ``` [10:43:52] [2/15] [blankeclair@bast161:~]$ grep nameserver /etc/resolv.conf | cut -d' ' -f2 | while read -r i; do echo "$i:"; dig @"$i" commons.wikimedia.org AAAA | grep HEADER; done [10:43:52] [3/15] ::1: [10:43:52] [4/15] ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36562 [10:43:52] [5/15] 10.0.17.136: [10:43:53] [6/15] ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 32118 [10:43:53] [7/15] 2602:294:0:b23::111: [10:43:53] [8/15] ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 63498 [10:43:54] [9/15] 2001:41d0:801:2000::4089: [10:43:54] [10/15] ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 1923 [10:43:54] [11/15] 38.46.223.204: [10:43:55] [12/15] ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 47670 [10:43:55] [13/15] 51.75.170.66: [10:43:56] [14/15] ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 18967 [10:43:56] [15/15] ``` [10:44:02] lmao [10:44:18] uh [10:44:22] everything's not resolving? [10:44:31] why tf [10:44:44] this is like a super critical everyone gets out of bed and comes help priority task on Phorge [10:45:19] okay but like girl wtf why [10:45:23] why is this happeninggggg [10:46:53] could be lots of things [10:47:08] I assume those external IPs aren't ours, maybe they blocked us for spamming requests [10:47:28] maybe something happened with our host and they're blocking DNS traffic somehow [10:47:48] huh [10:48:19] what servers have powerdns again? [10:48:24] we should look at the pdns-recursor logs at least, first is figuring out why our own recursive resolvers aren't working [10:49:01] everyone runs their own instance [10:49:06] https://github.com/miraheze/puppet/blob/main/modules/base/manifests/dns.pp [10:49:20] ah, each server then? [10:49:51] I believe, although I'm not sure: https://github.com/miraheze/puppet/blob/main/modules/base/manifests/init.pp#L20C4-L26C6 [10:50:05] idk when this is true [10:50:19] mwtask181 has pdns-recursor, neat [10:50:29] uh [10:50:35] it should be firewalled off... [10:55:17] Hello, I'm here to chat with you! ;) [10:55:20] 03IP14 [[m:User:2.73.144.214]] 03Tiny create14 [[m:Translations:Wikimedia chapters/5/gn]] 04(+2)14 10URL:12 https://meta.wikimedia.org/w/index.php?oldid=29163648&rcid=3657902314 "Created page with "н"" [10:55:40] kicked [10:55:47] who even is this [10:56:00] some spammer, it was also on #miraheze-tech-ops earlier today [11:05:57] [1/7] ``` [11:05:57] [2/7] [1] commons.wikimedia.org: Failed to get IP for NS ns0.wikimedia.org, trying next if available [11:05:58] [3/7] [1] commons.wikimedia.org: Failed to resolve via any of the 3 offered NS at level 'wikimedia.org' [11:05:58] [4/7] [1] commons.wikimedia.org: Ageing nameservers for level 'wikimedia.org', next query might succeed [11:05:58] [5/7] [1] commons.wikimedia.org: failed (res=-1) [11:05:58] [6/7] ``` [11:05:59] [7/7] hmm... [11:07:09] i'm not really in the mood to digest pdns logs tho ^^; [11:07:20] i'll be a blunt pig and become the dns resolver [11:10:32] yeah i have no ide what i'm doing [11:19:49] [1/2] hmm, typical pdns log: [11:19:49] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1408410328043356190/1755861588829.txt?ex=68a9a3d5&is=68a85255&hm=0f0e5faceee354f3b5338c79195ebaa8b51ff1ac816946df114036f25714cfda& [11:19:59] [1/24] [1] wikimedia.org: Trying to resolve NS 'a0.org.afilias-nst.info' (1/6) [11:19:59] [2/24] [1] Nameserver a0.org.afilias-nst.info IPs: 2001:500:e::1(0.00ms) [11:20:00] [3/24] [1] wikimedia.org: Resolved 'org' NS a0.org.afilias-nst.info to: 2001:500:e::1 [11:20:00] [4/24] [1] wikimedia.org: Trying IP [2001:500:e::1]:53, asking 'wikimedia.org|A' [11:20:00] [5/24] [1] wikimedia.org: Got 7 answers from a0.org.afilias-nst.info (2001:500:e::1), rcode=0 (No Error), aa=0, in 121ms [11:20:00] [6/24] [1] wikimedia.org: accept answer 'wikimedia.org|NS|ns2.wikimedia.org.' from 'org' nameservers? ttl=3600, place=2 YES! [11:20:01] [7/24] [1] wikimedia.org: accept answer 'wikimedia.org|NS|ns1.wikimedia.org.' from 'org' nameservers? ttl=3600, place=2 YES! [11:20:01] [8/24] [1] wikimedia.org: accept answer 'wikimedia.org|NS|ns0.wikimedia.org.' from 'org' nameservers? ttl=3600, place=2 YES! [11:20:01] [9/24] [1] wikimedia.org: accept answer 'ns0.wikimedia.org|A|208.80.154.238' from 'org' nameservers? ttl=3600, place=3 YES! [11:20:02] [10/24] [1] wikimedia.org: accept answer 'ns1.wikimedia.org|A|208.80.153.231' from 'org' nameservers? ttl=3600, place=3 YES! [11:20:02] [11/24] [1] wikimedia.org: accept answer 'ns2.wikimedia.org|A|198.35.27.27' from 'org' nameservers? ttl=3600, place=3 YES! [11:20:03] [12/24] [1] wikimedia.org: OPT answer '.' from 'org' nameservers [11:20:03] [13/24] [1] wikimedia.org: determining status after receiving this packet [11:20:04] [14/24] [1] wikimedia.org: got NS record 'wikimedia.org' -> 'ns2.wikimedia.org.' [11:20:04] [15/24] [1] wikimedia.org: got NS record 'wikimedia.org' -> 'ns1.wikimedia.org.' [11:20:05] [16/24] [1] wikimedia.org: got NS record 'wikimedia.org' -> 'ns0.wikimedia.org.' [11:20:05] [17/24] [1] wikimedia.org: status=did not resolve, got 3 NS, looping to them [11:20:06] [18/24] [1] wikimedia.org.: Nameservers: ns1.wikimedia.org(0.00ms), ns0.wikimedia.org(0.00ms), ns2.wikimedia.org(0.00ms) [11:20:06] [19/24] [1] wikimedia.org: Trying to resolve NS 'ns1.wikimedia.org' (1/3) [11:20:07] [20/24] [1] : no TA found for 'ns1.wikimedia.org' among 1 [11:20:07] [21/24] [1] : no TA found for 'wikimedia.org' among 1 [11:20:08] [22/24] [1] : no TA found for 'org' among 1 [11:20:08] [23/24] [1] : got TA for '.' [11:20:09] [24/24] [1] QM ns1.wikimedia.org.|AAAA child=(empty): doResolve [11:21:59] hypothesis: does pdns fail to resolve domains if the glue records for the nameservers do not have AAAA records? [11:26:57] hmm, tried on compass.education [11:27:17] https://cdn.discordapp.com/attachments/1006789349498699827/1408412208383131719/1755862035538.txt?ex=68a9a595&is=68a85415&hm=84e75a4dcb11cf227a84f094a228421314f12a12aaa8198e8f863e71de4a4eee& [11:27:46] did wikimedia change anything regarding ipv6? [11:28:00]