[01:12:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12173259 (10Papaul) [03:36:43] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12173316 (10Papaul) [04:20:15] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12173344 (10Papaul) [07:55:01] 10netops, 06Infrastructure-Foundations: Investigate Nokia Apply-path for prefix-lists - https://phabricator.wikimedia.org/T433662 (10ayounsi) 03NEW p:05Triage→03Low [09:12:57] 06Traffic, 06SRE: Prometheus Ferm MSS export does not work for IPv6 - https://phabricator.wikimedia.org/T433672 (10cmooney) 03NEW p:05Triage→03Low [09:30:40] 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Support LVS backend servers using nftables - https://phabricator.wikimedia.org/T433601#12173746 (10cmooney) >>! In T433601#12171476, @ssingh wrote: > The primary change that I think we need to make is in `modules/profile/manifests/lvs/real... [09:43:11] hello [09:44:07] could you guys please do something about the extremely slow speed and latency of Wikimedia DNS [09:44:52] there's only 1 server for EU and the Middle East and so slow [09:45:05] latency: average: 200 ms [09:45:12] cloudflare: 10 ms [09:48:55] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12173771 (10Ladsgroup) [09:53:00] MG2021: I assumed you're talking about DoH and DoT servers, how did you measured the latency ? ping? [09:54:00] @everyone Is there anyone here to check my room? [09:54:32] fabfur oh hi [09:54:39] i'll tell you now [09:56:26] 10netops, 06Infrastructure-Foundations, 06SRE: Consider removing BFD on datacentre IBGP peerings - https://phabricator.wikimedia.org/T433675 (10cmooney) 03NEW p:05Triage→03Low [09:57:34] MG2021: unfortunately we aren't really in a position to do that [09:58:15] just like Cloudflare we run the DNS service from all our POP locations. However we have much fewer of these than Cloudflare or those other large billion dollar comanies [09:58:18] *companies [09:59:04] setting up widely dispersed infrastructure like that is very expensive and we need to be judicious with our use of donor funds. [09:59:15] it's unfortunate but not something we can easily address I'm afraid [09:59:22] MG2021: an interesting insight could be to check if you experience the same latency with the other wikipedia services too [10:01:11] fabfur i'll upload my results here now [10:03:21] fabfur https://i.postimg.cc/dVtmyqNM/Screenshot-2026-07-31-025848.png [10:07:12] ok, thanks [10:07:36] replicating your test, it gave me the same result as ping, meaning ~30ms from my location [10:07:57] so unfortunately this confirms what topranks already stated [10:08:56] topranks I completely understand what you're saying, and you're absolutely right: they are large commercial companies and their expertise lies in this field. But at least, please increase the speed of your servers in Europe, the Middle East, and Asia, especially Japan, South Korea, and Australia, and definitely reduce the latency significantly. [10:09:27] MG2021: what you could try just to see if there is discrepency is an MTR from your location [10:09:45] mtr -b -w -z -c 10 wikimedia-dns.org [10:10:26] fabfur thanks for the test: where do you live, and tested it? EU, Asia, middle or US? [10:10:38] EU [10:11:02] MG2021: I'm not sure there is any issue with the servers or their response times. You are likely hitting some of our servers in our European POPs in Amsterdam or Marseille, and the distance is what it is [10:11:09] I tested from EU mainly [10:12:21] topranks  Amsterdam is so terrible, especially for insta, yotube, whatsapp voice/video calls [10:12:50] fabfur thank you so much Fab [10:15:25] topranks Most of the articles I found on Google about Wikimedia DNS all backlashed that Wikimedia DNS connects to the Amsterdam server in most parts of the world where it has been tested, which has a speed of about 200 ms! [10:16:15] we anycast the IP from all our POPs, so you will hit the location that your ISP sees as closest to you [10:17:48] topranks I know: Could you please tell me the countries of all the Wiki DNS servers around the world? [10:19:26] Ashburn (VA, USA), Dallas (TX, USA), San Francisco (CA, USA), Sao Paulo (Brazil), Amsterdam (NL), Marseille (FR) and Singapore [10:26:54] topranks These locations are really very few and do not meet the needs of all countries in the world, especially in Asia, the Middle East, and Oceania. And even if they do meet the needs, if Wiki DNS becomes very popular, it severely throttles. For someone living in Australia, for example, the small number of servers and the great distance cause a [10:26:54] lot of delay, and it becomes terrible, especially in: livestreaming/video/voice calls! In terms of online gaming, it becomes very terrible. [10:30:34] the latency to your dns should not affect livestreaming/video/voice calls [10:31:00] obviously there is a latency hit if you are a distance from our locations, but as I said it costs a lot of money to set up locations [10:31:27] we do not have the required hundreds of millions of dollars to expand the service as you suggest, unfortunately [10:33:10] MG2021: where are you located? [10:33:24] topranks For example, how much would it cost you if you added a server in Japan? [10:34:00] XioNoX France [10:34:50] ok, and what's your ISP? I'm also in France and I have 25ms latency [10:35:49] MG2021: please run the mtr command I posted above if you can, those results are very bad for France [10:36:01] XioNoX Orange [10:37:16] I don't know why when I use Wiki DNS, sometimes but not always, it lags a lot or freezes. [10:40:44] testing https://dnsspeedtester.com it shows me 34ms latency for Wiki-dns using Free [10:42:42] XioNoX thank you so much for the test: what is the lowest-latency resolver in your location [10:43:15] topranks mtr -b -w -z -c 10 wikimedia-dns.org [10:43:15] 'mtr' is not recognized as an internal or external command, [10:43:16] operable program or batch file. [10:43:43] nextdns is 19ms, cloudflare at 21 [10:43:59] quad9 at 18ms [10:48:12] 4ms to Cloudflare here [10:48:16] and all we had to do is sell the country to big tech [10:49:58] guys [10:50:10] what about control d's unfiltered and gcore [10:50:32] gcore is f* fast in eu [10:50:42] is so* [10:51:17] we usually don't use cloud services [10:51:45] we prefer to have control on wikimedia servers, especially the ones that serves sensitive content (like DoH) [10:51:58] for our user's privacy [10:53:40] sacrificing speed, stability, and latency over privacy [11:06:52] that test site is funny, it reports 10ms latency when I try cloudflare, but a dig is consistently 3-4ms [12:16:52] 10netops, 06Infrastructure-Foundations, 06SRE: JunOS: Investigate BGP PIC Edge / Protection - https://phabricator.wikimedia.org/T432381#12174142 (10cmooney) >>! In T432381#12131427, @ayounsi wrote: > Another optimization to consider: https://www.juniper.net/documentation/us/en/software/junos/bgp/topics/topic... [13:40:38] 10netops, 06Traffic, 10Cloud-VPS, 06collaboration-services, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12174538 (10brouberol) [13:40:41] 06Traffic, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack B8 maintenance - Thursday July 23rd 14:00 UTC - https://phabricator.wikimedia.org/T430929#12174540 (10brouberol) [13:40:57] 10netops, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 5 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12174539 (10brouberol) [17:48:09] 06Traffic, 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T433641#12175335 (10bd808) 05Open→03Invalid `lang=shell-session bd808@deployment-cache-text08.deployment-prep.eqiad1:~$ w 17:46:44 up 15 days, 40 min, 2 users... [18:55:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12175477 (10Papaul)