[08:34:02] Hello, since weekend API requests to wikidata servers are extremely slow. Is this a known problem? [09:02:28] Hello, since weekend API requests to wikidata servers are extremely slow. Is this a known problem? [09:04:19] Is anyone active in this channel right now? [09:06:47] What sort of api requests [09:06:58] arch2all: ^ [09:07:28] only reading requests [09:07:38] arch2all: and what does slow mean [09:07:42] Do you have any numbers [09:08:43] about tne or more times slower than normal [09:08:54] tne ->ten [09:09:41] arch2all: and do you know when it started [09:10:06] If you have an example api query, that would be extremely good [09:10:32] I maintain an architecture database which makes extensive use of wikidata/commons/wikipedia material and observe last days lot of timeouts and other problems, because of slow api requests [09:10:39] Lucas_WMDE: hi, fyi we've got a user reporting some api queries being about 10x slower than normal [09:11:05] arch2all: if you can tell me exactly when it started and an example of some queries, that would be great. [09:11:20] I want to be able to see if we can narrow down a patch that might have caused it [09:11:43] I noticed it first on saturday [09:12:15] And what sort of api query [09:12:23] Do you have an example of what you're running [09:14:09] i send some example api queries soon, but it occurs on all queries which our server sends. Is there maybe some (new) throttling active for frequent users? [09:14:24] There shouldn't be [09:14:34] And definitely not on a Saturday [09:15:24] IP of our server is 62.67.209.19 [09:15:28] are you sending a good user agent with your request? I think I remember some changes related to that recently [09:15:56] wait a moment I will check the script [09:17:17] User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1) Gecko/20021204 [09:19:48] hm [09:20:07] I doubt it’s related but you should still send a better user-agent according to https://meta.wikimedia.org/wiki/User-Agent_policy [09:20:13] that one just looks like a general browser UA [09:20:49] otherwise, examples of the requests would still be useful [09:21:12] Lucas_WMDE: can you see anything in logstash for the ip [09:23:17] ok, I will update the user-agent to the mentioned guidelines [09:23:34] rhinosf1: nothing out of the ordinary afaict [09:24:07] ty [09:25:25] example query: https://www.wikidata.org/w/api.php?format=json&action=wbgetentities&props=claims&ids=Q110774154 [09:25:55] That loads near instantly for me [09:26:51] arch2all: is this a consistent issue? [09:26:57] Or only timing out some of the time [09:27:45] Can I also ask if you know what data centre you hit? And if not, where in the world you are [09:29:41] If I invoke the queries from my local browser they are very fast, too. Only from server these problems occur. That the reason, why I asked for eventually IP-throttling [09:30:29] Ok [09:30:53] But maybe it's an other problem on our server, which causes this.... [09:30:59] arch2all: can you please ping wikidata.org from both your local pc and that server [09:31:07] And share the output of both [09:33:24] ping from our server to wikidata.org delivers good result times [09:33:56] What ip do you hit [09:34:04] 64 bytes from fra24s06-in-f3.1e100.net (142.250.186.99): icmp_seq=1 ttl=60 time=20.5 ms [09:34:44] oops, sorry that was a ping to google.de (for comparison) [09:34:50] Ye [09:34:58] Thought that wasn't WMF [09:35:05] 64 bytes from text-lb.esams.wikimedia.org (91.198.174.192): icmp_seq=18 ttl=58 time=19.3 ms [09:35:14] Hmm [09:35:31] Does your laptop hit the same? [09:37:47] arch2all: ? [09:41:48] sorry, connection issues [09:42:23] arch2all59: no problem, is your laptop hitting the same ip? [09:42:24] Laptop runs windows and i tried ping in console window, but I don't get a result. Do I need in Windows any further parameters? [09:43:23] Oh lovely windows ban ICMP requests [09:44:09] arch2all59: would you mind if we move this to a private task? I'm not really sure what could cause it and SRE have said they aren't doing any throttling [09:46:07] Ok, seems that the problem is located on our site. I try to investigate the reason. [09:47:00] Thanks for the help [09:54:35] No problem [09:54:41] Let us know if you need anything [09:54:50] I'll be around most of the EU day [10:01:13] Great, if I need further help/information I will check back here [10:01:45] Is IRC the best/recommended way for things like this [10:03:23] Irc is fine [10:11:04] arch2all59: one of our SREs thinks envoy is at fault [10:11:06] https://phabricator.wikimedia.org/T300366#7677828 [10:11:38] legoktm: does your friend have a workaround [10:19:16] that is interesting. I had similar issues with http/1.0 and posted this already in the Commons forum [10:19:19] https://commons.wikimedia.org/wiki/Commons:Forum/Archiv/2022/January#Commons_Server_liefert_nicht_mehr_in_http/1.0_aus_? [10:22:32] hm, giftzwerg’s second reply sounds like {{citation needed}} material to me tbh [10:22:33] 10[1] 10https://www.wikidata.org/wiki/Template:citation_needed [10:22:41] uh, ok, thanks bot [10:22:50] anyways I’m not sure which protocols or what point in time is being referred to there [10:34:41] Lucas_WMDE: http 1.0 works via most servers [10:35:02] as far as I know, exactly 1 cache proxy runs envoy and it's being fixed upstream [10:35:06] arch2all59: ^ [10:36:00] 1 cache proxy per dc/cluster [10:36:01] so 8 [10:37:23] https://phabricator.wikimedia.org/T271421 is the list [10:54:59] It is indeed related to the issues with envoy mentioned on phabricator. I replaced our URL loading module in PHP with a different one, that uses CURL routines and now it works! [11:05:35] great! [11:05:59] yay [17:19:44] RhinosF1: use something that supports HTTP/1.1 ? [23:04:55] @gry done. Actually I was already here. Also the "/msg alis list wiki" return good list of potential channels. [23:06:19] hello :-)