[08:12:26] <akosiaris>	 so, just to spell it out loud clearly, we had a bytedance scraper scraping wikidata and the responses were so fast that the p75 dropped?
[08:12:40] <akosiaris>	 oh wow, https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red?orgId=1&from=1673988329965&to=1674074729965&viewPanel=18
[08:12:46] <akosiaris>	 800rps static? 
[08:15:30] <akosiaris>	 why did all those static requests even reach the appservers though
[08:17:42] <akosiaris>	 and 1.5krps resulting in 303s
[08:17:45] <akosiaris>	 lol
[08:33:58] <diskdance[m]>	 is it bytedance the company? why they are scraping wikidata content?
[08:36:39] <_joe_>	 akosiaris: it wasn't all static requests though, there was a ton of requests for wikidata entitites
[08:36:49] <_joe_>	 I suspect the scraper was downloading static assets
[08:37:08] <_joe_>	 diskdance[m]: I guess you should ask them :)
[08:39:44] <_joe_>	 akosiaris: oh sigh
[08:40:21] <_joe_>	 so the requests were for http://www.wikidata.org/entity/QXXX, which 303s statically to https://www.wikidata.org/wiki/QXXX
[09:01:39] <claime>	 Amir1: Reminder that I'll be rebooting mwmaint1002 in an hour
[09:01:51] <Amir1>	 noted thanks
[09:05:14] <akosiaris>	 yup, that's the 303s
[09:05:17] <akosiaris>	 See Other
[09:05:33] <akosiaris>	 so, around 2.5krps was either static or 303s
[09:06:06] <akosiaris>	 and those are so fast that latency aggregations were calculated in lower levels
[09:06:19] <akosiaris>	 for some reason, I can't stop laughing thinking about it
[09:13:55] <diskdance[m]>	 so the latency was incorrectly calculated? 🤔
[09:15:27] <claime>	 Well it's technically correct
[09:15:30] <claime>	 Which is the best kind
[09:15:51] <claime>	 It just tells us we answered very fast "Go see somewhere else" lol
[09:26:11] <akosiaris>	 yup, exactly that ^ 
[09:26:23] <akosiaris>	 we could skip calculating on some things ofc
[09:26:34] <akosiaris>	 like 3xx and static content 
[09:27:30] <akosiaris>	 but we need to have a good reason/incentive (e.g. a proper SLO for an endpoint) to do that. Otherwise we will be calculating something more specific that noone will care much about
[09:28:20] <claime>	 The status code graph is hilarious
[09:28:37] <claime>	 No 303... 303 WALL
[09:31:14] <_joe_>	 we do have the data by endpoint btw
[09:31:19] <_joe_>	 and by handler
[09:31:32] <_joe_>	 we just don't use those in the graphs, maybe we should add that
[09:43:34] <_joe_>	 I have to read a wiki page, parse its content, and make a small edit from a python program. The wikitext is extremely simple. Is there a lighter alternative to piwikibot to do that?
[09:44:04] <_joe_>	 apart from doing everything myself I mean
[09:46:05] <taavi>	 mwclient (https://mwclient.readthedocs.io/en/latest/) is somewhat lighter than pywikibot
[09:51:02] <_joe_>	 taavi: ah right, I even used it once
[09:51:07] <_joe_>	 thanks, good suggestion :)
[09:51:55] <_joe_>	 why is writing software that needs to interact with humans so much less of a pleasure to write than software that interacts with other software and at most some user input
[09:53:25] <claime>	 Because humans don't have an API
[09:54:10] <_joe_>	 right the devs really messed up there
[09:54:53] <_joe_>	 and we've been waiting for the 2.0 version for too long now. The New Man is basically Duke Nukem forever
[14:08:51] <lmata>	 if anyone in the americas is able to volunteer for an on-call shift today please reach out to me before 11 am ET / 16:00 UTC. many thanks in advanced!
[15:01:59] <lmata>	 we have a volunteer, thanks jhathaway 
[19:53:00] <gehel>	 Hey! Is there anyone around who understands enough about reprepro to review: https://gerrit.wikimedia.org/r/c/operations/puppet/+/881710 ?
[19:53:10] <gehel>	 moritzm: ^ if you're still around
[19:55:34] <moritzm>	 let me have a quick look
[20:49:24] <inflatador>	 interestingly, there is a /srv/wikimedia/dists/bullseye-wikimedia/component/elastic710 with an August timestamp, but no component/elastic710 on apt-browser web UI