[07:36:15] hnowlan: since about 15:36 yesterday, the 5xx rate from thumbor has tripled. Is there a known issue? [07:36:35] hnowlan: [I noticed because the resulting rise in 5xx turns up in the swift monitoring too] [07:49:24] any idea on how can I delete a dashboard (https://grafana-rw.wikimedia.org/d/pr6ZUm5nz/haproxy-cluster-view?from=now-6h&to=now&orgId=1) adding a redirection to another one? [07:51:59] One way I have used in other orgs in the past was to rename it with something like Deprecated in the name and adding a Text/Markdown widget with a link at the top. [07:52:25] The upside of that is that people will see the place to go to has changed, and they can hopefully fix the link they clicked [08:18:06] vgutierrez: yeah what klausman said [08:18:13] godog: cheers [08:18:16] sure! [09:56:40] claime: can you help me out with assessing impact of parsoid parsing on jobrunners? I want to deploy an experiment soon, and I'm not sure it's safe... https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/932175 [09:58:09] duesen: That would mean parsoidCachePrewarm becomes time-critical, right? [10:08:34] claime: not in the sense that anything would fail if it is slow. If it's slow, this just means we'd parse twice. [10:08:59] This is intended to be temporary - the idea is to see how it affects the load on jobrunners [10:09:26] Eventually, we'd want to have cache writes enabled in the endpoints, but restbase would no longer hit them for every edit. [10:10:13] From the perspective of parsoidCachePrewarm, this amounts to the same: the new version generally isn't in the cache when the job runes. [10:11:32] claime: I want the change to be big enough to have a visible impact, but not so big that it might cause serious disruption. Doing it on a small wiki instead of enwiki would be safe, but useless. I wouldn't be able to see the impact. [10:11:45] But we could do 50% of enwiki, or we could do frwiki or whatever first... [10:11:57] ...or we could just turn it on for everythign at once. [10:12:36] I was hoping you could tell me what that the biggest step would be that is still safe to just revert after 20 minutes of the load gets too high [10:18:06] Right. If we parse twice, the second parsing would be a direct call to parsoid right ? [10:19:07] Or if not direct, it would be a parsoid host doing the second parse, not a jobrunner [10:25:52] Is there a calendar event for the fortnightly puppet office hours? [10:28:16] jbond: ^^^ [10:30:45] (why yes, I do have more Stupid Puppet Questions, why do you ask? :) ) [10:44:12] duesen: since we know which wikis are the big ones [10:44:30] I would suggest de,fr, check how things are, and then go for en [10:53:54] Emperor: could you try this https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=MnNyZ2d1ZnA5anViZ3JxZzZzNDk2OXVvMm9fMjAyMzA2MjlUMTQwMDAwWiBqYm9uZEB3aWtpbWVkaWEub3Jn&tmsrc=jbond%40wikimedia.org&scp=ALL [11:00:29] duesen: I agree with e.ffie, let's do (de+fr), then en if everything goes well [11:05:43] duesen: what time would like to have a go? [11:06:44] in 2-3h from now? [11:40:02] fabfur: there is a puppet-merge pending with your name on it: `Added new bullseye instance for cache-upload in deployment-prep` - Would you like me to merge it? [11:40:28] btullis: sorry I'll do immediately [11:40:47] No problem. Feel free to merge mine too, if it asks. [11:40:53] ack [11:41:36] done, thanks [11:41:43] 👍 [12:25:47] jbond: looks to have worked thanks [12:26:09] (might be worth trying to get it into the WMF staff calendar, though?) [12:34:46] Emperor: i agree; jobo is that (getting the above calender event into the staff calender) be something you know how to do? [12:48:05] let me see [12:54:55] thanks :) [13:16:19] taavi: o/ (if you have time) - I'd need to figure out where https://recommend.wmflabs.org/ is served from, I see that it resolves to a wmfcloud proxy and I suspect it belongs to a horizon project, and I am wondering if there is a quick way to check (super ignorant about it in case) [13:17:36] (if anybody else knows lemme know!) [13:17:53] elukey: https://openstack-browser.toolforge.org/proxy/, it's a bit slow to load but should show you that information eventually [13:19:05] nice thanks! [13:27:27] taavi: found it, thanks a lot! [14:23:47] urandom: interesting turn of events! [14:24:55] sukhe: I'm lucky that way ☘️ [14:25:47] :P [15:29:19] effie. claime: sorry I was busy, I guess we have to punt it to Monday. [15:55:26] hnowlan: dunno if you just did something to thumbor, but error rate has dropped back to good levels since ~15:20 UTC [19:33:00] urandom: I think that adapter thing itself is suspicious. you guys replaced cable, SPF, switch port.. but not that adpater thing. and it might make sense that you have light on server side but not on switch side if the adapter in between is broken [19:33:17] RJ45->SPF she said [19:33:25] SFP [19:42:05] mutante: isn't the SFP and adapter the same thing in this context? [19:42:52] did not sound like it to me. might be wrong [19:42:58] I thought that was a way of supporting arbitrary interface types on the switch (fiber, rj45, etc) [19:44:05] Ok, I'll double check this with X.ioNoX [20:33:11] " Elon Musk and Mark Zuckerberg agree to hold cage fight " [22:57:50] mutante: Who do you think is going to win?? [22:58:40] denisse: Zucky, it is rumoured he does Jiu-Jitsu [22:59:35] May 7 2023 - Mark Zuckerberg competed and won his first jiu-jitsu tournament [23:09:04] mutante: haha, let's hope they won't hurt each other badly. [23:09:17] 39°C here btw... [23:10:03] oh my, get paletas [23:12:58] "Paletas de hielo" (popsicle)? :P [23:13:12] Other "paletas" are just candy. [23:14:55] yea, I meant the frozen ones