[07:12:15] It shouldn't be hard to improve that [12:36:38] i'm going to create a python script that does it [12:36:53] i've mocked up a php script https://github.com/miraheze/MirahezeMagic/pull/441/file [12:37:43] Let me review then [12:38:55] The problem with the lua one is it collects the list of keys for every type [12:39:07] Rather than * and just count based on name [13:53:07] hmm it's slow in python but faster then using it in the exporter [14:01:04] looks like things are much faster if used as a mediawiki script (as php on its own is the same speed as the python script) [14:02:08] oh nvm [14:17:11] ok going with python [14:22:12] @rhinosf1 [14:22:13] https://github.com/miraheze/puppet/pull/3499 [14:28:58] @paladox lgtm [14:34:49] hmm [14:34:56] i don't know how to get the password to work [14:35:07] because puppet escapes the string so does \' \' [14:35:33] @paladox what is puppet passing then? [14:35:43] With a fake bit in the password [14:36:13] it's doing $, &, @ and # [14:36:25] not sure if the later two are cuasing the issue but the former is definitely [14:37:27] Hmm [14:38:29] So puppet is modifying what it passes to --password [14:51:53] i guess for now i'll have to set the password in the script using erb [14:52:01] i don't like it but for now it'll do [16:11:38] metrics are working again @rhinosf1 [16:12:12] although there's one oddity in https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki?orgId=1&viewPanel=48 [16:12:53] includes * in the graph [16:13:47] Can we exclude it grafana side @paladox ? [16:14:01] * is only for the total graphs [16:14:07] i'm not sure [16:26:19] fixe dit [18:44:49] @rhinosf1 parsoid jobs have been re-enabled for a-c [18:44:59] @paladox yey! [18:45:11] @paladox have you updated upstream task with cause? [18:45:25] _thinks it's a very interesting case_ [18:45:29] Not yet, i carn't say it's fully resolved until we wait a while [18:45:33] Okay [18:45:46] @paladox was a task ever created for topranks' work? [18:45:54] i expect spikes, but i expect it to go down quickly. [18:45:57] No [18:46:01] Yes it will spike [18:46:48] he's disabled https://bgp.he.net/AS25160 their side or what ever he's changed it's worked. [18:46:53] that may be our isp [18:46:55] dunno [18:47:06] It'll be something in the middle [18:51:11] AS1299 is a tier 1 [18:51:29] Without seeing the mtr, I can't say much though [20:01:18] @paladox what's going on with refreshLinks on rainworldwiki [20:01:35] If it wasn't for that, it would have a near 0 backlog [20:01:38] it keeps going down and then backup [20:01:42] There's no issues with pre warm [20:02:02] @paladox ye I can see, why [20:02:13] i'm not sure [20:03:02] there's a lot of recent changes on that wiki [20:03:07] Ye weird [20:03:22] But prewarm is looking like a good start @paladox [20:04:08] @paladox id go more tbh, maybe another 5 letters [20:04:16] ok [20:05:59] done [20:07:21] @paladox 3/5 trouble wikis are already on [20:07:30] ok [20:12:43] @paladox let's give that another half an hour [20:12:49] Then I'd go up to 50% [20:28:15] @rhinosf1 wondering if you could help get https://gerrit.wikimedia.org/r/c/mediawiki/services/poolcounter/+/971544 and https://gerrit.wikimedia.org/r/c/mediawiki/core/+/972724 merged pls [20:28:27] we're using both in prod. [20:30:05] @paladox you added PoolCounter? [20:30:11] yeh [20:30:42] @paladox awesome [20:30:49] Since when? [20:31:03] days ago [20:31:38] we first tried the redis one but that would hit the issues with redis slowness so i went about adding ipv6 support and deployed the software [20:32:15] @paladox why add WikiLambdaFunctionCall [20:32:21] We don't have wikilambda [20:32:24] oh [20:32:27] i'll remove it then [20:38:40] @paladox parsoid still looks fine [20:40:41] @paladox I do wonder what l will be like [20:40:47] Because global user pages [20:40:55] hmm? [20:41:26] @paladox you could easily generate a couple of thousand jobs in a few seconds [20:42:31] I'm going to test it [20:45:52] That did something insane [20:46:46] I will check on it in 10 minutes [20:52:37] It looks to be coping relatively fine [20:52:55] Hey @originalauthority [20:53:22] Hola [20:53:31] @originalauthority you were typing? [20:53:45] Unintended [20:53:54] I was reading the past convo [20:53:55] @originalauthority ah [20:57:15] @paladox I guess I need to fix wikimedia being down causing issues with instant commons again [20:57:22] It's taken us down before [20:59:49] @paladox https://grafana.miraheze.org/alerting/grafana/-bk-DCynz/view?returnTo=https%3A%2F%2Fgrafana.miraheze.org%2Fd%2FGtxbP1Xnk%2Fmediawiki%3ForgId%3D1%26from%3Dnow-30m%26to%3Dnow needs fixing [21:00:10] oh right [21:01:12] done [21:04:32] @paladox why vanished from https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki?orgId=1&from=now-30m&to=now&viewPanel=43 [21:04:47] have no idea [21:05:01] all i did was changed a value and click save and exit [21:05:13] @paladox weird [21:05:29] it's back [21:05:43] It's back ye [21:09:11] @paladox I would be inclined to go all in once we drop below 2.5k jobs [21:09:53] We're currently at 8.2k and going down [21:10:01] Because I simulated a spike [21:10:15] Pool counter is having an impact [21:10:41] Or at least I suspect pool counter [21:11:20] Because something is causing jobs of 1 type to be processed no more than some number at a time [21:11:39] Looks like ~200 [21:18:18] @paladox back in a bit [21:18:28] @pixldev I think we fixed the caching issues [21:24:05] 👏 [21:24:15] absolutely awesome [21:26:25] we shall add this as emoji or sticker lol [21:34:46] I think this has to be my favourite reason for the job queue breaking in all its failures [21:36:48] I'm back too [21:38:17] @paladox I see no reason why not to do all [21:38:24] That had utterly zero impact whatsoever [21:38:32] On prewarm [21:40:40] @paladox why not commonswiki? [21:40:55] because i see wikimedia disable it on their commons wiki [21:47:42] @paladox I can't find the current config searching github so [21:47:52] you won't [21:47:55] the file is too big [21:48:00] so won't be indexed [21:48:05] it's in InitialSettings [21:49:34] @paladox what line? [21:50:08] I found it [21:50:16] So they have wikidata & commons off [21:50:27] I mean I can understand wikidata, it doesn't use VE [21:50:43] Commons, maybe the same [21:51:36] Hopefully this works [21:51:49] Should be a performance boost for VE users [21:53:09] @paladox what's going on with LocalGlobalUserPageCacheUpdate ? It's holding at 236 jobs [21:53:24] i'm not sure [21:53:43] Ye I'm not [21:53:47] It's weird [21:53:59] Can you see what wikis it's on @paladox