[02:26:01] @originalauthority so any luck with clean up https://gitlab.com/telepedia/garda? [11:46:53] heyo, anyone familiar with older mediawiki, and know if the old_text column in the text table is needed? [12:09:51] https://meta.mirabeta.org/w/rest.php/createwiki/v0/wiki_request/15 looking good [12:10:18] the API also now allows querying of suppressed wiki requests, if you have the appropriate user right that is [12:10:27] https://github.com/miraheze/CreateWiki/commit/9d0dcca93e109bdbf9a663f354ed28f2b44e80d8 [12:18:04] oops, wrong timestamp format [13:13:55] [1/2] @bluemoon0332 it appears createwiki is not working properly. I approved the following request (twice) and it has not created the wiki. Obviously, it also did not attach the requestor as bureau. [13:13:56] [2/2] https://meta.miraheze.org/wiki/Special:RequestWikiQueue/42604 [13:14:36] also @Stewards ^ [13:15:11] this is probably the jobqueue, since I've not upgraded CW on prod [13:15:34] I guess it is not playing well with Kafka [13:15:46] Why is it when shit breaks it is always either the Job Queue or CentralAuth [13:15:48] Always [14:37:54] This seems to be a theme [14:42:23] @jph2 the job actually was running, it was just taking a long time to be executed, see the comments of that wiki request [14:43:25] “NOTE/b: A wiki with this subdomain already exists.” [14:43:32] Is the markup meant to show like that? [14:43:44] I'd guess not [14:43:59] It's good that doesn't work though, if it did, there would be a XSS vuln [14:44:07] I mean not letting the boxes accept markup makes sense [14:44:10] Yeah was gonna say [15:01:42] letting it accept actual markup would be nice as well as a larger box/interface so messages have breathing space [15:01:52] naturally it should not take raw html though [15:10:45] Maybe [[w:Markdown]]? [15:10:46] [15:11:36] wikitext is what I mean specifically, conflated the two [15:16:24] Oh right i forgot we have our own wiki markup language that supports bold and stuff [15:16:28] 💀 [15:16:48] My intelligence Is infinite [15:57:40] I should have been more specific tbh, I just think about markdown rather often even though that's just not what you call it here [16:10:41] @bluemoon0332 job queue still acting up/very slow or do you reckon it will be fine if someone does a wave of approvals now? [16:31:44] there seem to still be some shenanigans with CreateWikiJob [16:35:08] Grafana says there have been instances of CreateWikiJob taking over 10 minutes to execute [16:35:38] and an instance of smw.changePropagationClassUpdates that took hours [16:37:01] processing rate is 0.0 which makes me think jobs aren't even running right now [16:40:05] think this would warrant a createwiki pause? [16:40:12] @cosmicalpha [16:40:31] There is some weirdness on at least 3 wikis [16:40:51] That metric isn't active, jobs are running. 30,000 ran since this morning. [16:41:39] hmm [16:42:39] I'd say business as usual for now, if it gets too bad I can consider disabling the CreateWikiJob (though that's last resoty) [16:42:47] I will look though. [16:42:51] https://meta.miraheze.org/wiki/Special:RequestWikiQueue/42673 this taking a while [16:43:55] which ones? [16:44:25] Not meta [16:44:49] 3 hours is not good [16:45:05] It's probably some config needing tweaking around how many get done at once [16:45:13] Cause Kafka has concurrency rules [16:45:58] [1/2] Oh... I need to increase changeprop resources [16:45:58] [2/2] also has concurrency settings [16:49:09] I'm going to increase RAM to 6GB and cores to 12 on changeprop so we have more threads [16:49:34] the wonders of having the resources to accommodate that [17:05:25] Wowza [17:09:25] has this been done? [17:09:29] the jobqueue is so slow [17:10:20] Honest question why did we move to Kaftka [17:12:43] the old Redis jobrunner system is quite inefficient (read: 'dumb') and not really designed for MediaWiki's purposes [17:12:47] so it was very easy to overload it and have jobs be lost [17:13:26] Kafka is much more efficient, handles things basically on-demand, and is fine-tuned for MediaWiki purposes [17:14:11] but the trade-off is that it needs more resources and if not given enough, it's very slowwww [17:31:00] Kafka will be much better in the long run but it might take some tweaking and getting used to. It's very different. [17:33:38] I'm working on it now [17:47:33] Wait so it’s more efficient by a factor of 10 by needs vastly more resources? [17:59:49] Its not only more efficient but more reliable. Redis does not guarantee things are received Kafka mostly does (if we configure it right) [17:59:59] CPU utilization is much better but it's quite the memory hog, IIRC. [18:00:08] Correct [18:00:54] Kafka will also have a lot more benefits to allow us to support other services in the future also and if we are using anyway, why not make it work for job queue? [18:04:13] @agentisai FYI I increased cores on changeprop to 12 and sockets is still 2 so we have 24 threads running there now, we have 27 queues so almost one thread for each queue, with a few tweaking to concurrency it should be pretty good now. I also tweaked some Kafka config as well. [18:04:21] I wouldn't necessarily think Kafka was the best choice. Something lile RabbitMQ would've been a better choice. [18:04:32] But alas we continue the legacy of copying wmf. [18:06:47] [1/2] That won't help us support other services I want to for monitoring, etc... also. Like Navigation timings, and also Kask for session eventually I want to look into (not certain just a consideration). Kafka is a dependency of quite a bit of stuff, so if we have the resources to support it we will, otherwise we'd still end up with it eventually then we [18:06:48] [2/2] have more work with even more services. Kafka isn't a WMF-only service, its used in a lot of organizations. [18:09:44] [1/2] RabbitMQ supports a wide variety of monitoring, so I'm not sure what you mean there. [18:09:45] [2/2] I'm aware that it isn't a WMF only thing. Dependency for what exactly? [18:10:31] Mediawiki extensions for monitoring [18:10:58] Like? [18:17:08] [1/3] I said it above, Navigation Timings, also EventBus only supports Kafka thus RabbitMQ wouldn't work for jobs unless we made a new extension is would only be more work than Kafka also. [18:17:09] [2/3] Also, RabbitMQ for example is good if the queues are empty but when there is a high volume, Kafka is better at ensuring things are received for large data RabbitMQ is not designed for large scale queues, so at some point if we get even more jobs then now, it would not continue to be an option and we'd need Kafka anyway which would then be double the work. Using RabbitMQ would not [18:17:09] [3/3] necessarily be good indefinitely, maybe better in short term but not long term. Kafka is also more scalable and easier to launch redundancy in case one goes down we could setup another node on another cloud server for example to easily switch and provide more scalability also. [18:18:47] When researching Kafka prior to doing all this work to use it, I did run accross RabbitMQ and even looked into it quite a bit, but from everything I saw Kafka is better for what we want. [19:55:21] Is this still processing? the wiki doesn't exist yet. or should I approve it again? [19:55:34] no need to approve again [19:55:51] all that does is queue another job, which isn't all that useful if jobs are not really working [20:18:11] Please do try again if it isn't yet, as it means the job isn't queued. A wiki that was created long after that has already been processed and everything ran within 30 seconds. (including main page creation, etc...) [20:21:43] We have 3x more jobs ran after my recent tuning of kafka and changeprop in the past 2 hours then we had in the past 12 hours before that so seems to be working better now. [20:23:21] I just approved it again. I also approved 42270 over 5 minutes ago and no wiki yet. [20:23:35] I see the job was submitted this time why didn't it create... [20:23:53] Shenanigans [20:23:54] Ope. Disregard. 42270 just showed up. Just slow I guess. [20:24:34] it did now [20:26:26] Whats odd is one created 1 minute ago submitted at the same time as one 5 minutes ago, that isn't slow it seems it is only pushing them in bulk even though for CW it should be immediately one at a time. [20:28:36] 🤷‍♂️