[03:33:08] Was about the report it as well earlier today but wiki'ing delayed me. 😭 [09:21:17] Aww shucks, I guess we can't host your HUGE wiki. [12:31:27] plot twist: shadowbanned again (WHY??????) [12:31:37] and of course no email of reason [14:42:01] how does one get smited from github [14:43:58] the first time, the cause was clear because the name of the organization I belonged to was rewritten into a severely offensive insult, resulting in all members of the organization being shadowbanned (btw I had never committed to that org's repository). however, this time, I had already left that org, so I have no idea what the cause is [14:45:06] insane [18:08:23] Mmmm, Miraheze is dfying. [18:10:14] huh? [18:10:40] Well, the DOORS Wiki was loading fine, and now it takes like a minute or more to load pages. [18:10:58] Probably this. [18:11:14] Yeah, that's what I was thinking. [18:11:40] didn't 161 have issues a week or so ago? [18:12:04] 161 has a lot of issues for some reason. It's almost always the cause of outages these days. [18:12:29] when that happens it's usually 1 of 3 wikis [18:13:11] are wiki contents not evenly distributed across the dbs? [18:13:19] We probably need to rebalance wikis. [18:13:20] I think it's just we have too many of the resource-heavy wikis on that one cluster [18:14:01] but like e.g. if we got rid of commentstreams then that might rebalance the wikis enough [18:14:43] I meant we need to move some of the very large wikis on db161 to other clusters, because I think we have a lot of them on just that server. [18:15:04] I would have assumed that wikis would have files found across multiple dbs to spread the load more evenly [18:15:39] CommentStreams might help the issue until something else causes it for the same type of issue. [18:15:53] I think we just need to move some wikis to another cluster entirely. [18:16:02] It's been a long time since we rebalanced. [18:16:28] if this isn't the case, why not? [18:20:08] No idea what you mean. [18:20:59] the implication of there being several resource heavy wikis on db161 is that the entire content of a wiki is stored in a single database [18:21:31] but unless I'm missing something, storing the content across several dbs would spread the load much better [18:23:22] 1 wiki = 1 database (or 2 if it has cargo). we have db clusters with about thousands of databases on each. but it seems like all the most resource-heavy wikis other than Fisch are all on db161 [18:25:14] okay, so one db per wiki makes sense as something analagous to namespacing, but why would the underlying have to all be on the same cluster? [18:25:43] They don't they just ended up there. [18:26:07] That's what rebalancing would fix we could move some and spread it more evenly across clusters. [18:26:38] how? [18:27:30] CreateWiki makes wikis on the server with the least amount of databases by default, by coincidence, db161 had the least amount of databases when the most intensive wikis were first created and imported. [18:30:03] what I was trying to get at was more of if there was a reason why you couldn't have ~half of the db in one cluster and half in another (the specific distribution is unimportant) so that dbs that are queried frequently don't concentrate all of the load onto one or a few physical machines [18:31:15] For complex queries it'll involve cross-DB operations like joins and we use MariaDB which isn't distributed out of the box. [18:32:35] It'll also mean that slow operations will bring every database down instead of just one. [18:33:43] what are the slow operations? [18:35:24] Fetching a page with lots of CommentStreams operations [18:36:33] You could only split a wiki across multiple databases for those stuff that support virtual domains and we're confident it's fully developed and will stay [18:37:16] stuff being wikis, dbs, or extensions? [18:37:32] Query patterns more like [18:37:44] You can't do cross database joins [18:38:03] Some stuff in mediawiki support having it on a separate database to the main DB [18:38:14] And some extensions have separate db config to the main [18:38:53] It has to be carefully thought out to ensure you don't do cross DB joins which is impossible and you have to consider the risk of duplicating data which would need to be replicated across 2 clusters [18:39:05] It's a lot of effort and more complex than just splitting tables up [18:39:30] And would require more DBs spread across more hypervisors which costs money too [18:40:11] I see [23:41:52] I was talking to @cosmicalpha in a support thread about a list of wikis permanently enrolled in NextTide, and https://meta.miraheze.org/wiki/NextTide has the current version of that idea. [23:42:05] [1/2] Also we should look into these 2 db161 spikes. [23:42:06] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1552827497228533911/image.png?ex=6ab706cd&is=6ab5b54d&hm=697cb6f8db1981e4a8ea0dc4b4464fa0dcc7b4555ad46f4487f4a8b30412d0a7& [23:44:36] do we have a proclist [23:46:02] [1/2] here's parsoidcacheprewarm in yellow [23:46:02] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1552828489009205268/image.png?ex=6ab707b9&is=6ab5b639&hm=ae55b8d148f9487e8e0b7687832c83e86f2d034112c06160f9747515cbdaae98& [23:46:20] We have auto saved proclists whenever there is high load