[00:06:07] Point of order, is it wise to create wikis when file system issues are ongoing? Slightly worried that might hit container assignment issues given lack of space... [00:08:10] Thats why i said it was unexpected but if sre says go for it then 🤷‍♂️ [00:08:13] (Like what was happening in that January stetch) [01:07:15] Hmm, I think container data might not be stored on the servers that are encountering storage problems. Might be fine to create, worst case worse, we have a script to fix. [01:32:24] yeh, creating a container isn't affected i think [02:18:04] Finally found a solution to the container listing [02:18:26] Should hopefully be able to get working on that shortly [04:50:12] how's the swift stuff? [05:09:52] I've got a way to recover the container listings, but that'll have to wait for later. Rebalancing is ongoing, but we may need to get another storage node if it doesn't work right (if only for the wiggle room). [06:42:31] Can anyone tell what exactly caused the sudden spike in disk space usage on the last 2 days? [06:42:51] I'm just curious. Not just because i'm a wiki owner, but because it is a technical curiousity to me too. [06:42:58] it's a mystery [06:44:03] it seems like space started to run out before people started spamming dump requests, but I guess this contributed too in the end [06:44:47] Ouch. [06:45:54] Maybe it has something to do with the thumbnail generation issue tho? Deleted images and old thumbnails not being removed? [06:47:40] [1/2] not a tech expert tbh, just hanging around lol [06:47:40] [2/2] but ooking at previous messages here thumbnails gonna be deleted for now? [06:47:58] and renewed later [06:48:05] Oh wow. [08:31:16] is it possible that a select few wikis are being quite a bit hefty on storage space? [08:31:43] maybe finding a way to run a check on storage space usage of each wiki might be a good way to find the cause... [08:35:27] one wiki w/ ~40k files hogged up request dump queue really bad previous week, everyone else stuck and MacFan had to cancel it manually [08:36:03] but in other circumstances such wikis aren't a problem? [08:37:25] although recently updated Content Policy prohibits usage of wiki as simply file storage [08:38:59] wikis about games remd to have a lot images from game extracts, like icons, etc, that's unavoidable [08:57:02] i'm still confused why you can't get a dump of the files and their sizes - i would expect that to be available through the swiftac pretty easily [09:53:18] <.labster> Permanent retention of dumps is a bad idea anyway. It’s not a true backup. My wiki had a 2.6 GB file sitting around since 2020 that I didn’t know about (another crat made it). I deleted it last week. I can’t think of a good reason to retain them longer than a week. [09:54:17] wait a sec [09:54:30] [[Backups]] [09:54:31]

[09:56:33] which of scheduled ones fits users' needs tho? like closest to request ones [10:02:38] <.labster> I have an ex-coworker who is a maintainer of Kiwix so in theory we could make downloadable wikis, but those don’t want the full dumps, just current content. [10:08:37] Ohhhh thats a Great idea @.labster [10:10:30] <.labster> Speaking of pre-pandemic stuff: https://github.com/kiwix/web/issues/110 [10:38:01] I mean to the dump thing [10:38:17] Where we keep dumps around for a specified time and then delete [12:22:37] This is best practices, many organizations have a set retention policy then delete or they just overwrite. Depending on regulatory requirements. [12:28:20] [1/3] Probably is wise to start being stricter on massive galleries of images or wikis treating as image dump. May need to implement monitoring or somehow set a hard limit. [12:28:21] [2/3] Reducing max file upload size might help. [12:28:21] [3/3] Reducing possible thumbnail options can help to cut down and regain space. [12:35:15] I thought we already had a limit around a couple of megabytes. [12:35:24] All that needs to be discussed by the community [12:40:06] Well since uploads are disabled, can't check. If it's set to 10 mb, maybe we need to lower to 5 mb. [12:43:00] [1/2] Not everything has to be a rfc. As of current, this is a purely technical matter. [12:43:01] [2/2] This is why miraheze has such problems. Tech team is being hindered by insistent community improperly interfering with rfcs on things not their concern. Sometimes limits need to be set, staff need to have the authority to make emergency changes or to alter to allow for continued service. [12:44:41] Yes, I saw the debated over premium. There are hosted wikis being a strain on servers, those need to be dealt with because current resources allocated is insufficient. [12:51:07] this notion of bothering or arguing with tech team is well known. In fact one resulting fallout caused a volunteer to quit because community didn't like personal judgement call on harshly dealing with an annoying user who didn't know their place. It's stupid and ridiculous behavior. This can't continue. [13:07:46] I would honestly advise & insist that new incorporation sets oversight of tech team to the board instead of community as current really isn't effective 🤔 [16:09:11] Oversight of the tech team sits with the Board now and always has [16:12:18] If I may recommend, maybe offer them up to the wiki archive team before deleting? [16:22:47] [1/3] Ok, well if you understand my full point especially in regards to my response to avengium on tech rfcs then you would realize lack of guard rails to this apparent Decentralized autonomous organization contributed to the crisis. [16:22:47] [2/3] Lack of tech volunteers is impacted by past community micromanagement of tech teams. [16:22:48] [3/3] Both of these must be addressed by new team. [16:22:50] We do periodic uploads to the internet archive [16:23:01] Of public wikis [16:45:21] The community don't micro manage the tech teams at all? [16:45:42] The tech team ask for feedback, but that's not mandate and neither were they ever bound to it [16:46:20] I haven't seen "community micro manages sre" cases [16:47:00] avenguim just drop a single message [16:49:24] You guys had an incident where annoying user chose to step out of line and was dealt with by tech team member thus triggering community outrage and the tech member quitting [16:49:40] Avengium shouldn't have posted as that's out of community scope and jurisdiction. [16:50:14] Avengium is a member of the community and should be allowed to post his opinion [16:50:37] And this opinion is out of scope [16:51:26] We do censor the community, period they are free to share their opinion [16:51:41] [1/2] The tech team shouldn't be moderating the community though also I don't recall that incident [16:51:41] [2/2] If you have the exact example, it can probably be explained [16:52:26] I'm not censoring him, I'm informing why suggestion of rfc isn't useful [16:52:30] the annoying user you're referring to had a fair amount of rights on Meta as full on volunteer tho (not defending him, just stating the fact) [16:54:08] Max file size policy is something we should definitely discuss with the community if we were to consider it [16:54:34] I agree on file size suggestion tbh [16:54:42] Uhhh that’s not a community issue to be decided on [16:54:58] We generally prefer to give the community a bit of time to give feedback and we listen. We don't know every thing you guys do on every wiki. [16:55:00] That’s a us decision because we don’t have unlimited resources [16:55:02] That was the first flare up that unfortunately CA pushed causing the person to quit [16:55:34] Yes it's an SRE decision overall but I don't think we'd be doing it without saying why to the community and allowing them to comment. [16:56:16] Need more information as I still don't think that's a tech issue? [16:56:27] At the end of the day, it's a physical and hard restriction, so the limit will never go but explaining to the community what fair usage is and allowing them to say if they think they might have an issue is very fair. [16:57:47] m3w, you wasn't really involved in or observed Miraheze/Meta untill last few weeks? genuine question [16:59:11] [1/2] From what I recall, CA forced through a proposal while Rhinosf1 requested courtsey time to handle privately. CA refused and situation blew up. [16:59:11] [2/2] A user did something to annoy a tech person and got demoted or banned. [17:00:01] I've been here and I'm more aware of things then you think [17:00:51] So the problem there isn't the community have oversight over tech, but tech have too much influence on the community if annoying someone in SRE can get someone banned? [17:07:33] is this the user that was terms of use banned? [17:07:45] or is this nale [17:07:57] *T&S [17:30:35] This was not the recent situation, this was at least 2 years ago or so. My understanding is that was factor olin Rhinosf1 resigning [17:32:43] Oh [17:33:19] Is that the one that we found was the troll [17:33:32] I’m not aware of another user [17:34:06] @rhinosf1 can you add more context as im unsure what incident this could be about? [17:34:17] riddles ... [17:35:15] [1/3] Not too sure I would agree from tech perspective as you have hard physical limits, if current file storage is an issue and reducing is what allows continued service then tech has no choice. [17:35:15] [2/3] At that point, tech would simply advise on what is reasonable. [17:35:16] [3/3] Enforcing existing rules on not treating as a personal file storage backup is probably something to consider. [17:35:44] I can remember the incident referred too. Not much of the details though. [17:36:12] Nothing to do with any time I resigned though [17:36:48] Which user was it @rhinosf1 [17:37:18] @paladox I believe it surrounded a group of users wanting to revoke permissions of a user [17:38:04] I'll be honest, I think the conversation needs to move on. [17:38:12] Oh [17:38:27] [1/3] Probably was deemed a troll that earned roles on meta. [17:38:28] [2/3] That was the first time that I started to even advise CA in DM. Even back then, I knew of him disregarding would eventually become a problem. Sadly, all went downhil last week. [17:38:28] [3/3] I'm sorry to everyone for not notifying when I had knowledge yet was asked fir confidentially [17:39:22] @m3w please move on [17:39:26] Ok [17:39:54] Well time for a reset [17:40:48] Roles should probably be better defined with better guidelines of conduct [17:52:26] [1/2] A little late to the party, but going to ask that we leave off speculating in vague terms on events from 2+ years ago. [17:52:26] [2/2] Regarding roles and definitions, those involved with transition already agree that a more cohesive comms plan and strategy is a good idea, my understanding is that's already being worked on outside of public channels. [17:57:51] @rhinosf1 how do we get beta working? [17:58:03] i don't remember the process if you delete all the json files [18:01:05] @paladox how is it broken [18:01:12] wiki shows as missing [18:01:16] Apart from the fact it's beta [18:01:37] @paladox believe delete but sometimes it needs a dummy json file [18:01:52] Eval.php is good at refreshing it [18:02:21] That's more of a CA query though as that's a CreateWiki issue not a beta issue [18:13:39] hmm [19:33:58] <.labster> Are file uploads working yet? [19:43:48] Try https://github.com/miraheze/CreateWiki/blob/master/maintenance/generateMissingCache.php [19:48:29] Hey, what's the current swift status? [19:49:07] Anything I can advise on? [19:52:42] File uploads have been re-enabled [19:53:42] which disks have space? are things getting actively deleted? [19:54:08] they all have space [19:54:32] fixing the weight fixed the disk. Still running towards the low side but all disks are load balancing better. [19:55:02] is there an idea of how much space all of the images WOULD take up? someone said earlier that they expected that even running a query like that would take days, but i think that is probably not correct [19:55:42] well because of the incorrect stats in AC (need to be fixed by reinserting the objects) it would take a long while [20:01:17] oh, the AC is just wrong? ok [20:01:30] can you remind me what you changed about the weights? [20:03:06] changed it to match disk size [20:03:52] OH it was actually just uniform? i see [20:04:05] yeh [20:04:19] @paladox not sure if you’re still having issues with beta, but also worth noting that we don’t use beta.betaheze.org any more. Now we have meta.betaheze.org and test.betaheze.org. I saw recently (not today) that they were broken and fixed them. [20:04:57] oh [20:05:29] I also know how to fix large objects for DD [20:05:39] And so instead of betawiki we have testwikibeta and metawikibeta [20:05:41] requires putting .rlistings on the segment container. [20:07:01] I think we need to still have large ones done manually even if they go to swift, as other muse you hold up all dumps for hours [20:07:18] otherwise* [20:07:36] (Since only one runs at a time) [20:08:05] <.labster> When I tried to dump ATT wiki it queued for a while, then failed. Which is less than awesome user experience. [20:09:11] btw I have found on a couple wikis that the CommentStreams extension breaks dumpBackup.php [20:11:12] Yeah, we still don't have full support for large files at the moment. [20:11:35] Well ATT is big enough that even s text dump is an 8GB file [20:21:54] wasn't comment stream an overall broken extension, or it's another one? [20:22:13] of comment extensions [20:38:42] Yeah large wikis like ATT never worked with data dump