[07:40:49] o/ I'm writing some onboarding docs / notes / guide for a new hire. When someone creates a wikitech account and adds an SSH key, should they be abkle to ssh anywhere straight away? (I'm guessing no and some sort of approval needs to be waited for?) [07:41:41] addshore: correct, you'll need to get approved to toolforge / added to a cloud vps project [07:42:03] And only after that point could someone get to a bastion? [07:42:28] correct [07:42:33] Thats awesome and fine though, I just need to add a bullet point to reach out to get added to one of our projects then after making their account :) [07:56:19] Also any idea if we have a best practice post for managing SSH keys (for those that havnt before), it seems most of our docs jump straight from making keys to ssh config to logging int [08:57:29] !log tools drain tools-k8s-worker-53 [08:57:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:37:01] !log tools controlled undrain tools-k8s-worker-53 (T291546) [11:37:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:37:05] T291546: Puppet agent failure detected on instance tools-k8s-worker-53 in project tools - https://phabricator.wikimedia.org/T291546 [11:43:57] looks stable again :] [16:52:05] !log tools.paste Clean up spam, add captcha, disallow forever pastes [16:52:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paste/SAL [16:56:30] balloons: it now shows as last used 7 years ago [16:58:15] RhinosF1, yeah, the admin tool didn't allow for much granularity. After marking spam, this was what remained [16:58:56] balloons: is it worth existing? [17:00:34] RhinosF1, a good question. It will be interesting to see what use it gets when it's not overrun with spam [17:01:36] balloons: hopefully some [17:01:45] I hope it's only use isnt all spam [17:01:53] Too many pastebins exist though generally [18:06:33] !log tools launching tools-nfs-test-client-01 to run a "fair" test battery against T291406 [18:06:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:06:39] T291406: POC: puppet-provision a cinder-backed NFS server in eqiad1 - https://phabricator.wikimedia.org/T291406 [18:07:12] !log toolsbeta launching toolsbeta-nfs-test-client-01 to run a "fair" test battery against T291406 [18:07:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:37:30] musikanimal, in regards to T290768. Is there any value in creating a project for you now, or how else can WMCS help? I don't want to mislead as we can't yet commit to that much storage. [18:37:31] T290768: Request creation of wikiwho VPS project - https://phabricator.wikimedia.org/T290768 [18:45:32] balloons: I understand, I know it's a big ask! I suppose it'd be helpful to have a project with the standard quota so we can test WikiWho for a *very* small Wikipedia. That allow us to get acquainted with the stack and installation process, and maybe even identify some ways to optimize storage. But, none of this matters in the end if we can't support the big wikis [18:48:05] you could still do the puppetization work and maybe one day it could be possible to have the code on cloud VPS but mount more storage from the cloud into it? [18:49:00] externa cloud storage mounted while all else is in house I meant.. shrug [18:49:01] yeah exactly [18:52:56] it's not so far off to support the use case. The infrastructure to scale is in place with ceph and cinder. I'll add a note to the ticket [19:57:01] !log tools.notwikilambda deleted and recreated `update` deployment, it was stuck unable to roll out a new replicaset due to quota; Iā€™m not sure where the new replicaset came from, maybe I accidentally rollout-restarted it? [19:57:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL [20:22:46] bstorm: hi, wrt T281203 how many dumps they keep per each wiki? [20:22:47] T281203: dumps distribution servers space issues - https://phabricator.wikimedia.org/T281203 [20:23:34] actually I wonder if something caused it to grow [20:48:23] @chicocvenancio, majavah: there's a report that https://github.com/toolforge/paws/pull/80 doesn't seem to be working properly - see https://el.wiktionary.org/wiki/%CE%A3%CF%85%CE%B6%CE%AE%CF%84%CE%B7%CF%83%CE%B7_%CE%B2%CE%B9%CE%BA%CE%B9%CE%BB%CE%B5%CE%BE%CE%B9%CE%BA%CE%BF%CF%8D:Bots#Please%2C_could_you_help_us where they're still getting warnings for the old variable [20:48:47] should I file a bug for that or will either of you be able to look into it? [20:57:52] Amir1: honestly I don't know. It seems natural that there would be a slow growth overall, and there is. It drops periodically on the cleanups, but the steady state trends upward and has since the inception of the dumps system. [20:59:41] bstorm: noted, I think it'll go down a bit in the next couple of months but it would be great to take a deep dive to see why it's moving upwards this fast [20:59:48] e.g. compare https://dumps.wikimedia.org/commonswiki/20210901/ and https://dumps.wikimedia.org/commonswiki/20210701/ [20:59:56] commonswiki-20210901-image.sql.gz 25.7 GB [21:00:13] commonswiki-20210701-image.sql.gz 249.8 GB [21:00:25] heh [21:00:33] That's a fair difference. [21:00:56] Anyone logged-in to Wikitech that can review/revert https://wikitech.wikimedia.org/?diff=1926186 ? [21:01:01] I haven't had the brain space to even wonder about the actual content on those lately. [21:01:13] hauskatze: done [21:01:17] There are arbitrary datasets exported from the analytics hadoop cluster on the dumps servers too. I have no idea if they are contributing to the growth, but seems possible. [21:01:20] Amir1: :) [21:02:15] bstorm: so when the cleanups of old months gets triggered, it'll clean up (and some other, like flaggedtemplates and flaggedimages) and give us some time [21:02:27] I'd be happy to take a look if given access :D [21:02:48] Those run on production servers [21:03:14] I know there's a process for that, but I don't even know what it is at the moment :) [21:03:49] hmm, in a month I can check it myself but I don't think it'll become an issue until then [21:03:53] Amir1: most of the dumps content is managed by Ariel. You could certainly hook up with them to ask questions about retention, etc. THe WMCS crew are really just responsible for make sure the host is up and nfs is working. [21:04:11] bd808: noted. Thanks! [21:04:22] (and getting bigger hardware when needed) [21:04:23] I thought WMCS team maintains them [21:04:46] we run the boxes, but not the content if that makes sense [21:04:51] legoktm: I think I may have found your mystery. That change is NOT deployed. The last build of that image was not tagged "latest" [21:04:53] doing that now [21:05:28] sure [21:07:57] * bd808 should probably stop using "we" and "WMCS" interchangeably at some point [21:08:09] :) [21:08:16] bstorm: ahhh, thank you :) [23:07:29] * legoktm hugs Skynet [23:50:49] Amir1: I'm pretty sure that *decrease* on image table was due to the cleanup that was done taking img_metadata out of it [23:51:48] Platonides: yup. You know I did that? [23:52:07] nope [23:52:09] * Reedy giggles [23:52:23] It was only img_metadata of pdf files but yeah. [23:52:40] I missed your point above, sorry [23:52:48] Djvu will come later [23:52:55] It happens šŸ˜ [23:54:27] changing some of those .gz into .xz should help as well