[09:46:36] !log deployment-prep removed some stray whitespace in /var/lib/git/operations/puppet that was preventing rebase on deployment-puppetmaster04.deployment-prep.eqiad.wmflabs [09:46:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [09:57:36] Hey all, I am struggling to find enough memory resources for a bot task. It needs up to ~8Gi, and this will slowly increase with time. [09:57:41] If I deploy it to Toolforge as a cronjob using Kubernetes with increased resources requested as described on wikitechwiki, it seems to be unable to get started at all. And I am still at the low end of how much memory I would like to allocate. [09:57:51] So, what are my options? Do we have another platform that has more memory available? [09:58:03] For those who wonder: it's a Python script that uses the pandas library, and the critical part is to load two large datasets into dataframes to perform a database-style join operation. [09:58:11] Thanks in advance for all help! [11:53:47] you could try loading it in an actual database and see if you can do it there; or you could request a wikimedia cloud project, create a vm there with enough memory and do it there [12:03:49] gifti: thnx; I have already considered to make a detour via a dedicated tool database in order to perform the join operation therein, but it would probably need memory on a similar scale and I am not aware of the limits on database servers [12:24:14] s51071__templatetiger_p, which is listed on https://w.wiki/5j6i as a kinda-problematically large database, seems to have around 7 GB of data [12:24:24] `SELECT SUM(data_length + index_length) FROM information_schema.tables WHERE table_schema = 's51071__templatetiger_p';` [12:24:30] s51290__dpl_p is at around 9 GB [12:25:06] I think a separate cloud VPS project might be a good idea [12:28:51] huh, they’re far from the largest databases though [12:29:14] mix’n’match is at 124 GB (plus 34 GB for large catalogs?), quickstatements at 47 GB [12:29:21] `SELECT table_schema, SUM(data_length + index_length) FROM information_schema.tables GROUP BY table_schema ORDER BY SUM(data_length + index_length) DESC LIMIT 25;` [12:30:28] Lucas_WMDE: iirc, with the databases listed there, the main problem was the amount of write traffic they were receiving [12:31:13] ok [12:33:20] size on disc is not my problem as much as I am aware [12:33:56] what amount of data are we talking about here? [12:34:00] not sure about database internals; if I join two tables, does the database engine have to load everything into memory? I think yes, but I am not sure... [12:34:15] two tables, 10M and 60M rows [12:34:17] depends on the query and the database indexing [12:34:40] yeah, that's not a problem on toolsdb on elsewhere [12:34:49] or* elsewhere [12:38:06] okay, I will give it a try via a tool database. needs some coding this evening, but I think I'm able to evaluate whether this works out for my usecase. If I can avoid another layer of complexity (cloud vps), I'd happily do so :-) [12:39:47] thanks for the input at this point. if I have follow-up questions, I'll show up again [14:20:25] an alternative is use sqlite in the container /tmp directory, the container storage limit is 10GB including the container image, so it has near 9GB of free space, it is storaged in the same node the container is running, so it does not use NFS