[03:10:35] Hure [10:48:37] !log tools restarting toolsdb to apply new config T353093 [10:48:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:48:42] T353093: [toolsdb] MariaDB process is killed by OOM killer (December 2023) - https://phabricator.wikimedia.org/T353093 [12:19:06] !log tools restarting toolsdb again to apply a config fix T353093 [12:19:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:19:10] T353093: [toolsdb] MariaDB process is killed by OOM killer (December 2023) - https://phabricator.wikimedia.org/T353093 [13:13:16] !log admin restarted nova-fullstack on codfw as it was stuck (and alerting through stale prometheus file) [13:13:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:30:02] Hi again, now I'm hitting a major problem in the migration to Kubernetes: tools I was using like pdflatex is no longer available; coud or what other options [13:30:15] Sorry, incomplete message [13:31:18] Hi again, now I'm hitting a major problem in the migration to Kubernetes: tools I was using like pdflatex, wget, pdftoppm and convert are no longer available; could they be added or what other options do I have? Remote exec in other nodes seem to get stuck... [13:32:30] hi jem, are you using the build service or the already-built kubernetes images? [13:32:54] Hi, dcaro, and thanks [13:33:24] I guess the already-built, I'm just launching a webservice with type php7.4 [13:34:08] I'd recommend then trying to use the build service, that will allow you to specify the packages you need when building your image [13:34:25] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Installing_Apt_packages [13:34:35] I can help you getting started if you want :) [13:34:52] otherwise, you'll have to open a task requesting the extra package to be added to the image [13:35:24] * jem takes a deep breath [13:35:42] something like https://phabricator.wikimedia.org/T313550 [13:36:58] Ok, I guess if it's a one-time task, I should face it [13:38:36] it is yes :), you have an example of php app here https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Buildpack_PHP_tool, to install packages to that, all you'd need is create an `Aptfile` at the root of the repository, with the list of packages you want installed [13:39:34] Ok, and the root of the repository would be the root of my tool, /data/projetc/jembot, is that correct? [13:41:35] no, your code has to live in a public git repository, you can get one for free in wikimedia's gitlab by clicking "create repository" under "git repositories" on the left bar on your tool's page in toolsadmin.wikimedia.org [13:41:37] https://usercontent.irccloud-cdn.com/file/CzuOcNA9/image.png [13:41:52] in case you need one [13:42:44] Hmmm, this is getting complicated :( [13:43:41] (that also helps making sure your code is publicly published, part of the toolforge rules) [13:45:54] The problem is that my code is published from inside my tools, with my own code to show it [13:46:34] That way I can hide comments that aren't "cleaned" yet, link functions to other files and so on [13:47:37] If the code has to be fully published then this is a real problem for now [13:48:42] Yes, you have to publish the exact same code that you run [13:49:09] :( [13:49:43] Things were going too easy... [13:49:43] I understand though that as that has not been enforced, there's an effort needed to remove private things and such from the code [13:50:37] Yes, and not just for being private, but for being messed up sometimes [13:52:40] that's ok :), nobody's code is perfect, and most of the time, as long as it does the job you need done is enough [13:53:54] let me know if you want some help. For secrets/config and such you can use environment variables (see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars_Service) [13:54:05] so your code does not need to have them hardcoded [13:54:18] Ok, dcaro, thanks, and do you think a task phab requesting those packages would be "approved"? [13:54:48] I understand that would avoid the need to use git, build, etc. [13:54:49] eventually maybe, depends also on how big the image is afterwards, as that image is used by everyone in the cluster (it's a shared image) [13:55:01] you should still though publish your code [13:55:05] (the same you run) [13:55:11] It is published :) [13:55:30] if I do a diff of the one you published with the one it's running will I get no differences? [13:55:39] But in my way and with better control over it (in my opinion) [13:56:10] The differences would be in the comments and the links [13:56:31] then it's a slightly modified version of the code you are running, not the code you are running [13:57:00] Yes, although the "code that produces effects" is the same [13:57:22] well, that's not a qualifier for open sourcing your code xd [13:57:44] there's many ways of making the same effect with different code :) [13:58:14] I know, but the alternative would have been tools not working yet or with less functions [13:58:15] I understand the similarity though, but yep, the requirement is to publish all the code you run [13:59:03] But having to do it with an external tool... well... [13:59:38] gitlab is also supported by the foundation, and it's a core part of toolforge [13:59:39] If I use git, would I still be able to update things from the shell, with text editors, etc.? [13:59:46] so you could say it's the same tool [14:00:01] yes, as long as you push them to the git repository after [14:00:11] Hmm [14:00:48] (you can also use any other public git provider if you want though, like gitlab.org, github, gerrit, etc.) [14:01:05] Ok, I have to think carefully about it [14:01:21] For now, I will have to go back to Grid [14:01:23] sure, let me know if I can help in any way (git tips or whatever you need) [14:01:33] Thanks, dcaro [14:02:40] I understand your point (or WMCS point), but this doesn't fit very well in my many-years-old workflow [14:03:32] it's the only way we can allow having another bunch of many-years workflows :) [14:03:45] s/allow/enable [14:03:51] :) [14:04:09] I understand and I fully respect all your work here [14:04:20] (as in, it's not us that want to push people, it's just that the grid project died many years ago and we can't keep dragging it through new linux versions) [14:04:30] Yes, I read about it [14:05:11] And I'm not and won't complain about the need of the migration [14:06:12] I appreciate it :) [14:07:32] :) Thanks again, I will be thinking which is the way to go [14:09:43] fyi. moving to the build service is the longer term approach, even for other k8s tools (if you think in the long term) [14:10:39] Ok [14:40:02] !log tools deploy toolforge-builds-cli 0.0.10 (T341067) [14:40:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:40:06] T341067: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 [16:49:37] !log tools restarting toolsdb before it's about to go OOM, enabling performance_schema for debugging [16:49:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:07:24] jem: It is currently highly unlikely that a request to add new apt packages to the php or python containers will be granted. It is not impossible, but it would require convincing arguments about the general utility of the packages to the majority or even supermajority of tools using that image. [18:09:12] It seems theoretically possible to make a buildservice based "reusable" image that contains various runtime elements but mounts content like PHP files from somewhere else at runtime. I don't know if anyone has done a proof of concept for that yet. [18:10:28] Long term, getting to a place where you are comfortable with the perceived social risks of sharing your code in a public git repo is ideal. [18:11:01] Like David said, nobody's code is perfect. :) [18:32:40] jem: https://en.wiktionary.org/wiki/auch_nur_mit_Wasser_kochen [18:34:31] all the code is full of bugs:) [18:44:34] Heyo, I made the wish to never use npm, and so I am using pnpm. But the Heroku buildpacks used for the Build Service only support npm/yarn. I am using "toolforge build start -e USE_NPM_INSTALL=true github.com" in order to use  "npm install", but it is not working, any idea? [18:44:48] "The `Heroku Node.js npm Install Buildpack` uses the command `npm ci "--production=false"` to install your Node modules. This command failed and the buildpack cannot continue. See the log output above for more information." [18:46:00] Tried `--envvar USE_NPM_INSTALL=true` too, trying my luck, no changes [18:47:12] I guess it will be the same problem if using `NPM_CONFIG_PRODUCTION=true` [18:47:47] Lofhi: is the "USE_NPM_INSTALL" setting described in Heroku buildpack documentation, or are you hoping it will do something because of other information you have? [18:48:05] ehhh [18:48:20] the first solution would be to throw pnpm I guess, but no, it is from the doc: [18:48:28] https://devcenter.heroku.com/articles/nodejs-support#using-npm-install [18:49:31] Sorry for the copy pasting: [18:49:31] tools.hitaden@tools-sgebastion-10:~$ toolforge-build start --help [18:49:32] [...] [18:49:32] Options: [18:49:33] [...] [18:49:33]   -e, --envvar TEXT      Environment variable and value to set during build [18:49:34]                          (format `NAME=value`) [18:52:06] *nod* I guess we need to figure out if the actual buildpack reads an envar or if the `config:set` magic in the actual heroku stack does something different. [18:52:11] https://phabricator.wikimedia.org/paste/ and https://paste.toolforge.org/ are handy pastebins when needed. [18:53:19] Ah, fine, will remember [18:53:27] It does look like that is read from the environment... https://github.com/heroku/heroku-buildpack-nodejs/blob/82c952e82e092a39a3d2899b369b1faa9da14137/lib/environment.sh#L33 [18:53:32] Do I need to open an issue? [18:54:12] And do you think that using the envvars service... could help? [18:54:23] Not sure about the "build context" [18:54:38] Lofhi: it would not hurt to have a phab task to track the problem. [18:55:52] d.caro is the most knowledgable how build service does what it does, but he's gone for the day (and maybe most of the rest of December?) [18:56:29] Lots of staff folks will be out in the next two weeks unfortunately. [18:56:29] I think I can try, will take 10s [18:58:08] `toolforge envvars create USE_NPM_INSTALL true` didn't work :D [18:58:15] Doomed... :-( [18:59:42] Wait, didn't use `toolforge webservice buildservice restart` before rebuilding [18:59:50] I think the `--envvar USE_NPM_INSTALL=true` is probably on the right track. I'm not sure that buildservice has any access to the data set with `toolforge envvars`. [19:00:42] Bit sad, this means I am gonna need to use npm [19:01:43] Hahahaha: https://elements.heroku.com/buildpacks/unfold/heroku-buildpack-pnpm [19:02:20] If switching to npm is a viable short term workaround that may be best to keep you moving, but it is also very reasonably to ask for help figuring out how to use the config settings in the node.js buildpack [19:02:34] yea sure [19:03:08] I think that figuring out how to support arbitrary buildpacks like that pnpm one you found is also on the roadmap, but I don't know how far off it is in reality. [19:03:41] d.caro was poking at adjacent issues just this week in getting a mono buildpack working [19:04:02] Meh, I mean when you are not using the mainstream tools you can only blame yourself [19:04:18] I looked to generate a package-lock.json with pnpm, nada [19:04:35] You can import, no export, and it seems they don't like the idea to support export [19:04:47] https://github.com/orgs/pnpm/discussions/3367 [19:05:14] oh wait [19:05:40] `npm i --package-lock-only` could be a solution, but it will not fix the --envvar not working, so I will open a ticket [19:06:36] npm is so slow [19:06:40] save me [19:07:02] `npm ERR! Cannot read properties of null (reading 'matches')` [19:07:04] great [19:13:49] Alright, will just try to work with npm and pnpm, gonna turn off my radiator for winter [19:15:59] for logs purpose: I deleted the node_modules folder, used `npm install`, then `pnpm install` in order to be still able to use it [19:16:28] `npm i --package-lock-only` now works and I have a `package-lock.json` and a `pnpm-lock.yaml` [19:16:55] you can't use `npm i --package-lock-only` if you used pnpm BEFORE to install, but you can do it the reverse way, fine [19:22:19] Still logs purpose: Multiple lockfiles found: package-lock.json, pnpm-lock.yaml => "If you have yarn.lock checked into your project, but want to use npm to build on Heroku, add yarn.lock to your .slugignore file." [19:26:14] great, it is not working [19:28:02] It should, "The .slugignore file causes files to be removed after you push code to Heroku and before the buildpack runs.". Guess I am running out of options. [19:31:29] "Slugignore only controls what ends in the app slug, not what is in the Heroku repo. So if you ran heroku run bash and checked with ls, you would find that the things in .slugignore were not there. Use .gitignore if there's something you don't want to include in the git repo." [19:31:47] (╯°□°)╯︵ ┻━┻ [19:33:37] Okay, going to have a separated git branch only used for the build service; let's try [19:45:57] Not working, I thought there was a way to specify the branch within the url, but no [19:46:21] you can do that with --ref [19:48:04] but there is no way to use it with `toolforge build start url` :-( [19:48:38] OH [19:48:43] --ref TEXT             Branch, tag or commit to build, by default will use [19:48:44]                          the HEAD of the given repository. [19:48:46] true, mea culpa [19:48:54] yoooupi [19:50:51] oh my words, it is currently building [19:51:14] new error unlocked [20:16:38] No idea why node complain for TypeScripts types [20:17:13] Building a production release locally works, with Heroku buildpack, it fails [20:17:35] ```src/types/dbTypes.ts:15:23 - error ts(2580): Cannot find name 'Buffer'. Do you need to install type definitions for node? Try `npm i --save-dev @types/node```` [20:17:58] Sub-problem: `-e NPM_CONFIG_PRODUCTION=true` does indeed not work [20:18:17] No reason to ask for dev depencies in a production env [20:18:52] dependencies* I will open a ticket on Phab' with all the steps, enough spamming for today [20:26:33] !log tools restarting toolsdb to avoid upcoming oom crash [20:26:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:01:10] Alright, sleepy. T353557 done. Thanks for help bd808 taavi. [21:01:10] T353557: Toolforge Build Service doesn't pass envvars to the buildpack - https://phabricator.wikimedia.org/T353557 [21:01:19] o/