[05:29:33] I had an interesting problem trying to use the toolforge build service [05:30:18] I built my image once, then I added some more packages to Aptfile, then I built it again, but now the new packages aren't there [05:31:20] if I start an interactive session in this image with kubectl, I can see the correct packages in /layers/fagiani_apt but that layer doesn't seem to be mounted as part of the root overlay [05:33:16] but I don't know how layers get chosen and assembled or what thing controls this [05:35:26] I guess it's a bit early for dcaro [05:36:00] I'll write it up on T319953 [05:36:01] T319953: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 [08:52:36] TimStarling: I'll give it a look [09:00:32] thanks [10:45:20] leloiandudu: it's ok for now, I ĺl try to reproduce/get more logs [11:10:14] dcaro: what is the thing that does that ldconfig hack? [11:10:37] https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/apt-buildpack/ [11:10:50] it's the buildpack that installs the packages [11:11:12] the script is https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/apt-buildpack/-/blob/move_to_api_0.10/bin/build?ref_type=heads [11:14:28] when I run "env" as a job, I don't see those environment variables. I don't think that profile.d hook is being run [11:14:58] that might be because of https://phabricator.wikimedia.org/T355214, what command are you running exactly? [11:15:34] (btw. just updated https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Job until that is fixed with some notes for ad-hoc commands on toolforge jobs) [11:41:28] toolforge jobs run --image tool-panoviewer/tool-panoviewer:latest --mount all --command 'env' --wait 60 test ; toolforge jobs logs test [11:43:02] profile.d will only work if a shell is being used, which it isn't [11:43:53] there's no parent process anyway -- I guess it could use exec but if there's no shell, that would explain the issue [11:48:56] even when I run a shell, I still don't see them -- how would that file in /layers/fagiani_apt/apt/.profile.d/ get sourced? [11:51:13] you'll need to use `launcher ` instead of a shell, it's that `launecher` process the one that loads the environment, so for example: [11:51:13] `toolforge jobs run --image tool-panoviewer/tool-panoviewer:latest --mount all --command 'launcher env' --wait 60 test` [11:51:18] ^that should work [11:51:58] or if running multiple commands, you will have to prepend each of them with `launcher`, or wrap all of it in a shell `--command 'launcher sh -c "command1; command2;"'` [11:52:17] or `--command "launcher command1; launcher command2"` [11:53:11] once the above task if fixed, the first launcher will be added automatically, so `--command "env"` will work, but you will still need to add the extra ones or wrap everything in a shell [11:53:33] oh ok, my procfile command is /usr/bin/echo "Hello, this is the Procfile multires entry point speaking." [11:54:00] since you always specify a command when you run the job, I didn't realise the procfile command was actually used for anything [11:55:14] you can instead just pass the procfile enty, for example if you have in your procfile `run-something: echo "hello world"`, you can run the job like `--command "run-somthething"`, and that will read the procfile and run what's there [11:56:09] note that profcile entries have precedence over commands, so naming the entry `ls: echo "hello world"` will effectively prevent you from running the actual system `ls` without using the full path [12:02:07] does it have to be like this? [12:02:36] is this heroku weirdness leaking into toolforge jobs? [12:13:05] I think buildpack-created containers have their own rules, and toolforge-jobs is at the moment not aware of them [12:13:34] we might have to extend toolforge-jobs to become more buildpack-aware and maybe add some extra flags? [12:14:03] can toolforge-jobs just prefix "launcher" to all non-procfile commands? [12:14:12] that's what dcaro just merged [12:14:24] and it's probably a reasonable workaround [12:15:00] https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/58 [12:15:42] good to hear [12:16:34] would you like to hear my other thoughts on the usability of this system? [12:17:01] yes please :) [12:17:40] in toolforge-jobs-cli, it's weird having the job name being positional and the command being a required named option --command [12:17:48] I think this was inherited from some other tool? [12:18:30] it can be fixed because you can just make --command be optional. If it's not present, interpret the positional arguments (together) as the command [12:18:59] so toolforge jobs run --image=... --name=... echo hello this is my command [12:20:34] regarding toolforge build run, I registered my gitlab repo in the toolinfo, and it's predictable from the tool name anyway, so it would be cool to have a default git URL [12:22:05] default git url: definitely, and it's something we have discussed already [12:22:42] the "jobs run" CLI, I think it was inspired by the underlying kubectl CLI, which is... awkward :P [12:23:07] are we assuming toolforge users will be familiar with kubectl? [12:23:30] Procfile is confusing to me -- I need to define a webservice but it won't actually be used because I'm just trying to make an image for running a job [12:23:32] not at all, though if they are, they can of course use that for advanced features [12:23:59] I think it should be assumed that toolforge users know basically nothing -- it's meant to be a low entry barrier service [12:24:33] for job images I would like to omit Procfile altogether and have jobs-api generate a stub one for me [12:25:39] I think copying the kubectl design was mostly a matter of convenience and easy implementation _for us_. I think we can likely come up with a "toolforge cli" v2 which abstracts a few more things away [12:25:59] like I say, this change can be made without breaking b/c [12:26:09] you can just make --command optional [12:26:20] so no need for v2 [12:26:21] one issue is that k8s require you to give a "name" to each job [12:26:37] yeah, just generate one, didn't the old grid engine do that? [12:26:48] just make it $tool-$(date) or something [12:27:27] yeah that's definitely possible, I don't remember why we didn't do it in the first place when creating the jobs-api, there might be some reason or maybe not :) [12:27:43] maybe you need the name to get the status later? [12:28:02] the tool I'm migrating doesn't need that, it just queues a job and polls NFS for result files [12:28:11] I think it was overall easier to implement it in this way, but it doesn't mean we cannot improve it over time, and maybe as you say it doesn't have to be a breaking change [12:28:40] re: Procfile, it doesn't technically require to have a "webservice", but just a list of ": " [12:29:09] wikitech says the first entry needs to be a webservice [12:29:35] yes, again it's our implementation, not the spec :) [12:29:45] so we can remove that requirement later on [12:32:54] will images be magically rebuilt in bulk or do I need to log in and rebuild once per year or so? [12:33:47] good question: one big advantage of buildpacks is that we can rebuild in bulk without user intervention, but we haven't discussed how to actually do it, as far as I know [12:34:18] most tool authors will not log in unless their tool is actually broken and don't want any maintenance responsibilities [12:34:47] so it would be good to get some sort of automatic updates [12:34:47] I think that was part of the reasoning for choosing buildpacks and (for now) not allowing custom docker images [12:35:15] if my Aptfile has a package in it that has a security release later, I would like to not have to care about that [12:35:47] yes, that is definitely part of the plan. if it will work as smoothly as we hope... we'll find out :D [12:39:12] sounds good [12:42:15] yep, partially using the code and only the code as instructions to build and run the application (Aptfile, Procfile partially) helps us to be able to rebuild, restart and overall maintain the application without the need of the maintainer intervening [12:43:24] thanks a lot for the feedback btw. [12:44:29] no problem [17:38:06] !log lucaswerkmeister@tools-sgebastion-10 tools.sal webservice restart # 500 errors [17:38:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL [18:20:06] no-stupid-questions: how could I figure out what project has a particular webproxy? I saw a question about https://phab.wmflabs.org which points to 172.16.5.140 internally, but that's about where my knowledge of the landscape ends. [18:20:49] thcipriani: https://openstack-browser.toolforge.org/proxy/, which is really slow to load but will eventually work [18:21:02] thanks taavi [18:21:10] to save you some time, it's probably in devtools [18:21:26] that's where I looked, but I didn't see that host [18:21:46] oh, but you're right [18:22:32] alright, that gives me more ways to dig now, thanks! And TIL. [19:13:18] !log lucaswerkmeister@tools-sgebastion-10 tools.lexeme-forms deployed 604b43e316 (l10n updates: it) [19:13:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [21:08:02] !log bd808@tools-sgebastion-11 tools.bridgebot Bridge #wikipedia-it to Telegram group (T356316) [21:08:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [22:39:03] Hi, I am in the process of deploying a Django app on Toolforge and am facing issues with static files. What is the recommended was to approach this? [22:40:30] @Danny: can you explain your problem in more detail? [22:41:54] Of course, thanks for getting back to me. [22:41:56] https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Django_OAuth_tool#Login_and_deploy has some info about serving static assets via uwsgi [22:42:30] Let me give that a skim real quick. [22:42:51] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service/My_first_Buildpack_Django_tool#Static_Files has some info for build service based deploys [22:43:14] I create an application locally and followed this guide for deployment: [22:43:15] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service/My_first_Buildpack_Django_tool [22:43:47] However, the directory mentioned in the link does not exist: [22:43:48] `/data/project/YOUR_TOOL_NAME/www/python/static` [22:44:08] It does not go farther than /data/project/syncbot/ [22:44:31] @Danny: :nod: Things are different with a build service based deployment. Try https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service/My_first_Buildpack_Django_tool#Static_Files [22:45:11] Okay! That makes more sense. [22:45:31] I actually tried adding that extra slash, but it did not seem to have an effect. This was the commit: [22:45:32] https://github.com/deni/syncbot/commit/87170b2db3bdf399f9fdbf27594100ee422a33d4 [22:48:10] I think blancadesal wrote that tutorial. https://gitlab.wikimedia.org/toolforge-repos/django-test-buildpack-app/-/blob/main/mysite/settings.py?ref_type=heads#L140-144 is her example. [22:49:06] Let me just make sure that they are actually the same. Maybe I've just been starring at code for to long to see that they are different. [22:50:08] I can see her deployed demo serving assets, so it is somehow possible. :) https://django-test.toolforge.org/static/polls/style.css [22:50:43] I just noticed the same thing! [22:51:16] I didn't see it mentioned in the tutorial, but she does use `whitenoise.runserver_nostatic` in INSTALLED_APPS [22:51:49] Maybe it's easiest if I just reach out to her directly. [22:53:15] She's usually around here during EU working hours [22:53:44] Lol, I guess I'll be working late tonight >.< [22:54:16] bd808, while I have you. I am not sure if you noticed, but I am seeing the message described in T353752. It seems that the job did not approve my account despite being in the groups mentioned in the documentation. Do you know why this could be? [22:54:44] Let me check on the bot to see if it is alive... [22:55:07] It would certainly explain it if the bot is not running :p [22:55:45] looks like the last run was 2024-01-31T22:54:00Z [22:56:14] Hmmm. Let me just see the public log. [22:57:07] Nope, do you see a more detailed log on your side? My username is "deni". [22:58:38] Let me run the bot in verbose mode. It does look like you should be approved as a Toolforge member [22:59:36] Sounds like that might tell. Thank you. [23:03:19] still looking. the bot may not be starting and also may not be sending me an email telling me that it didn't start... [23:04:10] 👍 [23:06:15] !log bd808@tools-sgebastion-11 tools.gitlab-account-approval Rebuilding image to see if that fixes the startup failures which appear to be related to the "launcher" append job service change [23:06:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.gitlab-account-approval/SAL [23:11:25] ok. the bot is confirmed running now, but it does not see a pending "deni" account to approve. [23:11:34] * bd808 goes to look at gitlab admin screens [23:14:38] @Danny: I can see your account pending approval in the gitlab admin area, but the bot is not seeing you. I would like to try and debug the bot if you can live without access for a bit longer. [23:15:27] Okay. Considering my needs at the moment I can accept that. [23:15:52] That being not having access to a bit longer to allow you to hopefully identify the root cause here. [23:16:32] legoktm reported T356097 a couple of days ago which may be related. [23:16:32] T356097: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 [23:16:56] * bd808 has been away and no caught up on his bug mail yet [23:18:14] Seems related to the script not running. [23:18:35] But if you can get it to run and it does not see me, that might be a separate issue. [23:19:15] Or maybe my username just broke the script :p