[04:55:14] mutante: first we need to get it hosted in the main k8s cluster. We also need to finish making sure that the new idm can handle all of the needed Developer account management tasks. Striker needs to be updated to talk to the idm instead of wikitech. A few more custom apps will need updating too. And then we can start figuring out how to do the LDAP to SUL migration. At current ra [04:55:14] tes of change… 3 years? [10:33:27] If I register 14 daily jobs (most of them run in less than 20 minutes), will it work? Should I make sure that they start at different times of day or @daily is enough and the system will make sure that I don't exceed my quote of max 10 jobs at the same time? [11:16:39] Hey! Acorns makes it easy to save and invest. Join me and you'll get a free $5 investment! As a perk of referring users, I can get a reward too. Learn more and see terms here. https://share.acorns.com/timothymcgrawpvt?advocate.partner_share_id=2595012575381532429 [11:20:11] my guess would be that `@daily` just translates to midnight UTC, so it’s probably better to spread it out manually [11:21:49] thanks (re @lucaswerkmeister: my guess would be that @daily just translates to midnight UTC, so it’s probably better to spread it out manually) [11:22:21] ah, no, apparently it is actually randomized? that’s good https://gerrit.wikimedia.org/r/c/cloud/toolforge/jobs-framework-api/+/927587 T331684 [11:22:35] (if you search for “toolforge midnight” on phabricator you can find a bunch of related tasks ^^) [11:27:13] yeah I just tried it out and it translated `@daily` to `16 20 * * *` for me [15:03:08] in `toolforge jobs run`, does `--wait` or `--mount all` imply `--no-filelog` by any chance? [15:03:12] I’m running the command [15:03:13] `toolforge jobs run --wait --image tool-lucaswerkmeister-test/tool-lucaswerkmeister-test:latest --mount all --command setup-build-rdf2graph -o setup-build-rdf2graph.out -e setup-build-rdf2graph.err setup-build-rdf2graph` [15:03:23] (in the `lucaswerkmeister-test` tool) [15:03:30] and the .out or .err files are nowhere to be found [15:34:03] Are there a toolforge bastion host running python 3.9 or newer? [15:37:46] python3.11 is available on toolforge-jobs. Why do you need to run in the bastion? [15:37:48] I don’t think so; you should be able to use `webservice python3.9 shell` instead [16:08:35] It's very convenient for testing to be running the same version on the bastion and on the production host. [16:31:43] how do I clear out my storage in harbor? [16:31:54] I already `toolforge build delete`d all my old images, but still got this error in a new build: [16:32:00] DENIED: adding 236.4 MiB of storage resource, which when updated to current usage of 958.9 MiB will exceed the configured upper limit of 1.0 GiB. [16:37:06] looks like `toolforge build clean` helped [16:37:28] (there’s still 317.19 mysterious MiB used in `toolforge build quota` but that should leave enough space for a new build) [19:31:24] I'm trying to get a celery job running on ``` [19:31:25] link-dispenser``` [19:31:26] , however the logs errored out with this: [19:31:28] ``` [19:31:29] 2024-01-27T19:25:51+00:00 [celery-5fcf4d4947-ngsls] bash: warning: shell level (1000) too high, resetting to 1 [19:31:31] 2024-01-27T19:26:07+00:00 [celery-5fcf4d4947-ngsls] ERROR: failed to launch: exec.d: failed to execute exec.d file at path '/layers/heroku_nodejs-engine/web_env/exec.d/web_env': fork/exec /layers/heroku_nodejs-engine/web_env/exec.d/web_env: argument list too long``` [19:31:37] Any idea what is going on ? [19:33:01] that sounds to me like there’s an infinite loop somewhere [19:33:12] a command trying to run itself or something like that [19:33:26] mainly due to the “shell level (1000) too high, resetting to 1” message [19:33:36] and then “argument list too long” could be because something keeps being appended to the command line [19:35:50] Hmm, the Procfile for the entry is just a shell script that invokes celery https://gitlab.wikimedia.org/toolforge-repos/link-dispenser/-/blob/main/Procfile?ref_type=heads [19:36:22] I'm pretty sure there isn't any infinite loop from my end :( [19:36:35] (Atleast not at the bash layer) [19:37:35] (This is the `start_celery.sh` https://gitlab.wikimedia.org/toolforge-repos/link-dispenser/-/blob/main/scripts/start_celery.sh?ref_type=heads) [19:40:29] There is a shebang at the start of that script, maybe that could be a potential problem somehow ? [19:41:41] hm, and celery is installed as a python package? [19:41:57] Yes via requirements.txt [19:43:43] oh, but I wonder if `celery` in `start_celery.sh` refers to the Procfile command? [19:43:54] yeah, if I run your image in docker locally, `type celery` says “celery is /cnb/process/celery” [19:44:17] so the Procfile entry apparently gets written as a real command to a directory in the $PATH, where it shadows the actual celery you want to run [19:44:30] (`docker run --rm -it --entrypoint='bash' tools-harbor.wmcloud.org/tool-link-dispenser/tool-link-dispenser:latest` if you’re curious) (re @lucaswerkmeister: yeah, if I run your image in docker locally, type celery says “celery is /cnb/process/celery”) [19:44:43] so, you probably need to rename the procfile entry [19:44:56] and I bet this actually caused my problems in T320140 too? :D [19:45:01] let me try that in my tool [19:45:07] (though my error message was different) [19:46:42] Question: how can I get my patches to be verified automatically on Gerrit? 🤔 [19:47:46] Oh I see, what you mean, I'll keep that docker command handy for testing (re @lucaswerkmeister: yeah, if I run your image in docker locally, type celery says “celery is /cnb/process/celery”) [19:47:51] you mean, that CI automatically runs without someone else having to approve? (re @albertoleoncio: Question: how can I get my patches to be verified automatically on Gerrit? 🤔) [19:48:02] yeah it’s fairly useful ^^ (re @sohom_datta: Oh I see, what you mean, I'll keep that docker command handy for testing) [19:48:14] at least until you need to mount toolforge NFS into it or something, then you can’t test locally after all ^^ [19:49:31] Exactly. I wish I could see if there is a error before ask someone to review. (re @lucaswerkmeister: you mean, that CI automatically runs without someone else having to approve?) [19:50:03] for that, you need to be added to the allowlist in the integration config; see e.g. https://gerrit.wikimedia.org/r/c/integration/config/+/989227 or https://gerrit.wikimedia.org/r/c/integration/config/+/986673 [20:01:44] Yay! that fixed the issue :) [20:02:09] \o/ [20:07:30] So... do I self-apply by submitting a patch of myself, or do I need someone to "sponsor" me? (re @lucaswerkmeister: for that, you need to be added to the allowlist in the integration config; see e.g. https://gerrit.wikimedia.org/r/c/integration...) [20:08:37] I don’t know if there’s an official policy tbh [20:08:56] apparently the last few requests were “nominations” by someone else but I also found https://gerrit.wikimedia.org/r/c/integration/config/+/950162 which was uploaded by the person themself [20:09:38] so I think it’s worth a try :) [20:10:05] Kinda does, but it doesn't explain much: [[:mw:Continuous integration/Allow list]] (re @lucaswerkmeister: apparently the last few requests were “nominations” by someone else but I also found https://gerrit.wikimedia.org/r/c/integratio...) [20:10:57] I guess “Being on the list does not require a high level of trust” sounds promising [20:20:42] the threshold is roughly "obviously not a cryptominer". [20:22:23] it does make it easier if you've made contributions to the wikis, or have had a patch or two accepted, or have a toolforge tool, [20:24:50] I might have a little more than that [20:25:04] https://gerrit.wikimedia.org/r/c/integration/config/+/993234 [21:35:47] albertoleoncio: deployed, #wikimedia-releng for CI questions next time [23:49:28] @sohom_datta FYI, I added something about our Procfile issue to the documentation: https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Build_Service&diff=prev&oldid=2143118