[00:32:02] !log bd808@tools-bastion-12 tools.containers Built new tool-containers/bnc:latest container from tag:v1.9.0 (c14004bb) [00:32:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.containers/SAL [01:06:03] !log bd808@tools-bastion-12 tools.bridgebot Restarted bnc job to pick up tool-containers/bnc v1.9.0 [01:06:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [03:14:11] The actual build is generally around 300 MB but the dependency during the build is around 1.4 GB. If I build the thing on github action, I usually cache the dependency and only push the built version (~300 MB) to the server using rsync. It includes only the required runtime dependencies (`standalone` build). But that standalone build requires node22. [03:14:12] Now that I do not have node22 prebuilt into toolforge, I was trying to build a new image using toolforge build service (what I previously did using github action) `--use-latest-versions`, all the build time dependencies are downloaded then the thing is built. But after the build process, the build service tries to cache the dependencies itself (a smart choice in [03:14:12] most of the cases [03:14:13] ). Since the dependencies are around 1.4 GB, it is failing. [03:14:15] This is my hypothesis and the most sensible answer I have in my mind right now. (re @jeremy_b: 1.4GiB sounds large. does it need to be that big?) [03:47:43] you may want to put in a quota increase request then. see the link from bd808 [10:03:20] How can I run build service to choose different branch then the default branch? `--ref` seems like not working. I wanted to build this `build` branch (https://github.com/nokibsarkar/campwiz-frontend/tree/build) as a hacking attemp to circumvent the storage issue [10:09:26] `--ref` should work, I’m pretty sure I used it a week or two ago… [10:11:29] I am using `toolforge build start -L https://github.com/nokibsarkar/campwiz-frontend/ --ref build` and https://github.com/nokibsarkar/campwiz-frontend/blob/build/package.json file has no scripts with the command `next build`. But on the build process, I see the `next build` command running. [10:11:49] @lucaswerkmeister [10:12:34] the `next build` command is available at the main branch, not the `build` branch. [10:31:24] strange, it really seems to be checking out the main commit (eedb483e18247866b8544140bb4b4096da12ecdc) [10:31:28] (I tried it in a test tool) [10:31:38] no idea why… probably worth a Phabricator task if nobody else here figures it out [11:04:32] hmm maybe --ref is broken? I'm looking at the function here and it looks wrong to me https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/blob/main/internal/build.go#L784 [11:07:30] unrelated to --ref, but still related to campwiz, I'm not understanding why the harbor quota is still 851 [11:07:43] even if all images have been deleted, and GC has run [11:07:45] https://tools-harbor.wmcloud.org/harbor/projects/26921/repositories [11:08:44] * dhinus lunch [14:29:06] lucaswerkmeister: @nokibsarkar I created T394516 [14:29:07] T394516: [builds-api] Main branch is used even when a different "ref" is specified - https://phabricator.wikimedia.org/T394516 [14:29:26] I was just going to create this [14:29:54] the "quota" issue seems to have resolved by itself, maybe it just needs some time [14:30:02] thanks! [14:32:42] DENIED: adding 1.4 GiB of storage resource, which when updated to current usage of 457.6 MiB will exceed the configured upper limit of 1.0 GiB. [14:33:25] This is because I cannot switch to build branch (which might reduce it to ~400MB). [14:33:41] Should I apply for quota increase? [14:34:12] by the way, why current use of 457MB when I did not even do anything? [14:34:46] FWIW I’ve had issues with mysterious phantom quota usage before https://wm-bot.wmcloud.org/browser/index.php?start=01%2F27%2F2024&end=01%2F27%2F2024&display=%23wikimedia-cloud [14:35:09] it feels like some storage space just hangs around and eventually becomes free later :/ [14:35:25] (but that was a year ago, I haven’t “stress-tested” the build service since then) [14:35:28] but some minutes ago, it was 0 [14:36:04] Now it is [14:36:06] Registry [14:36:07] =================== [14:36:09] Storage [14:36:10] ----------- [14:36:12] Available 566.39Mi [14:36:13] Capacity 45% [14:36:15] Limit 1.00Gi [14:36:16] Used 457.61Mi [14:36:18] ok that’s very weird :( [14:36:25] My build even did not succeed [14:36:29] I guess that was in fact more like three weeks ago, before https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/commit/fff150a384202af2d6f4f53ca8f6242d1b2e3b51 (re @lucaswerkmeister: --ref should work, I’m pretty sure I used it a week or two ago…) [14:37:51] toolforge build clean [14:37:52] NOTE: This will remove all your built images to clean up space, you'll have to start a new build to create a new image before being able to restart any running webservice/job or start a new one, are you sure? [y/N]: y [14:37:54] Deleted 0 artifacts from harbor repository tool-campwiz/tool-campwiz [14:38:10] Deleted 0 artifacts, meaning nothing to delete [14:38:41] But still `toolforge build quota` gives the same 45% usage [14:45:23] the fact that after "build clean" the quota is not zero is expected (it needs to wait some automatic garbage collection) [14:46:03] I would suggest applying for a quota increase, something like 5GB or even 10GB should be fine [14:46:25] yeah, but after i start a build, but the build ends with error, should not consume any space. [14:47:09] @nokibsarkar yeah not sure about that, maybe the build starts to upload SOMETHING, then fails, not really sure [14:56:39] I applied for 3GB conservatively at T394520 [15:12:57] I have a solution for this, but how can I push it? (re @wmtelegram_bot: T394516: [builds-api] Main branch is used even when a different "ref" is specified - https://phabricator.wikimedia.or...) [15:17:19] you should be able to create a fork of the repo, and submit a Merge Request [15:20:42] done at https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 [15:22:47] Any gerrit review thing? [15:23:28] Simply grepping does not check for word boundary, so "main" would possible also match "mainabc". (re @nokibsarkar: done at https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132) [15:23:51] Probably you could add the ref as an argument to git-ls-remote? [15:23:51] @nokibsarkar nope, Toolforge components are in GitLab, not Gerrit [15:51:46] Error: unexpected status from POST request to https://toolsbeta-harbor.wmcloud.org/v2/toolforge/builds-api/blobs/uploads/ (https://toolsbeta-harbor.wmcloud.org/v2/toolforge/builds-api/blobs/uploads/): 401 Unauthorized [15:52:47] how can i solve this issue? [15:56:39] is that in the pipeline? that's normal, because your fork doesn't have the required secrets [15:57:00] I will run a pipeline from the parent project when all the comments are resolved [15:57:10] thanks for working on this by the way! [15:57:10] yes (re @wmtelegram_bot: is that in the pipeline? that's normal, because your fork doesn't have the required secrets) [15:57:42] I think if you fix the "nil" comment, then it looks good to go! [15:57:46] how did you end up at toolsbeta? (re @nokibsarkar: Error: unexpected status from POST request to https://toolsbeta-harbor.wmcloud.org/v2/toolforge/builds-api/blobs/uploads/: 401 U...) [15:58:14] @jeremy_b the pipeline for toolforge components pushes images to toolsbeta first, for testing [15:58:18] and to "tools" only after merge [15:59:16] dhinus, sure makes sense. also was a bit out of order was underground with limited connectivity. [15:59:57] I think nil comment is like a opinion based. Because, we already checked if err is not a nil, then return that err. The next line, err must be nil. So, why not be explicit? Otherwise, we can just remove the previous check altogether. (re @wmtelegram_bot: I think if you fix the "nil" comment, then it looks good to go!) [16:03:59] let's discuss in gitlab so others can see the discussion [16:04:24] I'm double checking the code [16:05:36] I see what you mean now [16:06:12] Yeah. And I love being explicit (python's philosophy: explicit is better than implicit) 😁 [16:07:12] running the pipeline now [16:08:59] in the meantime, can you please update the merge request title to describe the change, and not the issue? [16:09:23] if you look at the previous MRs you can see some examples [16:11:07] Oh. That thing. 🤣. I was confused what I should put. [16:11:32] no worries, it can be confusing when it's the first time you are contributing! [16:13:05] we also generally squash all commits to a single commit and "push --force" to the MR branch [16:14:05] that's just a practice that we use in the team, other teams prefer multiple commits [16:14:15] the pipeline is green! [16:14:36] In github, I usually squash all things during merging to the main branch. [16:15:25] that's what I used to do in my previous company... but in toolforge we decided to squash before merging [16:15:54] this is not a big deal, but just to let you know of what is expected for toolforge repos [16:16:12] I used a combination of web + vs code for this patch. So, I am not sure what to do now. What should be the command to sqhash the commit? [16:17:16] we usually just "push --force" after every change, instead of pushing many commits. now that you already have many commits you can use "git rebase -i main" and replace "pick" with "squash" for all commits except the first [16:18:06] other methods here: https://stackoverflow.com/a/25357146/1765344 [16:19:04] "git reset --soft main" is probably the easiest [16:19:42] followed by "git add" and "git commit" to create a single commit that will replace your old ones [16:20:32] A storm is going on in my area, so it is possible for network outages right now. [16:21:38] I will also go offline soon, we can continue this on Monday, I'd also like dcar.o to double check this MR before it's merged and deployed. [16:22:12] if you want to request the quota increase today, you might still have time for someone in the US to apply that [16:22:28] I already requested and someone gave me +1 (re @wmtelegram_bot: if you want to request the quota increase today, you might still have time for someone in the US to apply that) [16:22:52] This request (re @nokibsarkar: I applied for 3GB conservatively at T394520) [16:23:05] nice, I didn't see that! [16:31:04] andrewbogott will look into applying that quota change ^ [16:33:01] Is the title proper? [16:37:04] I was replying about the title in GitLab :) [16:37:17] it's actually tricky to find a good one here! [16:37:40] I gave one [16:38:27] yep, I was trying to come up with an improvement :) [16:38:48] I replied in the MR [16:38:53] Ok [16:40:39] have you managed to squash the commits? I can also do it if you prefer [16:43:15] I think I just squashed them all [16:43:19] the subject looks good [16:45:06] Yep. Squashed them all. [16:48:33] thanks, approved. I think this can get merged on monday unless somebody adds additional comments. [16:49:18] Yaaay!!! Thanks for the assistance 🥰🥰 [16:49:41] thanks again for your contribution! I hope you'll send more in the future :) [16:54:47] @nokibsarka the quota has been bumped to 4GiB [16:55:33] * dhinus offline, have a good weekend! [16:55:51] * andrewbogott waves [17:31:31] requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/api/v1/namespaces/tool-campwiz/pods/campwiz-756bfc9d8b-bzzt7/log?container=webservice&follow=False&pretty=True×tamps=True [18:02:10] nokibsaarkar: \o hi, where/when/how are you getting that error? (looks like maybe `toolforge webservice logs`? though I just tried on your tool `campwiz` and it worked) [18:04:01] u sure it worked? in my logs it is showing 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] node:internal/modules/cjs/loader:1228 [18:04:01] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] throw err; [18:04:03] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] ^ [18:04:04] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] [18:04:06] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] Error: Cannot find module '/workspace/server.js' [18:04:07] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] at Function._resolveFilename (node:internal/modules/cjs/loader:1225:15) [18:04:09] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] at Function._load (node:internal/modules/cjs/loader:1055:27) [18:04:10] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] at TracingChannel.traceSync (node:diagnostics_channel:322:14) [18:04:12] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] at wrapModuleLoad (node:internal/modules/cjs/loader:220:24) [18:04:13] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:170:5) [18:04:15] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] at node:internal/main/run_main_module:36:49 { [18:04:16] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] code: 'MODULE_NOT_FOUND', [18:04:18] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] requireStack: [] [18:04:19] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] } [18:04:21] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] [18:04:22] 2025-05-16T17:58:08+00:00 [campwiz-756bfc9d8b-bzzt7] Node.js v22.14.0 [18:04:32] where does my built code resides? [18:04:37] that looks like nodejs missing some modules, not k8s showing errors [18:04:50] the built code is inside the image, under /workspace [18:05:59] oops, i need to have an adapter then [18:06:26] because in this case it must run .next/standalone/server.js file [18:07:42] you can have a script defined in package.json with the path, then call that `npm run `, like `npm run ontoolforge` or something [18:07:57] That patch I uploaded would REALLY help me from patching things up. Because in this transition period, I still need to ensure the tools works on the WLF sites as well as toolforge [18:08:21] https://www.irccloud.com/pastebin/xmlwjX1H/ [18:08:25] that works yes [18:08:52] what do you mean with "the patch I uploaded"? (just joined the chat xd) [18:08:59] bash: node: command not found (re @wmtelegram_bot: you can have a script defined in package.json with the path, then call that `npm run `, like `npm run ontool...) [18:09:26] this thing (re @wmtelegram_bot: T394516: [builds-api] Main branch is used even when a different "ref" is specified - https://phabricator.wikimedia.or...) [18:11:43] let me check it out [18:15:26] testing it xd, what a silly bug [18:26:44] +1, merging and deploying (will take a few minutes) [18:33:04] nokibsarkar: btw. this is what I meant with adding a script entry in package.json: https://github.com/nokibsarkar/campwiz-frontend/pull/178/files [18:33:19] (no need to merge if you prefer not to do it, it's not needed) [18:34:16] oooo. my bad. sorry. I would check both of the versions, just in case, you know [18:35:02] np :), I'm deploying the changes to the builds service on toolsbeta first, give it ~15min to deploy on tools, I'll let you know here [18:43:26] I tested that PR that I sent in my local dev environment and it starts up :) [18:43:29] https://usercontent.irccloud-cdn.com/file/rUiiyXzy/image.png [18:43:43] not sure there the hostname there comes from (looks like a commit hash?) [18:44:03] aaahh, it's the container name xd, nm [18:44:27] the patch deployed? [18:44:38] on my local environment, not on tools yet [18:44:58] oh ok [18:45:05] (that is using both the patch to builds service you sent, and the patch to the package.json I sent to the frontend code) [18:45:32] oh ok [18:46:01] tests are clean on toolsbeta, deploying on tools [18:46:07] looks like i am the guy with all issues [18:46:52] sorry for that :), it's really appreciated that you are patient enough to work with us on getting them solved [18:47:13] nokibsarkar: okok, you should be able to use `--ref` now without issues [18:48:25] gtg. but will be around a bit later, let me know how it goes (and/or if you find more issues) [18:49:15] oh sure. i am running the build with --ref now [18:51:09] oh, and extra thanks for the patch 👍 [19:00:03] Seems like everythings smooooooth like butter. Thanks everyone for the support in this migration process. Now, I want to sign up for your CI/CD beta program [19:00:47] I mean, a long time ago francesco told me about this [19:03:00] And, because of thsi --ref flag, I only need 300 MB as my space not even 1 GB [19:03:48] I think I was referring to the GitLab CI/CD, you can already use it for any repo in GitLab, no need to “sign up” 🙂 (re @nokibsarkar: I mean, a long time ago francesco told me about this) [19:04:13] Oh. Is that like github action? [19:04:18] Yes, similar [19:05:09] my current implementation >> https://github.com/nokibsarkar/campwiz-frontend/blob/main/.github/workflows/deploy.yaml [19:05:20] would this be achievable? [19:06:46] I am pushing two targets simultenously. one is on digitalocean by WLF. another is toolforge now. And the workflow is very messy right now [19:06:51] I think so but you it will take a bit of work. I would also consider moving some parts to the “toolforge build”, instead of pushing the assets to Git. But there might be downsides to that. [19:07:31] Actually, pushing to another branch was the actual magic behind such a low space requirement + node 22 support [19:07:54] Well Node22 is supported now, the low space might also no longer be a problem with the extra quota… [19:07:57] I couldn't have both at the same time [19:08:15] sure? since when? (re @dhinus: Well Node22 is supported now, the low space might also no longer be a problem with the extra quota…) [19:08:47] tools.campwiz@tools-bastion-13:~$ toolforge webservice node22 shell [19:08:48] type must be one of: [19:08:49] bookworm [19:08:51] * buildservice [19:08:52] * jdk17 [19:08:54] * node16 [19:08:55] * node18 [19:08:57] * perl5.32 [19:08:58] * perl5.36 [19:09:00] * php7.4 [19:09:01] * php8.2 [19:09:03] * python3.11 [19:09:04] * python3.9 [19:09:06] * ruby2.1 [19:09:07] * ruby2.7 [19:09:09] * ruby3.1 [19:09:10] * tcl8.6 [19:09:12] tools.campwiz@tools-bastion-13:~$ [19:09:13] not yet supported [19:09:17] Yes so Toolforge has two ways of running apps: using one of those images, or building your own image [19:09:24] If you build your own image with “toolforge build”, you can use Node 22 since yesterday [19:09:42] oh, yeah. my current implementation [19:10:09] first i build my code using node22 and push to another branch [19:11:02] Yes so that’s the part that I thought maybe can be simplified, but you’ll have to do some tests… You can also keep on doing this, but with GitLab CI. It should work. [19:11:37] then, from that branch, I start toolforge build service. But in that branch, everything is compiled, so nothing to compile for the build service except echo "Hello World'". But I successfully tricked the build service into thinking it is compiling an image in node 22 [19:12:22] That's how I cheated my way into saving space + node 22 simultenously [19:12:32] 🤣 [19:12:49] I have to go, but feel free to experiment with GitLab CI, you just need to add a .gitlab-ci.yml file [19:12:58] Ok [19:13:12] Thank you very very much [19:13:27] Hi bro [19:13:49] What ups