[06:26:43] Hi, is there a way I can run a grpc server on toolforge? [07:51:44] nokibsarkar: you can run it as long as it's not exposed to the internet (you can use it as a backend for another service in toolforge). We only allow http services to be exposed (through an https proxy). What is it that you are trying to do? [07:58:28] I have a backend (http), that needs another grpc server for task management (no, I am not using toolforge jobs yet). SO, yeah, it would work even if it is not exposed to the internet (as long as my http backend can access it) [07:58:56] @dcaro [08:01:33] nokibsarkar: then you can do so by running a continuous job with `--port XXX` and addressing it from the http server as `grpc://:` see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Configuring_internal_domain_names_for_your_jobs [08:04:07] nokibsarkar: just updated a bit the docs there https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Creating_internal_services [08:13:26] !log toolsbeta renew expiring Puppet CA cert [08:13:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:14:26] !log project-proxy renew expiring CA cert T392792 [08:14:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL [08:14:30] T392792: project-proxy puppetserver CA about to expire - https://phabricator.wikimedia.org/T392792 [12:46:23] Hello everyone. Is there a way to know if I am running in Toolforge when I run a bot? [12:48:39] depends a bit on the environment (e.g. on the bastion and in a non-buildservice web service, `/etc/wmcs-project` contains “tools”, but that’s probably not the case in a build service tool) [12:48:54] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars is probably the best solution, just define an environment variable THIS_IS_TOOLFORGE=y or whatever ^^ [12:52:42] There were a doc here: https://wikitech.wikimedia.org/wiki/Obsolete:Toolforge/FAQ#How_can_I_detect_if_I.27m_running_in_Cloud_VPS.3F_And_which_project_.28tools_or_toolsbeta.29.3F [12:54:06] Can we write to the official file about testing whether or not it runs in Toolforge? [13:01:36] that page says it’s obsolete, so probably not a good idea to edit it now [13:03:00] +1 for the envvars solution, might depend on what you want to do in detail though [13:04:39] I maintain a library, so I would like to have a solution that works for all bots. [13:05:16] you want it to know when it's running in the bastion too? [13:07:03] you can rely on the envvar `TOOL_TOOLFORGE_API_URL` being set everywhere that a job/webservice runs (in k8s, not manually in the bastion) [13:09:36] huh, I didn’t see that one in `toolforge envvars list` [13:09:51] Thank you. Well, It seems no `TOOL_TOOLFORGE_API_URL` @dev.toolforge.org? [13:11:39] correct as dcaro said above, you should not be running any application code on the bastion in the first place [13:13:13] lucaswerkmeister: yep, those don't show up there, I'll add a task to show them too (I actually wanted when I added them, but I forgot 😞) [13:13:29] i thought we had a task for that already [13:15:08] maybe.. .let me search [13:16:38] I don't see it :/, I'll create one, I'll merge if/when I find the old one [13:18:11] Thank you. So the official detection method is to test `/etc/wmcs-project`? [13:21:57] nope, test for the envvar `TOOL_TOOLFORGE_API_URL`, the `/etc/wmcs-project` might not exist for some jobs [13:27:03] Thank you [13:30:04] I added a bit more info on the global vars here https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars#Globally_set_environment_variables [13:30:24] and the faq on how to know if you are running on toolforge https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars#How_to_know_if_my_code_is_running_inside_toolforge? [13:33:02] kanashimi: yw! let us know how it goes! [13:34:21] Thank you. I'll test the code. [17:07:12] Can anyone tell me what am I doing wrong here >> https://github.com/nokibsarkar/campwiz-test/blob/main/Procfile .This file (before the new upgrade to use latest buildpack) was running fine. I even got output back then. But after that upgrade, I am not able to run any golang code (even without `--use-latest-versions` flag). What do I have to change? [17:09:46] Now it is showing `2025-05-15T17:02:43+00:00 [campwiz-backend-758dbffbf8-gm69k] sh: 1: ./campwiz-test: not found` [17:43:08] nokibsarkar: you should use just `sh -c 'GIN_MODE=release campwiz-test' [17:43:25] the binary the buildpack builds is in the path, but not in the current directory [17:43:46] https://www.irccloud.com/pastebin/rmS1BCe8/ [17:43:57] (running your image on my local laptop) [17:44:42] I don't have the necessary config xd [17:44:44] https://www.irccloud.com/pastebin/v5kPS2lf/ [17:53:36] nokibsarkar: just tested https://github.com/nokibsarkar/campwiz-test/pull/3 on one of my tools, it seems to be starting the process (fails to load the config, as I don't have any) [17:53:41] https://www.irccloud.com/pastebin/ZWD2L1qa/ [18:22:58] thanks, it is working. But I am facing another issue. I need a file named `.env` (and another `.env.yaml`). I created those files on tool homepage `~` location. Now, if I run `toolforge webservice buildservice restart --mount all`, it cannot access my `.env` and other files. (re @wmtelegram_bot: nokibsarkar: you should use just `sh -c 'GIN_MODE=release [18:22:58] campwiz-test') [19:14:19] nokibsarkar: how are you starting the tool? I don't see any image built with that PR code, so I rebuilt it in your tool (`campwiz-bot`) `toolforge build start https://github.com/nokibsarkar/campwiz-test.git -L`, then restarted the webservice, and the webservice came up [19:14:46] it says `Database connection successful`, I'll stop it so you can start it whenever you want though [19:16:24] nokibsarkar: I left a file `/data/project/campwiz-bot/dcaro.whatidid.2025-05-15.txt` with the commands I used in case it helps [19:19:12] So, my actual code is currently using `toolforge build start https://github.com/nokibsarkar/campwiz-backend.git --use-latest-versions` after `become campwiz-backend`. (re @wmtelegram_bot: nokibsarkar: I left a file `/data/project/campwiz-bot/dcaro.whatidid.2025-05-15.txt` with the commands I used in case it...) [19:35:39] I'm not in front of a laptop anymore, but it might be the way that code looks for the config file, or the config file on that project, the campwiz-test one is working :/ [19:38:11] actually, campwiz-test used `viper.AddConfigPath("/data/project/campwiz-bot")` to add that path into consideration. But I was thinking, this is just a hack, not a solution. (re @wmtelegram_bot: I'm not in front of a laptop anymore, but it might be the way that code looks for the config file, or the config file on...) [19:39:13] You can use the environment variable $TOOL_DATA_DIR instead of hardcodding the path there [19:39:43] Is that what you mean with "solution"? [19:40:31] nope, but close. I was thinking `/workspace` would be as `/data/project/toolname` like solutions [19:48:43] /Workspace only has the source code for the app though [19:49:14] So, I cannot access any other files? [19:49:47] any other configuration files? (I need two .env type and one public/private key pair) [19:50:04] Yes, under $TOOL_DATA_DIR you have the tool home directory [19:50:34] That is not a very trivial solution, but ok. [19:52:41] You can use envvars too instead of .env file (viper has great support for it) [19:54:24] I am thinking that [19:58:08] File based config doesn't work very well with the 12-factor envvar focus of buildpack based deployments. [19:59:34] But it is not very easy to organize complex configuration as yaml or json [20:01:05] It basically becomes a tooling problem on the app side. I run some tools that have good support for mixing public config files delivered as part of the build with private envvars delivered at runtime, but it takes work on the app side to support such systems. [20:06:48] in my opinion, it's not that much worth it. But that's just my opinion. Configuration usually gets complex and requires various namespacing and it's not that feasible to make a flat list and just use envars [20:08:20] how can i see any error log? [20:09:00] how much of that is secret? (re @nokibsarkar: in my opinion, it's not that much worth it. But that's just my opinion. Configuration usually gets complex and requires various ...) [20:11:33] about 50-60% [20:11:53] sounds like an outlier [20:12:35] `toolforge webservice --mount=all buildservice logs -f` shows my tool is running on port 8000 but https://campwiz-backend.toolforge.org/ is not giving any response [20:13:01] https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/blob/main/etc/bridgebot.toml is a complex config for a golang app that also allows envvar specification of secrets. [20:14:12] @nokibsarkar: Try `webservice logs -f` to see the logs [20:14:13] so try curling it? (re @nokibsarkar: toolforge webservice --mount=all buildservice logs -f shows my tool is running on port 8000 but https://campwiz-backend.toolforg...) [20:15:11] no luck yet (re @wmtelegram_bot: @nokibsarkar: Try `webservice logs -f` to see the logs) [20:16:13] @nokibsarkar: I definitely see log output. The latest timestamp is 2025-05-15T20:15:27+00:00 (~30 seconds ago) [20:16:43] 2025-05-15T20:16:24+00:00 [campwiz-backend-5988bdbcf8-rdx8p] [GIN-debug] Listening and serving HTTP on localhost:8000 [20:17:22] it is weird that there are 2 running pods [20:17:26] Then why 404 on https://campwiz-backend.toolforge.org/ (https://campwiz-backend.toolforge.org/) [20:18:30] 'Listening and serving HTTP on localhost:8000' sounds like the app is not binding on 0.0.0.0 (all interfaces) like it needs to [20:18:47] but source does say 0.0.0.0 https://github.com/nokibsarkar/campwiz-test/blob/384d2bac7168625b1db24ac06625b09f4ea3f8a8/main.go#L65 (re @wmtelegram_bot: 'Listening and serving HTTP on localhost:8000' sounds like the app is not binding on 0.0.0.0 (all interfaces) like it ne...) [20:19:06] ok, let me check again [20:19:50] nice spot @jeremy_b. That is almost certainly related [20:19:56] That was a test one. The actual current one is https://github.com/nokibsarkar/campwiz-backend (re @jeremy_b: but source does say 0.0.0.0 https://github.com/nokibsarkar/campwiz-test/blob/384d2bac7168625b1db24ac06625b09f4ea3f8a8/main.go#L6...) [20:21:45] you're porting over something that was already working on another host? [20:22:12] it is already live on https://campwiz.wikilovesfolklore.org [20:22:24] @jeremy_b [20:23:13] But I made it open source and wanted to move on wikimedia cloud service [20:23:23] I have to leave for a bit. github web doesn't allow searching code even just for a single repo without login :( (re @nokibsarkar: That was a test one. The actual current one is https://github.com/nokibsarkar/campwiz-backend) [20:23:24] we are running out of space [20:23:49] find the corresponding line in the current repo (re @jeremy_b: but source does say 0.0.0.0 https://github.com/nokibsarkar/campwiz-test/blob/384d2bac7168625b1db24ac06625b09f4ea3f8a8/main.go#L6...) [20:24:25] yep, i patched it : 2025-05-15T20:23:59+00:00 [campwiz-backend-7d85b566d8-6lt28] [GIN-debug] Listening and serving HTTP on 0.0.0.0:8000 [20:25:19] Now it is working. Yaaaaaay!! [20:25:52] Thanks @jeremy_b . The solution was binding on 0.0.0.0 [20:25:59] still 404. a different 404. (re @nokibsarkar: Then why 404 on https://campwiz-backend.toolforge.org/) [20:26:13] yep. That is ok. [20:26:48] now I need to switch over from the previous backend to this new one on https://campwiz.toolforge.org [20:27:39] Also, I need to migrate all of my data from previous database to new database. And lastly, I need a new OAuth Client [20:49:05] Yaaaa!!!!!. So the tool migration is complete. [20:57:02] Another issue happened, it is due to using node 18, some functions are not available. Can anyone tell me how can I use nodejs 22? [20:59:29] https://packages.debian.org/nodejs [21:00:33] 22 is in experimental. idk if there's a way to influence apt preferences, pins, sources.list [21:02:24] on my local machine (ubuntu), it is the latest version. I am not sure whether I opted for the experimental version or it is stable for ubuntu [21:03:30] https://stackoverflow.com/questions/55510862/how-to-specify-node-version-in-heroku sounds like it’s supposed to take the `engines` in package.json into account (at build time) [21:05:06] I build the nextjs app on github action and push the built version to toolforge. [21:05:42] But if I run `toolforge webservice node22 start`, I get error [21:06:00] but it runs for node 18 [21:11:32] we do not have a node22 container, but the `toolforge build start --use-latest-versions` builder stack's heroku/nodejs builder should support it. [21:15:44] what is the procedure for that? I mean, currently I am putting all my files under `/www/js` and just start the server. [21:22:26] I ran `toolforge build start --use-latest-versions https://github.com/nokibsarkar/campwiz-frontend.git` and it is doing some thingies. how can i provide build-time environment variables? [21:25:54] @nokibsarkar: `toolforge build start --help` and reading https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images will probably help you. [21:32:35] tools-harbor.wmcloud.org/tool-campwiz/tool-campwiz:latest - PUT https://tools-harbor.wmcloud.org/v2/tool-campwiz/tool-campwiz/blobs/uploads/b6b64388-6a8d-4864-bb2f-1cc56092de89?_state=REDACTED&digest=sha256%3Ab0d4f2458916ef3f4cf919e7fbbfb579afd2901a86917a51a92bed9ed99e29a6: DENIED: adding 1.4 GiB of storage resource, which when updated to current usage of 873.5 MiB [21:32:36] will exceed th [21:32:36] e configured upper limit of 1.0 GiB.[step-export] 2025-05-15T21:32:05.535079544Z ERROR: failed to export: failed to write image to the following tags: [tools-harbor.wmcloud.org/tool-campwiz/tool-campwiz:latest: PUT [21:32:36] https://tools-harbor.wmcloud.org/v2/tool-campwiz/tool-campwiz/blobs/uploads/b6b64388-6a8d-4864-bb2f-1cc56092de89?_state=REDACTED&digest=sha256%3Ab0d4f2458916ef3f4cf919 [21:32:37] e7fbbfb579afd2901a86917a51a92bed9ed99e29a6: DENIED: adding 1.4 GiB of storage resource, which when updated to current usage of 873.5 MiB will exceed the configured upper limit of 1.0 GiB.] [21:36:51] @nokibsarkar: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images#Out_of_quota [21:39:49] 1.4GiB sounds large. does it need to be that big? [21:51:50] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 17a80202ae (l10n updates: it) [21:51:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL