[12:36:48] JFYI I've upgraded grafana on cloudmetrics* to 8.5.9 [12:56:18] \o/ thanks! [13:21:38] https://sal.toolforge.org/ is down, can someone take a look? [13:22:05] error.log looks like lighttpd has trouble connecting to some PHP socket, if I’m not mistaken [14:12:11] sal.toolforge.org is back, thanks to whoever did something about it :) [14:12:33] spoke too soon, it’s gone again [14:13:31] let me take a look [14:13:38] works for me [14:14:27] I got 500 [14:14:45] seems intermittent (sometimes I get ok, sometimes 500) [14:17:18] ther's a bot crawling that host it seems [14:17:51] semrush or something [14:20:16] hmpf haproxy logs are so hard to decypher xd [14:21:10] tools haproxy does ingress in tcp mode and not http, which is why the logs have very little details, I usually use the front proxys access log if I need anything [14:21:25] from there I can only see that 500 is returns [14:21:27] *returned [14:25:13] I guess that I can just restart it (it seems that's what was done in the past) [14:26:42] !log tools.sal Restart to clear errors reported on irc [14:26:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL [14:27:05] that worked at least xd [14:27:27] let's see how it evolves... [14:35:29] I don't see any new errors, resuming normal operation xd [14:36:47] thanks ^^ [18:28:04] hola [18:28:59] hello, do you need any help? [18:30:42] yes [18:31:20] yes? [18:35:06] yes may good friend [18:36:04] Hey, I am working the the canasta project https://canasta.wiki/, and I would like to host a wiki on the wikimedia cloud services, but when I tried the Web Proxies, it forwards https requests as http request to the backend server, which causes the wiki not to respond, as it redirects all http requests to https , so giving out a 400 error, when accessed, at https://cliwiki.wmcloud.org/ [18:37:35] @amal_paul: your backend service needs to understand the X-Forwarded-Proto header properly [18:37:46] Guest72: I would suggest asking a question if you need help with something. [18:49:43] ooh Thanks, lemme try (re @wmtelegram_bot: @amal_paul: your backend service needs to understand the X-Forwarded-Proto header properly) [19:25:13] Hmmm [19:25:15] tools.dbreps@tools-sgebastion-10:~/src/database-reports$ git pull [19:25:15] fatal: unable to connect to github.com: [19:25:15] github.com[0: 140.82.113.3]: errno=Connection timed out [19:30:15] same thing again [19:31:17] nvm, for some reason it was set to clone over the legacy git:// protocol which I guess no longer works. Switching to HTTPS fixed it [19:31:40] legoktm: I think they use git@github? [19:31:46] oh, right [19:31:59] they removed support for that a few months ago I believe, yes [19:32:09] Previously it was git://github.com/mzmcbride/database-reports.git - I just switched it to https:// [19:32:29] Not obviously, git:// is a plaintext/insecure protocol [20:13:53] Hi! Have got my tool sort-of-working migrated to Kubernetes on Toolforge with the php7.4 image, but I need two additional binaries (jpegtran and ddjvu). What's a good way of adding them? I'm familiar with building Docker images, but it seems like only the standard images can be deployed at Toolforge? It's possible I can run them directly from the project folder, but I'm not sure if all the necessary shared libraries will be [20:13:53] available in the container [20:15:24] danmichaelo_: you can try building the binaries you want from within a `webservice php7.4 shell` session, but if that does not work you are really at the "my tool needs things that are not yet possible with the new k8s jobs system" place. [20:17:24] Toolforge does not support "bring your own container" and also does not yet have working "build a container for this tool" support. We are working on that second one though with a project to use buildpacks (similar to how the Heroku commercial web hosting service works) [20:17:38] Thx, didn't think of the possibility to build from within the container, will try that! [20:19:13] Seems like there is no c compiler there [20:23:02] hi danmichaelo_, is this about CropTool? some people have been complaining that it is down [20:24:40] Yup! A simple restart fixed the main problem, but cropping DJVU still doesn't work [20:25:01] Forgot to test that after I migrated to Kubernetes [20:25:16] jpegtran seems to work [20:25:33] Trying to compile djvulibre outside the container now [20:29:03] danmichaelo_: hmm... I thought we had compliers in all of the images. Sorry for that bad advice. The ruby27 image has compilers and dev libraries in it and is also based on bullseye, so things compiled from it may work in the php7.4 image. [20:29:16] danmichaelo_: awesome, thanks!! at https://meta.wikimedia.org/wiki/Wikimedia_Forum#CropTool I suggested you should add some more maintainers as a best practice [21:53:46] Will see if I can recruit some [21:58:05] Thx, bd808, compiling in the ruby27 image works. Unfortunately, I needed imagemagick too, so it will take a while [22:20:00] !log tools.stewardbots SULWatcher/manage.sh restart [22:20:06] oh wait [22:34:53] * bd808 wonders why stashbot was caught in a netsplit (at least I think that's what happened...) [22:43:49] bd808: i got a ruby service up and running and doing what I want it to do. I have some questions about how it works, though. [22:44:17] ragesoss: I can try to help explain. What parts are mysterious? [22:44:37] i followed your example, including the part where i did a `bundle install` from within a `webservice shell`. [22:44:57] so, why is that necessary? [22:45:15] what does running something within a webservice shell do? [22:46:34] ragesoss: the main thing that does is ensure that you are using the same version of ruby, bundler, etc in your interactive shell as are being used in the webservice runtime. The bastions may have different versions of the OS and everything else from the Docker containers that run on the Kubernetes cluster. [22:46:34] (also, the version of `bundler` is different in the webservice shell versus what's on toolforge normally, which means i have to do some messing with the bundler setup in order to run the same code outside of a webservice shell, since the different versions of bundler create incompatible gemfiles) [22:47:07] yeah. so you experimentally found out why to use `webservice shell` :) [22:48:25] okay... so it's still using the same filesystem that gets mounted by the container while also being available to the tool account... it's just that you have to do the `bundle install` from the container so that it uses the bundler version of the container. [22:48:33] basically you should not assume that things which work inside a Kubernetes managed container (`webservice ...`, `toolforge-jobs ..`) will also work directly on a bastion [22:49:36] ragesoss: correct. The NFS volume for your tool's $HOME is mounted in both places today which is how you can see the same files. [22:50:35] cool. so, like, instead of running `webservice shell` each time i need to adjust the dependencies, I could just include `bundle install` as part of the webservice start script. [22:51:14] Yeah, you could do that. [22:51:39] okay, i understand enough to stop poking at it now. [22:51:40] thanks! [22:51:51] That will add some variability to your container startup process, but it will probably be ok. [22:52:05] At some point in the future (and there will be lots of announcements about this when we know it's going to happen) that $HOME NFS volume will be replaced with a "persistent volume" that is only seen from the Kubernetes cluster. [22:52:42] sounds manageable [22:53:15] By that time we will also have some "build a custom container for my tool" process that will give each tool a build step that should make things like bundler much easier to deal with [22:55:12] okay. and i assume at some point there will be upgrades to the base containers such that it's got a non-EOL ruby version. guessing that's a Debian problem rather than a WMF problem. [22:55:51] the ruby version didn't give me any trouble, but it does make me slightly nervous running 2.5 [22:56:18] ragesoss: ugh. yeah that's more of a Debian problem, but yes we will keep adding newer versions. [22:56:34] It looks like we should be able to get 2.7 on bullseye right now [22:57:46] https://packages.debian.org/bullseye/ruby is 2.7 and the upcoming https://packages.debian.org/bookworm/ruby will be 3.0 [22:59:17] ragesoss: oh, it looks like we have the ruby27 image out there already. [22:59:42] so you can redo things with 2.7 by changing from ruby25 to ruby27 [23:00:21] okay, i'll try that [23:00:24] * bd808 should try migrating his crusty old test tool [23:07:10] okay, Ruby2.7... success! [23:07:47] gotta remember to stop the webservice before changing the template or ruby version. [23:45:24] !log tools.stewardbots SULWatcher/manage.sh restart [23:45:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL