[00:10:27] <wm-bot>	 !log damian-scripts@tools-bastion-13 tools.cluebotng-staging report deployed @ refs/heads/main
[00:10:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL
[00:10:48] <wm-bot>	 !log damian-scripts@tools-bastion-13 tools.cluebotng report deployed @ refs/tags/v1.1.1
[00:10:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL
[04:16:53] <wm-bb>	 <Xvansic> I will like to free up memory for my tool, I have cleared the build images or deployment build that I triggered but did not go through due to storage limitation. the tool is agpb, and I am unable run any latest deployments.
[07:48:08] <taavi>	 !log toolsbeta-logging bump object storage object quota
[07:48:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta-logging/SAL
[08:25:54] <taavi>	 !issync
[08:25:54] <ircservserv-wm>	 Syncing #wikimedia-cloud (requested by Majavah)
[08:25:56] <ircservserv-wm>	 Set /cs flags #wikimedia-cloud godog +Afiortv
[08:25:58] <ircservserv-wm>	 Set /mode #wikimedia-cloud +b $j:#wikimedia-bans
[09:04:51] <dhinus>	 @Xvansic please open a phab task including more details (commands you are running, errors you are getting, etc.)
[11:49:33] <wm-bb>	 <lucaswerkmeister> I just got a build service failure with an “wget: bad address 'github.com'” error o_O
[11:49:39] <wm-bb>	 <lucaswerkmeister> trying again now, it was probably just random
[11:50:03] <wm-bb>	 <lucaswerkmeister> (it was trying to run `wget -O - https://github.com/pelletier/go-toml/releases/download/v2.1.1/jsontoml_2.1.1_linux_amd64.tar.xz | tar xvJ`)
[11:51:32] <wm-bb>	 <lucaswerkmeister> ok, a rebuild worked 🤷
[12:35:53] <godog>	 yeah seems very much transient
[13:06:52] <godog>	 !log pywikibot update toolforge image to 10.3.2 - T401077
[13:06:53] <stashbot>	 godog: Unknown project "pywikibot"
[13:06:53] <stashbot>	 godog: Did you mean to say "tools.pywikibot" instead?
[13:06:54] <stashbot>	 T401077: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T401077
[13:07:05] <godog>	 yes thank you stashbot that's what I meant
[13:07:11] <godog>	 !log tools.pywikibot update toolforge image to 10.3.2 - T401077
[13:07:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pywikibot/SAL
[14:30:34] <wm-bot>	 !log damian-scripts@tools-bastion-13 tools.cluebotng report deployed @ refs/heads/main
[14:30:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL
[14:30:41] <wm-bot>	 !log damian-scripts@tools-bastion-13 tools.cluebotng report deployed @ refs/tags/v1.2.0
[14:30:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL
[17:30:09] <wm-bb>	 <lucaswerkmeister> apparently https://wdactle.toolforge.org/ is broken because https://wdactle.toolforge.org/assets/index-h2YuXZ3m.js no longer responds with a Content-Type header (so the browser refuses to run the JS)
[17:30:29] <wm-bb>	 <lucaswerkmeister> uwsgi prints a message at startup saying “!!! no /etc/mime.types file found !!!” – does that ring a bell for anyone?
[17:30:46] <wm-bb>	 <lucaswerkmeister> I’m not sure if the file would normally be in the container or mounted it from the toolforge environment
[17:33:27] <wm-bb>	 <lucaswerkmeister> --mount=all doesn’t seem to help either (https://lucaswerkmeister-test.toolforge.org/), that mounts a few other paths in /etc but not this one
[17:36:08] <wm-bb>	 <lucaswerkmeister> can I see my earlier builds somehow, to look what was inside their images?
[17:36:28] <wm-bb>	 <lucaswerkmeister> `toolforge build list` lists some, but the destination_image is …:latest for all of them
[17:36:42] <wm-bb>	 <lucaswerkmeister> and I can’t get harbor to show me the hashes of the previous images (if they still exist)
[17:49:55] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.wdactle deployed 69ac5a1656 (fix MIME types / Content-Type, unbreaking game)
[17:49:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdactle/SAL
[17:50:42] <wm-bb>	 <lucaswerkmeister> for now I work around this by shipping my own mime.types file and configuring it in uwsgi.ini https://gitlab.wikimedia.org/toolforge-repos/wdactle/-/commit/69ac5a1656
[17:50:51] <wm-bb>	 <lucaswerkmeister> but that doesn’t feel like a good long-term solution
[17:51:09] <wm-bb>	 <lucaswerkmeister> (I’ll probably create a phab task later if I don’t get a response here)
[18:20:43] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.wdactle deployed 067068df45 (move guessing area on mobile to bottom again, using interactive-widget=resizes-content)
[18:20:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdactle/SAL
[20:09:38] <bd808>	 @lucaswerkmeister: /etc/mime.types would not have ever been mounted from the k8s exec node. This sounds like a very uwsgi specific assumption problem. A fix might be installing the `media-types` apt package in your container.
[20:10:02] <wm-bb>	 <lucaswerkmeister> I guess I can try that, though I doubt it’ll work
[20:10:10] <wm-bb>	 <lucaswerkmeister> unless the apt buildpack symlinks it into the real /etc
[20:10:15] <bd808>	 oh... right because it will be in a weird palce
[20:10:22] <wm-bb>	 <lucaswerkmeister> (if I recall how that works correctly
[20:10:23] <wm-bb>	 <lucaswerkmeister> )
[20:10:25] <wm-bb>	 <lucaswerkmeister> yeah
[20:11:06] <wm-bb>	 <lucaswerkmeister> I assume the package must’ve been installed before, somehow
[20:11:15] <bd808>	 is there a particular reason you are using uwsgi as the http service there? You are already behind 2 layers of nginx
[20:11:36] <wm-bb>	 <lucaswerkmeister> (I thought it might be a trixie thing but apparently it’s actually ubuntu ^^ hence https://wikis.world/@LucasWerkmeister/115113327365492927)
[20:11:38] <wm-bb>	 <lucaswerkmeister> no particular reason
[20:11:43] <wm-bb>	 <lucaswerkmeister> I think it was recommended on wikitech at some point?
[20:12:24] <wm-bb>	 <lucaswerkmeister> can the 2 layers of nginx serve static files inside the container for me?
[20:13:03] <wm-bb>	 <lucaswerkmeister> aha, https://gitlab.wikimedia.org/toolforge-repos/wdactle/-/commit/d4b39b68a6 says I stole that from codex-playground, let me look at that
[20:13:10] <bd808>	 yes the base images for buildservice are Ubuntu. Either 22.04 or 24.04 depending on your use of `--use-latest-versions`
[20:13:42] <wm-bb>	 <lucaswerkmeister> yeah I used --use-latest-versions now because of some vite error probably related to node versions
[20:13:50] <wm-bb>	 <lucaswerkmeister> maybe last time I wasn’t using that yet and ended up on ubuntu 22.04
[20:14:19] <wm-bb>	 <lucaswerkmeister> aha, https://gitlab.wikimedia.org/toolforge-repos/codex-playground/-/commit/039d20f155 points to https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_static_tool#Step_2:_Install_and_configure_uwsgi_to_serve_static_files as the place where I got uwsgi from
[20:16:47] <bd808>	 https://www.npmjs.com/package/node-static would be a different option for serving the files.
[20:17:19] <wm-bb>	 <lucaswerkmeister> at some point I’m hoping to add SSR to that tool anyways so at that point it’ll become node-based
[20:17:27] <wm-bb>	 <lucaswerkmeister> /me looks where node-static gets MIME types from
[20:17:47] <bd808>	 I would assume some mess of npm dependencies :)
[20:18:04] <wm-bb>	 <lucaswerkmeister> yeah, starts at https://www.npmjs.com/package/mime
[20:18:17] <wm-bb>	 <lucaswerkmeister> “…and I show you how deep the rabbit hole goes.”
[20:19:41] <wm-bb>	 <lucaswerkmeister> ok that’s actually built in, doesn’t rely on /etc/mime.types https://github.com/broofa/mime/blob/main/types/standard.ts
[20:37:49] <wm-bb>	 <lucaswerkmeister> okay, I’m experimenting with https://gitlab.wikimedia.org/toolforge-repos/lucaswerkmeister-test/-/commit/ff2647a127
[20:38:15] <wm-bb>	 <lucaswerkmeister> without --use-latest-versions, it builds tools-harbor.wmcloud.org/tool-lucaswerkmeister-test/tool-lucaswerkmeister-test:latest@sha256:68b64dfcdc45f76c7b39df39114ef72c8c9e2be641a2a5c76d564786993fbda6 and shows a successful test at https://lucaswerkmeister-test.toolforge.org/
[20:38:29] <wm-bb>	 <lucaswerkmeister> and I can see in a local `docker run` that there’s an `/etc/mime.types` file in there
[20:38:42] <wm-bb>	 <lucaswerkmeister> from the `media-types` apt package
[20:38:47] <wm-bb>	 <lucaswerkmeister> (on Ubuntu 22.04)
[20:39:01] <wm-bb>	 <lucaswerkmeister> now doing another build with --use-latest-versions…
[20:40:21] <wm-bb>	 <lucaswerkmeister> that built tools-harbor.wmcloud.org/tool-lucaswerkmeister-test/tool-lucaswerkmeister-test:latest@sha256:b159906da7d2936fce6ef9a23e6e775680938129459ca10607d70e95fd87700c, and… the test at https://lucaswerkmeister-test.toolforge.org/ is still successful?
[20:40:28] <wm-bb>	 <lucaswerkmeister> firefox says: The script from “https://lucaswerkmeister-test.toolforge.org/index.js” was loaded even though its MIME type (“”) is not a valid JavaScript MIME type.
[20:40:29] <wm-bb>	 <lucaswerkmeister> o_O
[20:40:34] <wm-bb>	 <lucaswerkmeister> why didn’t it do that in wdactle then
[20:41:02] <wm-bb>	 <lucaswerkmeister> (maybe `type="module"`? does it only run mime-less JS for legacy `<script>` tags without that attribute?)
[20:41:42] <wm-bb>	 <lucaswerkmeister> but anyway inside that image there’s no mime.types anymore. so it is indeed due to --use-latest-versions
[20:42:09] <wm-bb>	 <lucaswerkmeister> IMHO that’s still worth a phabricator task, I doubt my tool was the only one relying on `/etc/mime.types` without being aware of it
[20:45:53] <wm-bb>	 <lucaswerkmeister> strace suggests gunicorn also uses `/etc/mime.types` FWIW
[20:46:17] <wm-bb>	 <lucaswerkmeister> along with a bunch of other candidate files, see `knownfiles` in `/usr/lib/python3.13/mimetypes.py` (adjust for your python version)
[21:23:54] <wm-bb>	 <lucaswerkmeister> filed T403303 for the issue
[21:26:40] <wm-bb>	 <lucaswerkmeister> and regarding my earlier question – it looks like Harbor throws away all the images but the latest one, so finding the hashes / tags of any older build images wouldn’t have helped me
[21:27:02] <wm-bb>	 <lucaswerkmeister> (I tried running both the old and the new image built from the reproduction steps in the task description and found that the old image was already gone again)