[09:55:48] (not sure if my previous message made it trough, repeating...) [09:55:49] Hi, maybe someone here can help me. I would like to know if the changes to the categorylinks table (cl_target_id) are now stable enough to use on all wikis (specifically, the toolforge replicas). [09:55:54] Also, is the change image/oldimage to file/filerevision complete and safe to use, at least on Commons (or the toolforge replica)? [10:01:12] MagnusManske: T385703 suggests that categorylinks is done everywhere, per T385167 file still has some edge cases remaining [10:01:13] T385703: Run the data migration of categorylinks - https://phabricator.wikimedia.org/T385703 [10:01:13] T385167: Run data migration script for file migration - https://phabricator.wikimedia.org/T385167 [10:02:51] Thanks taavi! [11:11:16] cc Amir1 ^ [11:11:24] (in case there’s anything to add) [11:40:19] !log lucaswerkmeister@tools-bastion-13 tools.translate-link deployed f350d5a97f (upgrade dependencies; also updgraded pip and wheel in the venv) [11:40:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [11:44:57] !log lucaswerkmeister@tools-bastion-13 tools.translate-link deployed 44d4a90377 (split dev requirements from prod requirements) [11:44:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [11:48:51] !log lucaswerkmeister@tools-bastion-13 tools.translate-link deployed 56cafa03b2 (restore prod dependency on typing-extensions) [11:48:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [11:52:57] !log lucaswerkmeister@tools-bastion-13 tools.translate-link deployed ed2860c073 (add health-check-path) [11:52:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [14:26:35] !log paws moving all home dirs with files named 'android' into /srv/paws/files-to-remove/ [14:26:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [15:36:16] Hi, need some help regarding volume attachments to VPS instances. [15:36:33] I created a 30G volume and attached it. Horizon says "Attached to /dev/sdb on voterlists-1". However, running df -h on the instance doesn't show any 30G mount. Soft rebooting the instance didn't help either. [15:39:43] sd0001: did you run the `wmcs-prepare-cinder-volume` script or otherwise mount it? (see https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_instances) [15:41:42] taavi: nope, I just used the "Attach volume" option in Horizon instances UI and thought that should do it [15:41:47] will check the help page, thanks [15:42:40] yeah, when you "attach" via horizon it adds the block device to the VM (which is visible in `lsblk`), but it can't automatically create a filesystem and mount it [15:46:38] you can think of "attach volume" as "plug the external drive into the computer." Initially you've got an unformatted device that needs to be initialized and likely the OS needs to be told something about what to do the next time this device is seen attaching. [15:47:03] Excellent analogy [17:10:26] !log tools.translate-link python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SECRET_KEY"]))' | toolforge envvars create TOOL_SECRET_KEY [17:10:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [17:10:32] !log lucaswerkmeister@tools-bastion-13 tools.translate-link commented out config.yaml, should use envvars instead [17:10:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [17:11:01] !log lucaswerkmeister@tools-bastion-13 tools.translate-link deployed eb8352b991 (read config from envvars) [17:11:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [17:11:10] (not that it matters – this tool doesn’t use flask.session anyways :D) [17:20:40] !log tools.translate-link webservice stop && mv www{,-unused-tool-now-runs-on-buildservice} && wget https://gitlab.wikimedia.org/toolforge-repos/translate-link/-/raw/2e2349a9fb/service.template && webservice start [17:20:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [17:26:23] !log lucaswerkmeister@tools-bastion-13 tools.translate-link deployed 3d7149ae3f (try removing prod dependency on typing-extensions again) [17:26:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [17:56:13] !log lucaswerkmeister@tools-bastion-13 tools.translate-link deployed e1efca1601 (migrate CI from GitHub to GitLab) [17:56:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [21:32:11] I'm trying to build a toolforge flask webservice with pywikbot (https://gitlab.wikimedia.org/toolforge-repos/wstranclude) [21:32:11] everything looks all right, until it tries to login [21:32:12] it apparently can't find the user-config.py despite it being in the same directory and the toolforge envvar PYWIKIBOT_DIR pointing to that dir [21:32:12] it keeps throwing up that there's no username and asking me to add a line to user-config that's already there: [21:32:13] "pywikibot.exceptions.NoUsernameError: ERROR: username for wikisource:en is undefined. [21:32:13] If you have a username for that site, please add a line to user config file (user-config.py) as follows: [21:32:14] usernames['wikisource']['en'] = 'myUsername'" [21:34:06] where is the `user-config.py` file? [21:34:19] right next to the app.py file [21:34:26] because according to `~tools.wstranclude/service.manifest` you’re running the webservice with `mount: none`, so it would only have what’s in the built image IIUC [21:34:56] it does have the rest of the files in the same dir, since eg https://wstranclude.toolforge.org/test/ uses app.py [21:35:33] `app.py` would come from the git repo, not the tool home directory [21:35:44] `user-config.py` is also in the git repo [21:35:52] oh. you’re right [21:36:00] sorry, I just assumed it would contain a password and therefore not be in git :D [21:36:12] (the passwords are in envvars) [21:36:17] but it looks like it just converts the environment into Python [21:36:19] 👍 [21:36:26] ok then I’m not sure what’s wrong [21:37:01] oh well [21:37:13] ok, if I do `docker run --rm tools-harbor.wmcloud.org/tool-wstranclude/tool-wstranclude:latest`, I get a message: [21:37:14] > Skipped '/workspace/user-config.py': writeable by others. [21:37:28] so it sounds like pywikibot might be skipping reading the file? [21:37:36] based on approximately the same confusion as I had ^^ [21:37:36] so chmod maybe? [21:37:55] if it talks about write permissions [21:38:11] maybe [21:38:36] let's see [21:38:38] though AFAICT the file is rw-r—r— in git [21:38:48] (ugh, telegram turned `--` into a long dash, disregard that) [21:39:01] so maybe the “writeable” is a misleading error message [21:39:13] not in local: [21:39:13] -rw-rw-rw- 1 heroku heroku 451 Jan 1 1980 user-config.py [21:39:25] not in local (in the webservice venv): [21:39:26] -rw-rw-rw- 1 heroku heroku 451 Jan 1 1980 user-config.py [21:39:26] oh, it is indeed world-writable in the image [21:39:38] (I also just checked with --entrypoint=bash -it on the docker) [21:39:49] maybe heroku does that by default [21:41:13] I… guess you could make an entrypoint shell script that chmod’s before exec’ing gunicorn, and reference that in the Procfile instead of running gunicorn directly? [21:41:18] ugly workaround but it might work [21:41:26] I can’t find any info on heroku making files world-writable at the moment [21:42:06] (but it looks like that might be default heroku behavior, I can also see it in the image of one of my tools – all the python files are -rw-rw-rw-) [21:42:10] ah, but I don't have the permission to remove write permissions live (at least in `toolforge webservice shell`) [21:42:21] hmph [21:42:25] it worked in local dokcer [21:42:36] I guess toolforge mounts the container differently then :S [21:43:37] does the webservice source have more permissions than toolforge webservice shell? [21:43:43] no idea [21:43:52] only one way to find out I suppose [21:44:05] I guess [21:47:45] ... and no [21:49:21] as a side note, just noticed "Skipped '/workspace/user-config.py': owned by someone else." above the great pile of login error stacktraces, in the webservice log. don't know for how long it's been part of the errors [21:50:40] the code that raises the warning is in `/layers/heroku_python/dependencies/lib/python3.12/site-packages/pywikibot/config.py`, and I can’t see an obvious way to bypass it (there’s no IGNORE_WRITABLE_USER_CONFIG envvar or anything like that) [21:51:10] if there’s *no* other solution, I guess you could add a super evil hack to your python code that monkeypatches os.stat() to return a fake mode for this file name before importing pywikibot :S [21:51:11] you sure that the permissions in the git are right? [21:51:57] AFAICT the permissions in git don’t matter, the build process seems to make all Python files world-writable (I can see that much in other tools) [21:52:13] also git doesn’t record unix permissions exactly anyways [21:52:55] I can't be the first person trying to use pywikibot in a flask webservice, can I? [21:53:19] probably not, but I think you might be the first person to do it in a build service tool [21:53:46] eh [21:54:34] could you help with the "super evil hack"? I'm not sure how it'd be done [21:54:37] found the docs, https://git-scm.com/book/sv/v2/Git-Internals-Git-Objects says there are only three modes (normal, executable, symlink) – git doesn’t record if the file was world-writable or world-readable anyways, so there’s not much you could get wrong inside the repo (re @lucaswerkmeister: also git doesn’t record unix permissions exactly anyways) [21:59:15] how would I run code before importing pwb? [22:00:32] one moment… [22:06:57] I think you’d want something like https://gist.github.com/lucaswerkmeister/8e29449688e945da866fd349e4a0757c (untested) [22:07:22] (“modifying” a stat_result turns out to be annoyingly difficult, it has a `__replace__()` method but that doesn’t work) [22:07:49] let's try [22:11:01] I updated the gist with a test file and I think it’s working as I expected [22:11:21] though I still *really* hope someone else will come in here with a better idea for how to solve this :D [22:11:31] indeed [22:13:01] maybe the solution really shoud be asking the pwb people to add a IGNORE_WRITABLE_USER_CONFIG envvar [22:14:21] yeah, if we’re going to have more build service tools (and there’s no way to change how the build service sets file modes) that feels like a better option [22:14:30] possibly stupid question: when `config.py` re-imports `os`, why won't it get th real function? [22:16:03] hang on, the error changed: for the last while it's been saying "Skipped '/workspace/user-config.py': owned by someone else." instead. [22:17:14] probably just giving uid as 0 in the hack would fix that? [22:17:35] perhaps, though I wonder why it happens [22:17:36] knowing that config.py checks _fileuid not in [os.getuid(), 0] [22:17:50] maybe I mixed up the stat_result members in the hack and the nlink turned into the uid or whatever [22:18:37] I don't think so, because config uses [4] to get the uid and the fifth version in the hack is filestatus.st_uid [22:19:56] *value not version [22:22:31] looks like it isn't using our modified os.stat() after all: still says "owned by someone else" despite the fifth value being 0 in our function [22:23:07] probably we should hack at os.stat() after pwb imports it but before it tries to get the config [22:23:19] though i'll be damned if I know when it first gets the config [22:24:42] or else, what if we dynamically create a new user-config.py? [22:24:46] it seems to get it as file scope, so as soon as pywikibot.config gets imported [22:24:58] oh wait [22:25:03] it’s user-config.py? with a dash? [22:25:11] my hack uses an underscore :P [22:25:15] fix that ^^ [22:25:27] will try [22:25:33] I just assumed it would be an underscore without checking [22:25:38] !log deployment-prep truncate all interwiki tables T400488 [22:25:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [22:25:43] T400488: Update interwiki table on beta cluster - https://phabricator.wikimedia.org/T400488 [22:26:18] also, random note, I think the “owned by someone else” error is legitimate, I’m looking at one of my tools and gunicorn is running as a different uid than the python file owner [22:26:27] idk why that error wouldn’t have been reported earlier then [22:27:04] did your tools use pwb? [22:27:07] nope [22:27:22] I haven’t used pwb in years, I’m afraid [22:27:59] good news and bad news [22:28:09] good news: it no longer says it's skipped user-config [22:28:16] bad news: still erorring out with the same error as ever [22:29:23] I suppose that further down the chain it ignores it still [22:54:04] after some more tweaking and bonking, login worked! [22:54:07] thanks for the help [23:01:24] \o/ [23:02:24] only took 4.5hours to get to log in