[09:08:16] !log tools.integraality Deploy 137f5b8, c816e41 8e0b9ca, 8dbc099 (T273226) [09:08:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [09:34:04] !log tools.bd808-test experiment with ingress annotations for reverse proxy (T294330) [09:34:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bd808-test/SAL [09:35:32] hey! ejoseph is a new team member on Search Platform [09:35:55] hello ejoseph !!! welcome!! [09:35:55] He's having issues connecting to primary.bastion.wmcloud.org [09:36:27] His public SSH key has been uploaded to https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/ but I suspect that wasn't what's needed :/ [09:38:06] you need to upload the ssh key in wikitech [09:38:14] toolsadmin is for toolforge tools [09:38:31] (I guess we can do better in documenting that...) [09:38:44] relevant docs: https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances [09:38:54] cc gehel ejoseph ^^^ [09:39:01] arturo: thanks! [09:39:09] this is the magic link: https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack [09:41:45] looks like it is already there, so some other problem [09:42:08] arturo: would you have a few minutes to help ejoseph to debug this? [09:42:13] yes, I do! [09:43:04] is ejoseph enrolled in any CloudVPS project? [09:43:22] I've jsut added him to the wikidata-query project [09:43:50] But only 5 minutes ago. I would have expected the keys to be already deployed on the bastion, even if he had no project access [09:44:00] Hi Arturo [09:44:26] Should I send a link [09:44:32] o/ [09:44:46] meet link* [09:45:06] sure, in a DM please [09:46:13] ejoseph: I found the problem: `Failed publickey for ejoseph` [09:46:56] it seems you are connecting with this key `ED25519 SHA256:gqIEv/9BwjtcbYVxGhZF9LLU4wHjETsOESKyUL85iM0` which likely isn't the one you set in wikitech [10:18:02] !log tools.integraality Deploy b4f12bc to help investigate T236960 [10:18:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [10:34:38] welcome ejoseph! [10:37:34] Thank you [10:43:09] !log tools.deno update proxy-github ingress object (T294533) [10:43:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.deno/SAL [10:43:12] T294533: tool-deno: update reverse proxy ingress annotations - https://phabricator.wikimedia.org/T294533 [10:44:23] !log tools.deno update proxy-deno ingress object (T294533) [10:44:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.deno/SAL [11:29:21] !log tools.integraality Deploy 5c547d8 [11:29:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [11:31:37] !log tools.integraality Deploy 30a2f2ba (T278156) [11:31:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [12:23:10] !log tools.moedata update proxy ingress objects (T294547) [12:23:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.moedata/SAL [12:23:13] T294547: tool-moedata: update reverse proxy ingress annotations - https://phabricator.wikimedia.org/T294547 [12:38:44] !log tools.integraality Deploy 0a8e7fb (T236590) [12:38:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [12:42:54] !log tools set `allow-snippet-annotations: "false"` for ingress-nginx (T294330) [12:42:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:48:10] !log toolsbeta update ingress-nginx via helm for `--watch-ingress-without-class=true` [12:48:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:35:18] !log paws set team toolforge/wmcsadmins as maintainers for github repo [14:35:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [14:47:38] arturo: for future reference, the ssh-key management in Striker (toolsadmin) does the exact same things as using the ssh-key management in wikitech. Both manage the public keys attached to the user's Developer account LDAP entry. [14:47:57] right [17:14:04] !log tools.integraality Rollbacked to 977236b42daef66818e561b49e64264a0c785454 for T294570 and restart [17:14:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [17:25:28] !log video move enconding04 to python3 T294520 [17:25:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [17:25:31] T294520: Error with video2commons Tool - https://phabricator.wikimedia.org/T294520 [17:47:06] what would be the least-worst way to run a command every time I start/restart a (python k8s) webservice? [17:49:50] !log video move encoding05 and encoding06 to python3 T294520 [17:49:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [17:49:53] T294520: Error with video2commons Tool - https://phabricator.wikimedia.org/T294520 [18:42:59] AntiComposite: inside or outside the container? but I'd do a bash script that wraps `webservice $1` and then runs your other command before/after [18:43:13] inside [18:43:27] well, inside *a* container [18:43:42] can you have your python app do it on startup? [18:44:02] no [18:44:21] https://phabricator.wikimedia.org/T169695#7378720 was merged [18:45:11] what I'm trying to do is to run a PEP 517 build to build the app before it starts [18:46:10] because poetry-core doesn't support editable installs yet, it would have to build every time the code is changed for it to take effect [18:46:19] tools.ldap@tools-sgebastion-08:~$ webservice python3.9 shell -- python3 --version [18:46:19] Python 3.9.2 [18:46:37] doing it in webservice shell would work on start, but not restart [18:47:06] why not? [18:47:51] well, I suppose I could run it before running restart too [18:48:16] k8s initContainer? [18:48:50] tried, something was broken with the user [18:48:59] it had the correct user id, but no permissions [18:52:14] other thought was patching the command to use a bash script that ran pip install . and then ran webservice-runner [18:52:35] but that would break if the webservice-runner invocation changed [19:06:30] poetry can just build the venv btw [19:06:57] yes, but that means installing poetry [19:07:01] on toolforge [19:07:24] on a venv [19:08:02] they don't recommend doing that, as it drags in a bunch of extra dependencies [19:31:57] I thought https://python-poetry.org/docs/#osx--linux--bashonwindows-install-instructions was isolated from the venv? [19:32:32] yes, if you use that method it is [19:34:21] I'm not saying its pratical, but you can have a poetry venv in toolforge that manages another venv that is actually used by your container [19:34:53] can you? when I tried that it just detected the venv it was in and modified that [19:35:19] maybe if you don't activate it but just run the executable... [19:35:21] there is a flag for poetry to tell it what venv to use, iirc [19:35:28] or that [20:11:11] anyone who uses https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ to compile changes on cloud VPS? I ran into the "Unable to find fact file" again when I pasted my instance FQDNs to compile on.. but I remember it used to work in the past and at least one of them is definitely not new so the facts have been synced for sure. I think there was some gotcha to it, like had [20:11:17] to change the hostname slightly due to the renames. but what was that? I tried wmflabs.org just because but that wasn't it [20:11:29] it should match the output of `hostname -f` [20:11:44] that's what I did though.. hrmm [20:12:06] one of them is new.. that's one thing, ok. but not the other one [20:12:50] vm.projectname.eqiad1.wikimedia.cloud [20:14:07] yea, somehow it doesn't have the facts about my instance, i'll risk it without compiling and just merge :) cloud-only [20:15:05] there was something in the past where it did not match the FQDN exactly when the compiler was involved [20:52:36] AntiComposite: we should figure out how to make `webservice` work out of the box with poetry! We would need to do some work on how the python container does what it does to run the wsgi entry point but I think it is a thing that we could make work. [20:55:02] The place that would need changes is https://github.com/wikimedia/operations-software-tools-webservice/blob/master/toolsws/wstypes/python.py. One approach might be to add some special cli argument like `--poetry` and then fork the whole wsgi container creation bit. Another could be to autodetect a poetry.lock and do things a bit differently. [20:55:42] We could put poetry in the base image for python too I think without causing anyone real problems. [20:56:05] Toolhub is using poetry, and I like it, so I'm interested in being able to use it in my volunteer work too. [21:06:38] I do think PEP 517 builds are the best way to do it, since that means support for any future tools instead of just one [21:22:22] (they're also faster, at least with poetry) [21:35:40] PEP 517 is more about pypi packaging than app deployment though isn't it? [21:36:23] sdist and wheels aren't typically a thing I worry about producing for a website deploy [21:37:23] for toolforge tools I'm more interested in provisioning a venv and a wsgi container [22:54:20] I'm working on deleting things from toolforge NFS but if anyone here would like to delete their logfiles that'd be great! [23:08:35] ugh, I'm not finding obviously easy things to delete [23:08:58] glamwiki is using 200+ GB but It looks like it might be useful historical data [23:19:32] !log tools.wd-image-positions briefly stopping tool (few minutes) to cycle the uwsgi.log [23:19:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [23:21:47] !log tools.wd-image-positions tool is back up but the pre-2021 logs are gone instead of backed up because the container doesn’t have zstd and I didn’t notice in time :( [23:21:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [23:22:18] well, that’s 34MB less disk space I suppose [23:25:10] thanks lucas [23:27:20] is it worth filing a task to add zstd to the tf-bullseye-std image? (T225380 was for adding it on the bastions) [23:35:40] As to whether or not to do it is complicated, but a ticket is a reasonable place to discuss that :)