[17:35:57] !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss [17:35:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [17:36:04] !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/StewardBot because of a connection loss [17:36:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [17:52:00] Kinda noob question, if I want to have stashbot log actions for a specific project do I need to setup stashbot somehow ? [17:53:33] no, you can just run `dologmsg 'message...'` [17:56:01] !log soda@tools-bastion-13 tools.matchandsplit ./matchandsplit/scripts/toolforge-deploy-new-version.sh [17:56:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.matchandsplit/SAL [17:56:16] Ah I see :) [18:02:10] \o/ [20:14:11] bd808: have a few to help me understand about tool-os-deprecation? It's been failing for a month or so with auth errors. I see that it gets auth info from /etc/novaobserver.yaml but I'm not clear on how that gets mounted into the container [20:40:05] andrewbogott: the /etc/novaobserver.yaml file at least used to be automatically mounted into containers by one of the magic Kubernetes bits. There were/are a number of files like that that get mounted like the socket for the sssd process. [20:40:29] ok that makes sense. It's not failing on the stage where it reads the file so I figured it was finding it someplace. [20:40:34] andrewbogott: I don't know enough to know if `--mount=all` still does the same things or not [20:40:47] my current theory is that I broke the creds somehow, digging into that now [20:41:29] * bd808 sees the "Last updated: 2024-05-05 10:25:25" on https://os-deprecation.toolforge.org/ now [20:43:06] I sort of expected toolforge-jobs load jobs.yaml to refresh the error log right away but my toolforge knowledge is many months out of date [20:44:15] that command would replace all of the job definitions, but not trigger a new run of a job using a cron timer [20:44:57] you should be able to run a job manually though to watch it succeed/fail. Let me refresh my understanding of the tool... [20:45:57] thx [20:51:08] andrewbogott: `webservice python3.9 shell -- tool-os-deprecation/buster.sh` will make it log and die right on your terminal. [20:51:23] nice, trying... [20:56:45] hmph, of course auth works on the cl [20:58:06] ok, it's a name vs id thing [20:59:35] ah! That sounds like progress :) [21:01:32] andrewbogott: the next creepy thing you are going to find out is that most of the talk to OpenStack logic in os-deprecation is done by a submodule snapshot of the openstack-browser code (which is still called keystone-browser at the source code level). [21:01:58] yeah, I'm already wading around in there [21:02:12] this is actually long overdue for a rebase, so maybe that's all we need [21:04:31] hm, interesting, after resetting the openstack-browser stuff to head it gets oom-killed [21:04:43] that's probably progress of some sort [21:05:53] oom-kill is not a thing I would naively expect. `--mem 2G` should make that even less likely [21:06:55] Actually I have to go do a thing now, I bet when I'm back the rebase + cron will have fixed everything [21:08:09] :crossed-fingers: :) [21:17:28] andrewbogott: if things are not peachy when you are back, a thing I would look at is rebuilding the $HOME/tool-os-deprecation/.venv environment. I was checking to see how outdated the libs there are and found that things are quite different in the newer libs/keystone-browser/requirements.txt file. [21:17:39] `webservice python3.9 shell -- tool-os-deprecation/.venv/bin/pip3 freeze -r $HOME/tool-os-deprecation/requirements.txt -r $HOME/tool-os-deprecation/libs/keystone-browser/requirements.txt` will show you what I saw. [22:40:05] hmmm still not peachy [23:04:11] bd808, I don't think I have rights (or know-how?) to push to https://phabricator.wikimedia.org/source/tool-os-deprecation.git, when you have a minute can you update the libs/keystone-browser/ submodule and push? That + a memory boost gets it working. [23:16:40] andrewbogott: https://gitlab.wikimedia.org/toolforge-repos/os-deprecation is the live repo. The diffusion one is a read-only mirror. [23:17:42] I can do a submodule bump for you, but it would be more ideal if you could self-serve incase there are follow up things you need to do. [23:18:57] * bd808 sees that andrewbogott is not a member of that tool's repo or of the parent group [23:19:54] bd808: ah, if it's actually on gitlab then I can figure it out [23:19:55] thx [23:20:06] andrewbogott: you are now a co-owner of the https://gitlab.wikimedia.org/toolforge-repos namespace, so you can do the needful on many things [23:21:38] great