[06:09:46] <wm-bb>	 <telemoud> Hey all, was just wondering is there a slow_query_log for toolforge user dbs? A quick google doesn't bring up any results. Also interested in any recommendations for profiling/optimizing Toolforge DB queries. Thanks!
[07:08:21] <JJMC89>	 you could try https://sql-optimizer.toolforge.org
[07:46:55] <dcaro>	 nice tool
[09:40:40] <Titore>	 Sorry to bother, bridgebot is double posting again 
[09:44:53] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.bridgebot Double IRC messages to other bridges
[09:44:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[09:48:07] <taavi>	 !log tools update pywikibot script image to v9.1.0 T363132
[09:48:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[09:48:10] <stashbot>	 T363132: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T363132
[10:29:49] <wm-bb>	 <sohom_datta> Any chance there is a way I can see outputs of healthcheck runs ?
[10:35:18] <arturo>	 @sohom_datta: I believe successful probes aren't recorded anywhere. With `kubectl describe <pod>` you may see failed ones.
[10:45:23] <wm-bb>	 <sohom_datta> Do healthchecks not use launcher ?
[10:58:27] <arturo>	 I don't know what launcher is in this context
[11:07:43] <wm-bb>	 <sohom_datta> My healthcheck is set to be the command "ping" with a custom "ping" command defined in the Procfile
[11:08:51] <wm-bb>	 <sohom_datta> But the healtcheck seems to be running `exec /bin/sh -c ping` instead of `/bin/sh -c launcher ping` (which it does for other Procfile commands)
[11:19:20] <wm-bb>	 <sohom_datta> arturo: If you have some time could you take a look at link-dispenser. The crawljob process keeps crashlooping with a exit code of 137 after adding a healthcheck script
[11:20:27] <wm-bb>	 <sohom_datta> I intially thought it was because of the launcher ping issue, but it still crashloops after fixing the issue
[11:20:54] <wm-bot>	 !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss
[11:20:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[11:21:05] <Bsadowski1>	 :o
[11:41:00] <dcaro>	 sohom_datta what's the tool?
[11:41:23] <wm-bb>	 <sohom_datta> The tool should be link-dispenser
[11:41:35] <dcaro>	 let me give it a look
[11:41:55] <arturo>	 I'm just reading about this `launcher` thing https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Procfile
[11:42:15] <arturo>	 @sohom_datta, I don't think you need to have the actual `launcher` keyword in your command
[11:42:20] <arturo>	 it will be added internally by the system
[11:43:19] <arturo>	 ie, I believe your job's command should move from `launcher crawljob` to `crawljob`, and the healthcheck from `launcher ping` to `ping`
[11:44:29] <dcaro>	 it should not hurt either though, launcher is capable of running nested
[11:45:17] <arturo>	 is launcher responsible for going to the procfile and executing the required entry?
[11:45:37] <arturo>	 the job defines
[11:45:38] <arturo>	 | Health check: | script: launcher ping
[11:45:39] <dcaro>	 not really, it's responsible from setting the environment variables to change paths and such
[11:45:55] <arturo>	 but k8s executes
[11:45:56] <arturo>	 Startup probe failed: command "/bin/sh -c launcher ping" timed out
[11:46:09] <arturo>	 and in the procfile there is this
[11:46:10] <arturo>	 https://gitlab.wikimedia.org/toolforge-repos/link-dispenser/-/blob/main/Procfile?ref_type=heads#L3
[11:46:44] <dcaro>	 there's one binary for each procfile entry generated at build time
[11:46:46] <dcaro>	 https://www.irccloud.com/pastebin/Z2lRs23O/
[11:47:08] <dcaro>	 that does something similar to 'launcher <command in the procfile entry>'
[11:47:15] <arturo>	 ok
[11:47:45] <dcaro>	 so `launcher ./scripts/ping_celery.sh` and `ping` should be equivalent in this case
[11:47:53] <dcaro>	 `launcher ping` should also be ok
[11:47:56] <dcaro>	 (and the same)
[11:48:04] <arturo>	 ok
[11:48:17] <dcaro>	 it seems it was a timeout 
[11:48:18] <dcaro>	 Warning  Unhealthy  4m49s (x887 over 34m)  kubelet            Startup probe failed: command "/bin/sh -c launcher ping" timed out
[11:48:24] <wm-bb>	 <sohom_datta> Is there a chance that it picks the unix utility instead
[11:49:14] <dcaro>	 sohom_datta I don't think so, as it is it should be picking up the procfile entry, you can change the name to something that does not collide if you want (ex. if you want to use the original ping, you'll have to pass the full path)
[11:50:03] <dcaro>	 I see that the check connects to celery (does a `celery ping` )
[11:50:48] <dcaro>	 how is it configured?
[11:51:55] <wm-bb>	 <sohom_datta> The celery config should be pretty simple, it's run using https://gitlab.wikimedia.org/toolforge-repos/link-dispenser/-/blob/main/scripts/start_celery.sh?ref_type=heads
[11:52:57] <dcaro>	 hmm, I'm not sure what's the envvars defined inside the liveness check, maybe `NOTDEV` is not defined there, let me check
[11:54:39] <wm-bb>	 <sohom_datta> Yeah NOTDEV switches the redis url from a local setup to the wikimedia endpoint
[11:55:41] <dcaro>	 I'm thinking that it might be trying to connect to localhost and just timing out
[12:01:41] <dcaro>	 so I did a test, and I think that the healthcheck should be getting all the environment variables set properly
[12:02:31] <dcaro>	 local.tf-test@lima-kilo:~$ cat healthcheck.sh 
[12:02:32] <dcaro>	 #!/bin/bash
[12:02:32] <dcaro>	 env > environment.out
[12:02:32] <dcaro>	 local.tf-test@lima-kilo:~$ toolforge jobs run --continuous --command 'while true; do sleep 10; date; done' --health-check-script '$TOOL_DATA_DIR/healthcheck.sh' --mount=all --image python3.11 test
[12:02:32] <dcaro>	 local.tf-test@lima-kilo:~$ cat environment.out 
[12:02:33] <dcaro>	 ...
[12:02:33] <dcaro>	 TOOL_DATA_DIR=/data/project/tf-test   <- this one is set by us
[12:02:54] <dcaro>	 hmpf, that should have been a paste, sorry
[12:04:27] <dcaro>	 running the healthcheck directly in the pod on k8s works too:
[12:04:39] <dcaro>	 https://www.irccloud.com/pastebin/b34oIB12/%60
[12:10:17] <dcaro>	 hmm, I think probes might have a different connectivity than the pod itself
[12:13:28] <dcaro>	 nah, I was able to curl without issues
[12:20:26] <arturo>	 BTW this account has apparently several builds recorded
[12:21:11] <arturo>	 tools.link-dispenser@tools-bastion-12:~$ toolforge build list | wc -l
[12:21:11] <arturo>	 6
[12:21:30] <wm-bb>	 <sohom_datta> It's for the same container, do you think it's running on a outdated build ?
[12:21:34] <arturo>	 not sure if this is expected, or there should be some kind of cleanup
[12:22:05] <arturo>	 @sohom_datta the destination is always tagged :latest, so I think is fine
[12:22:22] <arturo>	 I was asking more from the system point of view, if the system should clean up the old ones
[12:23:59] <wm-bb>	 <sohom_datta> It might be cause theres a bunch of replicas just chilling around as well https://grafana.wmcloud.org/d/TJuKfnt4z/kubernetes-namespace?orgId=1&var-cluster=prometheus-tools&var-namespace=tool-link-dispenser and I don't know how to purge/remove them
[12:25:04] <wm-bb>	 <sohom_datta> crawljob-5687c95b7-g4hpf should be the only one in use right now
[12:26:24] <arturo>	 you also have a webservice, no?
[12:26:53] <arturo>	 the replicasets are created by the deployment objects by kubernetes. I think they are harmless
[12:27:30] <Sohom_Datta>	 Yes, the webservices should be the link-dispenser-*-* ones
[12:27:30] <arturo>	 but if you really really wanted to get rid of them, you may stop the webservice and see if they are gone with the deployment resource
[12:29:04] <dcaro>	 arturo: about the number of builds, I think we keep the last 4 failed ones, and 2 successful ones for reference
[12:29:34] <dcaro>	 all they push to the same image (:latest) so only the last successful build is in the image (it gets overwritten)
[12:29:58] <arturo>	 ok, I think there are currently 5, all of them succesful
[12:31:22] <Sohom_Datta>	 I'm planning to leave the replicasets alone unless kubernetes tells me something about it
[12:32:47] <dcaro>	 hmm, I think that the timeout for health probes might be too agressive (1s)
[12:33:09] <dcaro>	 Sohom_Datta: yep, replicasets are generated automatically by k8s, should be ok to let them be
[12:33:43] <dcaro>	 the liveness probe takes ~2s
[12:33:56] <dcaro>	 hmm, I thought we had increased that at some point
[12:35:29] <dcaro>	 oh, I think it's the default from k8s
[12:35:37] <dcaro>	 we should increase that a bit
[12:38:00] <Sohom_Datta>	 It would be nice to be able to manually set it in the yaml
[12:38:17] <Sohom_Datta>	 (Also maybe the frequency as well)
[12:39:11] <dcaro>	 yep, manually changed it to 3s in the deployment object and it's running now, I'll send a patch
[12:39:56] <dcaro>	 https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/79 <- this should do the trick
[12:39:58] <arturo>	 I think we could try to make the system smarter, rather than exposing more options to users. Otherwise, eventually, we will end with the full range of k8s options exposed via the different toolforge interfaces
[12:41:00] <arturo>	 dcaro: LGTM
[12:54:27] <wm-bot>	 !log anticomposite@tools-bastion-13 tools.stewardbots ./stewardbots/StewardBot/manage.sh restart # SULWatcher3 not coming back up
[12:54:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[13:04:11] <dcaro>	 Sohom_Datta: any new deployment you make now should have the new timeout values :)
[13:04:16] <dcaro>	 (just deployed the patch)
[13:04:48] <wm-bot>	 !log anticomposite@tools-bastion-13 tools.stewardbots SULWatcher/manage.sh restart # SULWatcher3 disconnected
[13:04:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[13:10:33] <Sohom_Datta>	 It appears to be holding on for now 🎉
[13:10:58] <Sohom_Datta>	 Thank you so much for debugging and taking a look :)
[13:12:20] <wm-bot>	 !log anticomposite@tools-bastion-13 tools.stewardbots deploy ed1afde
[13:12:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[21:03:53] <bd808>	 bridgebot is going to be down for a little while as I convert it over to using buildservice images and job service instead of it's current hand-built Deployments. If all goes well y'all shouldn't really notice anything different when I'm done. If it goes badly, well then I guess I'm not done yet. ;)
[21:05:37] <wm-bot>	 !log bd808@tools-bastion-12 tools.bridgebot Switched from legacy system to buildservice and jobs configuration (T363028)
[21:05:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[21:14:47] <bd808>	 grrr... there is a bug somewhere in my matterbridge config file, but the error message is pretty much just that. :/
[21:15:06] <lucaswerkmeister>	 :(
[21:21:29] <lucaswerkmeister>	 bd808: bridgebot.toml has an unclosed quote in line 194, I believe (account="irc.lib)
[21:21:37] <lucaswerkmeister>	 (and apologies if you already debugged past that point by now)
[21:22:02] <bd808>	 lucaswerkmeister: yeah. no worries. I just found that myself :)
[21:22:07] <lucaswerkmeister>	 ok :)
[21:22:55] * bd808 live hacks that
[21:23:38] <bd808>	 I think the bridge should be up now...
[21:24:10] <lucaswerkmeister>	 irc->telegram seems to be working (even catching up on some old messages), telegram->irc not yet
[21:24:39] <wm-bb>	 <lucaswerkmeister> oh hey the bridge’s still alive?
[21:24:45] <lucaswerkmeister>	 aha there it goes
[21:24:52] <bd808>	 laaaaag
[21:25:01] <wm-bb>	 <lucaswerkmeister> test test
[21:25:15] <wm-bb>	 <bd808> hmmmm... only one way?
[21:25:37] <bd808>	 ok. I'll patch the repo and rebuild and restart and then we can start wondering if it will keep working :)
[21:31:05] <wm-bot>	 !log bd808@tools-bastion-12 tools.bridgebot built new image from f4022bd9 (T363028)
[21:31:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[21:32:26] <wm-bot>	 !log bd808@tools-bastion-12 tools.bridgebot Started `bridgebot` job (T363028)
[21:32:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[21:36:22] <bd808>	 Ok, the bridges seem to all be up and stable at this point. I hope the lag before was just because the `webservice shell` I was debugging from needed a bit more ram and cpu (which it should have now)
[23:37:20] <lucaswerkmeister>	 \o/ \o/ \o/
[23:37:25] <lucaswerkmeister>	 sounds awesome, thanks a lot!
[23:38:37] <lucaswerkmeister>	 I see there’s a service.template file in the tool’s home dir, does that do anything? (I don’t see a running webservice and IIUC www/static/ is served by tools-static instead)
[23:39:09] <bd808>	 it makes running `webservice shell` easier, but that's all
[23:40:35] <bd808>	 I'm going to test the potential message doubling fix later in my evening by forcing ZNC to connect to a new upstream server.
[23:40:52] <bd808>	 First I need a snack though :)
[23:43:39] <lucaswerkmeister>	 ah, I see ^^