[01:17:11] hey y'all, I'm getting the following when I run `toolforge-jobs list` (as my tool's user, on tools-sgebastion-10): [01:17:16] requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://api.svc.tools.eqiad1.wikimedia.cloud:30003/jobs/api/v1/list/ [01:18:08] known issue? me missing some kind of important announcement and not changing a config file? something else? [01:21:35] I can file a phab if desired, just wanted to ask first in case this is me doing something dumb [04:22:40] !log wikisp mars: Updated packages [04:22:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisp/SAL [07:06:07] GenNotability: which tool? [07:06:46] ^ yep, tested with a couple others and was not able to reproduce the error [07:08:54] found it, it's tools.vicbot2 [07:09:20] https://www.irccloud.com/pastebin/1933IZl4/ [07:09:22] that one? [07:09:25] looks like it's failing to parse cron expressions with spaces in them [07:15:49] * dcaro getting confused by the `dict` and `object` named variables around xd [07:16:22] 165 *actual.split(" "), [07:16:32] that might be it, the cron fields have more than one space between them in this case [07:16:41] yep. working on a fix [07:16:44] 👍 [07:28:19] dcaro: https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-framework-api/-/merge_requests/10 [07:39:57] !log tools deploying jobs-api 0.0.213-20230705073411-09895639 [07:39:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:40:16] GenNotability: your issue should be fixed. and please feel free to file tasks with the details if no-one is immediately around [07:41:12] godog: would it be possible to apply https://github.com/prometheus/prometheus/pull/12502/files to our prometheus debs? it seems to apply cleanly on top of 2.24.1+ds-1+wmf1, although I didn't test building yet [07:44:41] taavi: we could yeah, my worry is that we'll forget on the next prometheus upgrade and I guess we'll cross that bridge when we get there [10:59:16] !log deployment-prep shutting down and removing unused deployment-cache-text07 and deployment-cache-upload07 [10:59:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [12:23:25] taavi: cheers [14:25:14] !log tools.wikibugs Updated channels.yaml to: 06dace3d9e6817b50f3a7d190af48d1fdee6962d Remove #wikimedia-services [14:25:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [16:04:13] !log tools.watch-translations Deploy [16:04:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.watch-translations/SAL [16:06:07] !log tools.watch-translations Deploy 5f045e9 [16:06:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.watch-translations/SAL [16:32:56] !log tools deploy image-config 0.0.14 (no real changes, just ci rebuild) [16:32:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:46:01] how long does bastion take to recongise new ssh keys? [17:50:04] less time than it takes you to wonder if you should ask that [17:50:34] need me to poke into the bastion auth logs? :P [17:50:43] taavi: rhinosf1@bastion.wmcloud.org: Permission denied (publickey). [17:51:21] should be 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPZFSDt87n9gW37yitLntf3meXDgngvZ0+2GcR/JkI8v rhinosf1@gmail.com' [17:53:00] `Jul 5 17:43:16 bastion-eqiad1-03 sshd[4137117]: Accepted publickey for rhinosf1 from x.x.x.x` [17:53:07] can you share your `ssh -v` output? [18:14:15] !log wikistats MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ("gpe", "Ghanaian Pidgin", "Kroo brofo", 8) T335991 [18:14:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [18:14:19] T335991: Add gpewiki to wikistats - https://phabricator.wikimedia.org/T335991 [18:14:27] !log wikistats /usr/lib/wikistats/update.php wp prefix gpe T335991 [18:14:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [18:24:22] !log wikistats deleted broken tests [18:24:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [18:43:14] !log wikistats fix statsurl for wikitech [18:43:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [18:58:11] !log wikistats fix foundationwiki, ee/et.wikimedia, pt.wikimedia, wikimedia.at [18:58:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [19:16:47] !log wikistats dropped all non http 200 from miraheze table [19:16:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL