[07:58:32] <godog>	 greetings
[08:16:09] <godog>	 I've opened T418829 to track fourohfour overload btw
[08:16:09] <stashbot>	 T418829: fourohfour general unavailability / overload - https://phabricator.wikimedia.org/T418829
[08:16:27] <godog>	 please edit at will the description with more info/etc
[08:32:24] <godog>	 mmhh grafana.wmcloud.org == 503, taking a look at that first
[08:35:17] <godog>	 Mar 03 06:27:02 metricsinfra-grafana-2 grafana[3867793]: Error: ✗ migration failed (id = ensure rule_group column is case sensitive in returned results): Error 1273 (HY000): Unknown collation: 'utf8mb4_0900_as_cs' 
[08:35:21] <godog>	 joy
[08:39:08] <volans>	 which version of mariadb we have?
[08:41:48] <godog>	 so grafana is using a trove db in this case, not sure what version
[08:42:34] <godog>	 opening a task in the meantime
[08:44:43] <godog>	 T418831
[08:44:43] <stashbot>	 T418831: grafana.wmcloud.org unavailable - failed db migration - https://phabricator.wikimedia.org/T418831
[08:46:51] <godog>	 upstream issue is https://github.com/grafana/grafana/issues/118836
[08:46:52] <volans>	 comemnted there
[08:47:01] <godog>	 lol great timing
[08:48:30] <godog>	 I'll downgrade grafana and hold it for now
[08:52:56] <volans>	 but when it was upgraded? by who?
[08:53:52] <godog>	 https://phabricator.wikimedia.org/T416608
[08:54:36] <godog>	 and unattended-upgrades did the actual upgrade on metricsinfra
[08:54:56] <volans>	 right :/
[08:59:24] <dcaro>	 godog: thanks!
[09:00:07] <godog>	 yw dcaro, that adrenaline kick worked better than coffee
[09:00:19] <volans>	 lol
[09:17:35] <dhinus>	 morning
[09:26:53] <dcaro>	 just added a new redis dashboard to help debug issues (we were missing it)
[09:26:53] <dcaro>	 https://grafana-rw.wmcloud.org/d/e008bc3f-81a2-40f9-baf2-a33fd8dec7ec/redis-dashboard-for-prometheus-redis-exporter-1-x?orgId=1&from=now-24h&to=now&timezone=browser&var-namespace=&var-instance=tools-redis-5&var-query0=&var-prometheus=P8433460076D33992&editIndex=0
[09:27:11] <dcaro>	 fyi, very useful one-liner to replace datasources:
[09:27:12] <dcaro>	 dcaro@acme$ cat redis.grafana | jq '.panels[] |= (if .datasource.uid == "P8433460076D33992" then .datasource.uid = "${prometheus}" else . end)' | wl-copy
[09:30:20] <godog>	 very cool, thank you dcaro 
[09:59:48] <dcaro>	 I'm playing around with setting a local memory cache per-process to avoid hitting redis so often on fourohfour
[10:02:14] <godog>	 ack
[10:02:26] <dcaro>	 just saw the prod issues
[10:52:11] <dcaro>	 fourohfour sems stable now :)
[10:52:25] <dcaro>	 (🤞)
[10:53:01] <godog>	 neat, thank you dcaro 
[11:06:40] <dcaro>	 this is the current fix (applied in prod, where I have some extra logging still for debugging for a bit) https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/16
[11:42:57] <godog>	 neat
[15:58:15] <dcaro>	 Raymond_Ndibe: I'm deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/259, can you get ready to test in toolsbeta?
[15:58:30] <dcaro>	 (regarding the v1->v2 migration for jobs)
[16:11:51] <dcaro>	 hmmm.... the versions of the components deployed shown look weird in toolsbeta
[16:13:51] <dcaro>	 Raymond_Ndibe: ping?
[16:21:12] <dcaro>	 quick review (for the fourohfour) https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/16 if anyone has a moment (it's running in prod already)
[16:25:46] <dcaro>	 Raymond_Ndibe: I've deployed it to toolsbeta, deploying the jobs-api change on tools, I'll start with the tests in toolsbeta in parallel
[16:40:09] <dcaro>	 Raymond_Ndibe: tested in toolsbeta, will start migrating tools in chunks
[17:40:49] <dcaro>	 okok, done :)
[17:44:52] <dhinus>	 dcaro: nice! all went smoothly?
[17:45:03] <dcaro>	 yep :), had to do some manually, but minor issues
[18:16:23] * dcaro off
[18:17:11] <dcaro>	 cya tomorrow!
[18:18:32] <dcaro>	 Raymond_Ndibe: feel free to send the MR cleaning up the extra code we added to jobs-api for the migration if you don't find any more v1 jobs in a few hours (if you are still around)
[18:26:23] <taavi>	 T418897
[18:26:24] <stashbot>	 T418897: toolforge-deploy tests failure: Your local changes to the following files would be overwritten by checkout: components/jobs-api/2025_04_migration_of_all_jobs_to_version_2 - https://phabricator.wikimedia.org/T418897
[18:31:32] <dhinus>	 taavi: is that in tools or lima-kilo?
[18:31:44] <taavi>	 that is toolsbeta
[18:31:56] <taavi>	 .. which probably explains why I couldn't find the checkout on tools. :D
[18:32:02] <dhinus>	 :P
[18:32:47] <taavi>	 sigh
[18:33:16] <taavi>	 yep
[18:34:15] * dhinus off
[20:13:10] <dcaro>	 Oh yes, feel free to discard those changes, forgot to git reset after testing