[09:36:52] godog: sorry, I seem to have broken thanos on (depooled) thanos-fe1001 by running apt update to pull in a swift security fix. It looks like puppet is also trying to update /etc/thanos-rule/rules/recording_rules.yaml and exploding. https://puppetboard.wikimedia.org/report/thanos-fe1001.eqiad.wmnet/c480dab9db7aa7c33af546e69a73de331dd34172 [09:38:56] godog: https://phabricator.wikimedia.org/P43324 is the puppet output [09:39:49] Emperor: ack, checking [09:40:13] TY :) [09:42:41] Emperor: FWIW it isn't any more broken than it was before (i.e. it was failing last night too and likely since Id8457134d5 [09:42:51] i.e. feel free to continue with the upgrades, thank you for the heads up [09:43:07] I'll look into the fix [09:43:36] OK, cool, thanks. [09:52:55] back in working order [09:53:25] grand, thank you - and I see puppet has just run OK on thanos-fe1002 post-upgrade. [09:55:11] indeed [14:39:46] https://i.imgur.com/vwxSIPr.png [14:39:51] uh? [14:40:39] seeing that as well... looking [14:45:47] herron: FIY if you haven't seen on -sre, the actual dashboards seem to work, but going Dashboards -> Home leads to that error [14:47:42] claime: thanks, hmm yeah it seems the dashboard we've been using as the home dash has gone missing [14:50:57] interesting, yeah looks like the home dashboard is missing indeed [14:51:08] (meeting shortly) [14:56:46] the dashboard id was -K8NgsUnz afaics [14:58:28] that's what I have as well [15:22:37] yeah so restoring from backups it is, herron have you done that before and/or want to take a crack at it ? [15:22:50] its been a while but sure [15:23:29] ok! I'll peek at the db to see what we have to do to pull dashboards out of it [15:23:44] ty! [15:30:27] it'll be likely sth like this [15:30:29] select data from dashboard where title like '%home%'; [15:31:46] to get the json that is [15:33:02] thanks I think also where uid='-K8NgsUnz', looking at the json now will try creating a dashboard with it now [15:34:01] doh of course, I missed the uid column, that's better [15:35:27] "dashboard not found" when saving, troubleshooting that [15:37:50] https://etherpad.wikimedia.org/p/xfxxnU3ZOvnm_VoQjnCy fwiw [15:40:30] I'm seeing an empty pad, not sure if expected [15:41:23] weird, better now? [15:41:49] mmhh no [15:42:48] https://grafana-rw.wikimedia.org/d/7Ir8Jyo4k/herron-home-recovery-2?orgId=1&refresh=30s [15:43:08] not sure whats going on with etherpad 😅 anyways sorted out the import, going to rename and set this as home [15:44:08] sweet [15:44:35] set as home, should be sorted out now [15:45:40] * herron checks with incognito, sees home page [15:47:15] Comfirmed working signed in too [15:47:24] neat, thank you [15:47:29] awesome ty for checking [15:47:41] gg herron [16:15:21] Ty indeed everyone [16:31:19] herron: taking a look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/883223/2/modules/profile/files/thanos/recording_rules.yaml, given the error started on line 22, could it just be as simple as the spacing being different for those bottom 2 snippets? looks like `expr` might just need to be tabbed over to the left [16:32:16] ryankemper: hey possibly, I haven't had a chance to look closely at it yet but hope to today