[08:40:57] morning! [08:55:15] morning :) [09:23:34] this is new, 0 alerts :) [09:23:36] https://usercontent.irccloud-cdn.com/file/46VhTKLm/image.png [11:02:08] * dcaro lunch [12:25:47] Did anything happen to trove instances? Superset's instance doesn't appear in horizon [12:35:42] nvm, I believe I removed it some time ago. [14:15:41] dcaro, did blancadesal mention https://phabricator.wikimedia.org/T311466 to you? The TLDR is, how best to add support for mono/dotnet users? See https://paste.toolforge.org/view/0a862047 for the list of ideas we discussed [14:54:11] balloons: just checked, thanks! [15:03:10] As you probably noticed, we announced the shut down date for the grid, so that's kicked in more migrations. And pushed us to make sure we've closed all open threads [15:04:10] that's good :) /me still going over the >2k emails [15:05:08] dcaro: nice work! I got it down to zero last week but it only lasted a few hours [15:06:09] jbond went ICANN [15:06:12] *went to [15:07:16] back to the roots :), /me off for a bit [19:57:46] Did something happen to the application credentials from horizon? [20:02:56] Other than the upgrade, I don't think horizon has changed. andrewbogott I believe didn't make any UI changes for puppetfree for example [20:06:21] Rook, the last time I thought that it turned out that the set I was using had just expired [20:06:29] Unless you're talking about a UI being missing [20:12:53] I've been getting auth errors when working with things that have been using auth creds. Namely superset, paws, and I think that is what you (andrewbogott) described as the issue with the tf tester. When one sets no expiration on a cred, is there a secret expiration on it? [20:17:24] I don't know, let's see if it's documented anyplace [20:18:53] According to the docs they don't expire unless you specify. Now I'm looking in the database... [20:19:09] hmm... [20:21:20] https://www.irccloud.com/pastebin/saJlRd6f/ [20:21:31] So that's pretty definite [20:22:59] Rook: are any of the things you're trying running multiple times/second? [20:25:37] Also: I just changed some things in response to a log error, can you try again? [20:25:42] No, nothing is running rapidly. Just been getting auth errors either from tofu or from k8s (through the cinder csi). It is probably not repeatable, as I've replaced all the associated keys to get things moving again, though it did seem kind of like a pattern. I have one more thing to try, though probably tomorrow, as it feels inelegant to redeploy paws twice in a day [20:26:03] ok [20:26:22] I fixed a thing but the thing I fixed should've broken /all/ keys not just new ones [20:26:27] *old ones I mean [20:30:39] So the keys were broken in the recent past? [20:36:32] I'm not sure; this is more of a 'how did this ever work?' issue [20:36:39] but I think I have a fix underway [20:37:45] Sounds good. Is there any expectation that the keys may break again during this fix? [20:42:03] I'm not sure. In theory there's a 'migrate' command that will make them work. [20:42:38] I don't have time to get this patch all the way merged but fyi this seems to be what's missing: https://gerrit.wikimedia.org/r/c/operations/puppet/+/980465 [20:43:04] Alrighty, thanks. [22:09:18] etherp [22:09:39] sorry, was supposed to be a search in my irc window :P [22:12:43] dhinus, :-)