[11:08:51] <Guest36>	 Why is the following link no longer working?
[11:09:08] <Guest36>	 https://glamtools.toolforge.org/glamorous.php?doit=1&username=Scotch_Mist&use_globalusage=1&show_details=1%20Wiki%20Sites%20referencing%20Scotch%20Mist%20images
[11:10:32] <Guest36>	 !help
[11:10:33] <wm-bot>	 If you don't get a response in 15-30 minutes, please email the cloud@ mailing list -- https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
[11:17:24] <andre>	 That's a good question for a maintainer. https://codeberg.org/magnusmanske/glamtools/issues might be a place.
[11:23:58] <Guest36>	 Thank you - will follow ink ...
[13:48:07] <wm-bb>	 <nokibsarkar> I have a gut feeling, if this tool is working with categories, then the culprit might be the removal of the field `cl_from` (or, `cl_to`? I forgot which one was removed).
[14:10:39] <Emperor>	 Hi folks, do we expect the debian trixie image to generally Just Work? I span up swift-ms-fe-02 in the swift project (via pontoonctl) using debian-13.0-trixie and it's up but I am unable to ssh into it (Permission denied (publickey) - which makes me think the sshd is up but hasn't managed to be configured). Instance is https://horizon.wikimedia.org/project/instances/944ec746-94c1-47f9-8315-45c79822e82e/
[14:11:52] <Emperor>	 Looking at the log ( https://horizon.wikimedia.org/project/instances/944ec746-94c1-47f9-8315-45c79822e82e/console ) it might be that puppet and unattended upgrades were fighting over the dpkg lock, and thus Things Went Wrong ?
[14:14:34] <andrewbogott>	 Emperor: yes, it should just work although sometimes it is very slow. The quickest option for you is probably to just delete and try again but if you don't need to re-use the name you can leave the old one there for me to dig into.
[14:15:01] <andrewbogott>	 Emperor: oh, but... horizon links aren't transferable since they depend on context. What's the project + instance name?
[14:16:14] <wm-bb>	 <jeremy_b> swift/swift-ms-fe-02 sounds like
[14:16:30] <wm-bb>	 <jeremy_b> uuid isn't enough? :sad:
[14:16:57] <andrewbogott>	 oh no it is, I'm just being lazy
[14:16:59] <Emperor>	 andrewbogott: project: swift instance: swift-ms-fe-02
[14:17:17] <andrewbogott>	 Also, sorry to make you repeat yourself, I see you already said all that.
[14:17:35] <andrewbogott>	 for what it's worth, you are describing exactly the symptom of https://phabricator.wikimedia.org/T422509 which I thought was fixed :/
[14:17:44] <Emperor>	 andrewbogott: NP :) I don't have the spare quota to spin up another without deleting that, but I'm happy to leave it around while you poke it.
[14:17:52] <andrewbogott>	 ty
[14:19:52] <Emperor>	 [just LMK when you're done :) ]
[14:26:30] <andrewbogott>	 I didn't do anything, but... can you still not ssh?
[14:27:39] <andrewbogott>	 Emperor: ^
[14:30:10] <wm-bb>	 <jeremy_b> > last Puppet run was at Wed Jun 10 14:19:12 UTC 2026 (2 minutes ago).
[14:30:49] <andrewbogott>	 Yeah, I imagine that got things unstuck
[14:36:48] <Emperor>	 andrewbogott: yeah, I can now get in OK.
[14:37:05] <Emperor>	 [presumably the next puppet job unwedged itself]
[14:37:30] <andrewbogott>	 yeah -- as you thought it was just unlucky timing between unattended upgrades and puppet doing apt update.
[14:37:43] <andrewbogott>	 I don't think I need the VM to troubleshoot that so you can just get on with it.
[14:37:53] <Emperor>	 andrewbogott: if this happens again, should I report it again or Just Be More Patient? [pontoonctl gives up after 10 minutes, which is when I started looking]
[14:38:14] <andrewbogott>	 go ahead and drop a note on https://phabricator.wikimedia.org/T422509 if you think of it
[14:38:19] <andrewbogott>	 would be useful to know if this is rare or not
[14:38:24] <Emperor>	 ack, will do
[14:39:19] <andrewbogott>	 I can't think of a reason that race wouldn't be resent on every puppetized host ever.
[14:40:00] <andrewbogott>	 Although there's more for unattended-upgrades to do on a fresh VM
[14:48:41] <physikerwelt>	 Do you know when requests on https://toolsadmin.wikimedia.org/tools/membership/ are approved. I would like to support the request of AnkitSatpute for a live editing session.
[15:03:18] <andrewbogott>	 That's a Raymond_Ndibe question ^
[15:03:28] <andrewbogott>	 not always, but this week
[15:04:11] <andrewbogott>	 but physikerwelt if you can confirm that they are not a bot I can approve right now :)
[15:32:30] <physikerwelt>	 It's not a bot
[15:32:40] <physikerwelt>	 He is sitting next to me
[15:33:29] <andre>	 careful, maybe the bot dressed as a human!
[15:41:57] <physikerwelt>	 thank you, he can log in now
[15:42:47] <wm-bb>	 <jeremy_b> you could check for a pulse and temperature
[15:43:10] <wm-bb>	 <explorer_ankit> I am in, thank you!
[15:43:11] <wm-bb>	 <explorer_ankit> -- A human confirming not a bot ^^
[15:57:01] <andrewbogott>	 I realized right after I said that that you had clearly already confirmed he was not a bot :)
[21:23:47] <inflatador>	 I'm having a problem running puppet on a 6-week old beta cluster instance: `Error: Could not send report: certificate verify failed [self-signed certificate in certificate chain for CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]` . Any idea which certificate it's talking about?
[21:24:39] <inflatador>	 Hieradata for this instance appears to be using `deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud` for puppetmaster and puppetserver, so I'm not sure where the reference to `puppetmaster03` is coming from
[21:28:09] <wm-bb>	 <jeremy_b> was provisioning ever initially finished and working with the project local puppetmaster?
[21:29:00] <inflatador>	 no, Puppet is broken for sure
[21:30:20] <inflatador>	 ah, I got it to work! I had to move `/var/lib/puppet/ssl` out of the way and rerun puppet, then manually sign the cert on puppetserver
[21:31:51] <wm-bb>	 <jeremy_b> woot. yeah I was just typing about my vague memory for procedure to regenerate cert.
[21:32:37] <inflatador>	 All good, thanks for the suggestion
[21:39:04] <bd808>	 !issync
[21:39:05] <ircservserv-wm_>	 Syncing #wikimedia-cloud (requested by bd808)
[21:39:07] <ircservserv-wm_>	 Set /cs flags #wikimedia-cloud arturo -Afiortv
[21:39:09] <ircservserv-wm_>	 Set /cs flags #wikimedia-cloud rook -Afiortv
[21:39:11] <ircservserv-wm_>	 Set /cs flags #wikimedia-cloud blancadesal -Afiortv
[21:39:13] <ircservserv-wm_>	 Set /cs flags #wikimedia-cloud bliviero +Afiortv
[21:52:15] <bd808>	 inflatador: I wonder if we have been carrying the same puppetserver cert forward since deployment-puppetmaster03 days?
[21:53:06] <bd808>	 ah I see it was a stale upstream problem. backscroll reading comprehension fail.
[21:55:50] <mutante>	 except 6 weeks ago it also wasn't puppetmaster03?
[21:57:11] <mutante>	 something about this exact issue is common
[21:57:36] <mutante>	 searching for "deployment-puppetmaster03.deployment-prep.eqiad.wmflabs" in phabricator reveals a couple tickets with this
[21:57:38] <bd808>	 deployment-puppetmaster03 was the puppet server in 2018 based on some phab searching
[21:58:02] <mutante>	 I had this as well before, afair
[21:58:13] <mutante>	 and it was mysterious because that master is so long ago
[21:58:46] <mutante>	 https://phabricator.wikimedia.org/search/query/t9OcDj24nJv5/#R
[21:58:58] <bd808>	 deployment-prep has global hiera that switches puppet servers. A manual cert change step is needed on every new vm there.
[21:59:19] <mutante>	 one example: https://phabricator.wikimedia.org/T225557
[21:59:23] <mutante>	 this is 2019 and same host
[22:00:43] <mutante>	 ok, ack. I am just saying this exact error seemed very familiar, including the puppetmaster03