[08:11:15] <dcaro>	 Morning!
[15:05:50] * andrewbogott waves
[15:12:12] <balloons>	 Good morning!
[16:32:38] <andrewbogott>	 Anyone coming to checkin or is today a jetlag recovery day?
[16:34:29] <taavi>	 jetlag recovery day for me at least
[16:59:27] <taavi>	 andrewbogott: have you looked at which of our VMs are affected by https://lwn.net/Articles/954285/ already? seems like the broken bookworm kernel package is linux-image-6.1.0-14*
[16:59:54] <andrewbogott>	 I have not looked
[17:03:33] <taavi>	 ok! is cumin still broken for global runs?
[17:03:59] <taavi>	 seems to be :/
[17:06:17] <taavi>	 at least toolforge seems unaffected thankfully
[17:07:51] <andrewbogott>	 taavi: sorry I'm trying to attend the SRE meeting, I will try to catch up after
[17:08:03] <taavi>	 sure
[17:48:56] <andrewbogott>	 taavi: do you have a phab task for tracking that kernel bug?  I'm going to work on getting cumin working we can get a kernel version report...
[17:51:20] <taavi>	 andrewbogott: https://phabricator.wikimedia.org/T353178
[17:54:28] <andrewbogott>	 this works on cloudcumin1001: sudo cumin 'O{*}' 'uname -r'   Of course there are 100+ VMs that don't respond to cumin but I haven't picked through the list yet.
[17:55:08] <taavi>	 oh nice, I was getting 401s for some reason
[17:55:38] <andrewbogott>	 here's the firehose:  https://etherpad.wikimedia.org/p/cloudvpskernels
[17:59:18] <andrewbogott>	 hm, and etherpad threw away all my edits, I'll try again
[17:59:21] * dcaro off
[18:00:44] <andrewbogott>	 Is there a way to see if that patch is in place short of researching each kernel version individually?
[18:03:54] <andrewbogott>	 taavi: I see discussion of what version has the fix but not what version started with the issue. Do you think it's only 6.1.0-14 that's affected?
[18:04:16] <taavi>	 andrewbogott: my understanding is that 6.1.0-14 is the only affected version
[18:04:59] <andrewbogott>	 there is exactly one VM running that kernel
[18:06:28] <taavi>	 ok, which one?
[18:07:36] <andrewbogott>	 mint.language.eqiad1.wikimedia.cloud 
[18:07:56] <andrewbogott>	 But it's the current bookworm image, we've just gotten lucky that no one has created any new bookworm VMs in the last week.
[18:08:01] <andrewbogott>	 I'm going to build a new base image now...
[18:10:30] <andrewbogott>	 I think that's the VM from that long quota discussion last week
[18:11:10] <andrewbogott>	 https://phabricator.wikimedia.org/T352136
[18:40:15] <andrewbogott>	 taavi: the new debian-12.0-bookworm has 6.1.0-15-cloud-amd64 so we should be good now except for that one VM. I'll make a ticket for that one.
[18:40:33] <andrewbogott>	 Of course there are a few other changes in that new base image which I was hoping to test in codfw1dev for a bit longer :/
[18:40:43] <taavi>	 thanks
[18:41:04] <andrewbogott>	 thanks for bringing it up! The longer we waited the more of a mess it would've made.
[18:44:38] <bd808>	 That feeling when you figure out that you knew a thing to be true in August that you had completely forgotten about even being a possibility by November.
[18:44:56] <bd808>	 T353176
[18:44:57] <stashbot>	 T353176: GitLab users with only provider=cas3 identies are not found when Striker attempts to create GitLab repostories - https://phabricator.wikimedia.org/T353176
[18:48:29] <andrewbogott>	 Anyone know if/where to find kartik on IRC?
[18:48:55] <andrewbogott>	 bd808: three months is pretty good, sometimes I have to re-learn lessons I learned a week ago
[18:50:36] * andrewbogott grits teeth and uses slack
[18:50:53] <bd808>	 andrewbogott: look for kart_  in #translatewiki
[18:51:05] <andrewbogott>	 thanks!
[18:51:52] <bd808>	 I was surprised not to see them in #wikimedia-language. They might be in the telegram side of that channel.
[18:53:07] <andrewbogott>	 I'm doing the annoying thing of creating a ticket for someone and then telling them about the ticket in slack, and then telling them about the slack message on irc.  Should probably stop at that.
[19:00:22] <bd808>	 andrewbogott: you forgot a talk page ping on wikitech, mw.o, and officewiki & a follow up via email. 🤣
[19:00:50] <andrewbogott>	 It's always good to keep some options in reserve
[19:02:07] * bd808 lunch
[21:11:39] <bd808>	 andrewbogott: I saw your change to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/VM_images saying that images cannot be downloaded directly anymore. After running `set_proxy` I can `wget https://cloud.debian.org/images/...` from cloudcontrol1005.
[21:11:49] <bd808>	 `set_proxy` is magic documented at https://wikitech.wikimedia.org/wiki/HTTP_proxy
[21:12:13] <andrewbogott>	 that will be helpful! I'll add that to the script
[21:21:52] <andrewbogott>	 hm, or, I guess, the docs
[21:29:56] <taavi>	 it should be relatively trivial to have the script use the proxy to download the image