[08:47:03] <arturo>	 dcaro: that k8s message rings a bell. I have seen that before, but I don't thing it was an indicative of anything wrong happening 
[10:30:45] <dcaro>	 topranks: hey, do you prefer doing the gnmi thing here or on meet?
[10:31:57] <topranks>	 hey!  meet is probably best maybe?
[10:32:08] <topranks>	 can we push it back by 15 min though?  if that works for you?
[10:32:27] <topranks>	 I'm just getting my machine set back up properly after the trip (only got back last night) 
[10:32:51] <dcaro>	 sure, I'm in https://meet.google.com/aba-ybqy-ngi
[10:33:05] <dcaro>	 whenever you are ready
[10:33:09] <topranks>	 ok
[10:33:11] <topranks>	 thanks <4
[11:29:42] <dhinus>	 dcaro: you mentioned "released components-cli 0.2.0 on tools" in #-daily, but I don't see that version in gitlab, am I looking in the wrong place?
[11:38:22] <dcaro>	 let me check
[11:38:52] <dcaro>	 https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/releases/debian%2F0.0.2
[11:39:30] * dcaro typing error generato
[11:39:35] <dcaro>	 *generator
[11:39:38] <dcaro>	 the version was 0.0.2
[11:42:15] <dhinus>	 ah ok! :)
[11:46:59] <andrewbogott>	 arturo: can you reassure me that cloudgw1001 is inactive before I start the decom?
[11:47:14] <arturo>	 yes, let me check
[11:48:21] <andrewbogott>	 ty
[11:49:02] <arturo>	 andrewbogott: cloudgw1004 is primary, cloudgw1001 is secondary, wmcs.openstack.network.tests reports all OK
[11:49:23] <andrewbogott>	 great! I'm starting the reimages and then having breakfast
[11:49:47] <arturo>	 ok, dont forget to merge the puppet patch before the reimage!
[11:50:15] <andrewbogott>	 Already done :)
[11:52:49] <arturo>	 👍
[12:26:48] <andrewbogott>	 partman failed on 1003 because of course
[12:36:50] <andrewbogott>	 worked the second time for some reason
[12:43:24] <dcaro>	 standard behavior yep xd
[12:44:13] <arturo>	 :-(
[12:49:31] <arturo>	 andrewbogott: did you do the netbox DNS changes?
[12:49:41] <andrewbogott>	 I haven't touched netbox
[12:49:51] <arturo>	 https://phabricator.wikimedia.org/T382356#10520717
[12:50:02] <arturo>	 I'll do them
[12:50:06] <andrewbogott>	 thx
[12:50:14] <andrewbogott>	 1003 just started its initial puppet run
[12:51:02] <arturo>	 ok
[13:01:42] <arturo>	 andrewbogott: how is that going?
[13:01:58] <andrewbogott>	 It's done with the initial puppet run, wrapping up
[13:06:44] <arturo>	 ok
[13:09:14] <andrewbogott>	 arturo: ready!
[13:09:51] <arturo>	 ok
[13:10:58] <arturo>	 let me clear the NXDOMAIN for vlan1120.cloudgw1003.eqiad1.wikimediacloud.org
[13:12:38] <arturo>	 ok, the wmcs.openstack.network.tests cookbook is back to green
[13:13:08] <andrewbogott>	 great. time to fail over?
[13:13:24] <arturo>	 yes!
[13:13:42] * andrewbogott crosses fingers, knocks wood, etc.
[13:13:52] <arturo>	 you do it? or I do?
[13:13:56] <andrewbogott>	 go ahead
[13:14:53] <arturo>	 done
[13:15:08] <andrewbogott>	 seems fine!
[13:15:16] <andrewbogott>	 stashbot, still here?
[13:15:30] <andrewbogott>	 hmmmm
[13:16:03] <arturo>	 -_-
[13:16:37] <andrewbogott>	 I think we expected brief bot interruption?
[13:16:46] <andrewbogott>	 stashbot, I can see you in the sidebar
[13:17:03] <arturo>	 yeah, unfortunately, always happens
[13:17:15] <andrewbogott>	 do I need to restart it or will it do that on its own?
[13:17:24] <arturo>	 on its own,
[13:17:28] <andrewbogott>	 ok
[13:17:30] <arturo>	 yeah, nothing to do
[13:17:45] <andrewbogott>	 do you have an opinion about whether I decom 100[12] now or later?
[13:18:34] <arturo>	 I don't have an opinion. I don't think we will ever need them, they are faulty hardware in my opinion
[13:18:47] * arturo apparently has an opinion then
[13:19:16] <andrewbogott>	 ok, I'll start decom
[13:19:30] <arturo>	 ok!
[13:26:06] <andrewbogott>	 ...should I be worried about stashbot yet?
[13:26:49] <dcaro>	 just came back
[13:27:11] <andrewbogott>	 oops just in time for me to restart it
[13:27:21] <andrewbogott>	 stashbot, working now?
[13:27:21] <stashbot>	 See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help.
[13:27:26] * andrewbogott shrugs
[13:33:19] <andrewbogott>	 thanks arturo! Going to close some phab tasks now :)
[13:33:47] <arturo>	 andrewbogott: I see there are new per-HV CPU panels on https://grafana-rw.wikimedia.org/d/000000579/wmcs-openstack-eqiad-summary it is possible it is making the dashboard slow to load?
[13:34:37] <andrewbogott>	 I'm sure they make it slower although it still loads for me in a tolerable amount of time
[13:35:02] <andrewbogott>	 you can remove them if they're too much trouble; I added them when I was adjusting the overprovision ratio
[13:35:37] <dcaro>	 collapsing them in a section is also a way to prevent them from loading by default
[13:41:07] <arturo>	 maybe a dedicated dashboard would be better
[13:41:38] <andrewbogott>	 probably! For now I think it's fine to just remove them if they're in your way.
[13:41:43] <andrewbogott>	 want me to do that?
[13:59:22] <arturo>	 nah!
[13:59:25] <arturo>	 no problem
[14:42:33] <dhinus>	 if nobody objects, I'm gonna merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1113194 cc Raymond_Ndibe 
[14:43:00] <dhinus>	 it should be the last thing before we can close T362867
[14:43:01] <stashbot>	 T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867
[14:53:38] <arturo>	 dhinus: LGTM
[15:03:45] <dcaro>	 dhinus: lgtm, would be really nice if we can find out how to avoid kubeadm from creating that file/force it to create it empty, but well
[15:04:03] <dhinus>	 thanks, I'm merging now
[16:00:30] <andrewbogott>	 Should I assume that since the cookbook is called 'wmcs.openstack.quota_increase' that we don't have a cookbook for decreasing quotas?
[16:00:58] <arturo>	 I don't think we have
[16:01:50] <andrewbogott>	 ok!
[16:02:39] <arturo>	 I was planning on adding quotas support in tofu-infra
[16:03:20] <dcaro>	 andrewbogott: it might support using negative numbers
[16:03:43] <andrewbogott>	 huh, I'll try
[16:04:12] <dcaro>	 would be nice to add a note in the help at least though xd
[16:04:15] <dcaro>	 (if it does)
[16:05:09] <andrewbogott>	 "Increased quotas by -16 cores"
[16:05:30] <andrewbogott>	 seems to have worked
[16:06:23] <dcaro>	 hahaha
[16:06:25] <dcaro>	 nice
[16:07:29] <andrewbogott>	 doesn't work for ram though, says 'expected one argument'
[16:25:09] <andrewbogott>	 dhinus: can I assign you T386467?  I'm assuming you've done that most recently
[16:25:10] <stashbot>	 T386467: [wikireplicas] Create views for new wiki sylwiki - https://phabricator.wikimedia.org/T386467
[16:27:13] <dhinus>	 sure I'm doing all of those at the moment
[16:27:40] <dhinus>	 though I would like to make that a clinic duty thing //eventually// :)
[16:28:12] <andrewbogott>	 Speaking as the one on clinic duty right now: thank you!
[16:28:37] <arturo>	 *cough* or maybe offload to other folks *cough*
[16:28:48] <dhinus>	 :D
[16:29:13] <andrewbogott>	 yeah, come to think of it I think we're only responsible for keeping the service up and not for maintenance actions
[16:29:16] <andrewbogott>	 in theory
[16:29:50] <andrewbogott>	 but until I'm able to remember the name of the team who is responsible I probably don't get to complain!
[16:30:05] <dhinus>	 arturo andrewbogott please add your comments to T382607 which is exactly about that :)
[16:30:06] <stashbot>	 T382607: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607
[16:34:14] <arturo>	 dhinus: I will write a comment tomorrow, but I wonder if others would buy in whatever comes from that ticket?
[16:35:43] <dhinus>	 well, I was thinking about that, I think the only tricky one is option 3 where data-platform-sre would need to do more work
[16:35:54] <arturo>	 nevermind, I read now option 3 as "we ask ...", so that's fair
[16:38:10] <dhinus>	 different topic, I'm trying to understand why the KernelErrors alert fired on cloudgw1003 even with all the filters
[16:39:18] <dhinus>	 e.g. it detected a "priority_crit" message, but the only one I can find should be filtered by kernel-messages-ignore-regex.txt
[16:39:40] <arturo>	 dhinus: in the past I've manually run the script with `bash -x` to see some additional debug information
[16:40:04] <dhinus>	 maybe a race condition between the first run and the creation of kernel-messages-ignore-regex.txt ?
[16:40:32] <arturo>	 nah, they most likely are deployed in the same puppet run
[16:40:44] <dhinus>	 I tried re-running the script now, with an extended timeline of 30days instead of 30mins
[16:40:58] <dhinus>	 priority_crit and priority_err are both 0
[16:41:38] <arturo>	 https://www.irccloud.com/pastebin/SSv5dUOv/
[16:42:03] <arturo>	 https://www.irccloud.com/pastebin/KmTZodP5/
[16:42:21] <arturo>	 this may be it?
[16:42:31] * arturo needs to go offline now
[16:42:43] <dhinus>	 arturo: np, we can catch up tomorrow
[16:42:49] <dhinus>	 I might open a task
[16:43:09] <arturo>	 👍 thanks
[17:12:43] <dhinus>	 there was indeed a race condition, T386850
[17:12:44] <stashbot>	 T386850: [monitoring] KernelErrors alerts trigger incorrectly when a host is reimaged - https://phabricator.wikimedia.org/T386850
[17:13:11] <dhinus>	 or more accurately, a glitch in how the ignore-regex file is created by Puppet
[17:30:40] <dhinus>	 I'm planning to update k8s on toolsbeta tomorrow or Friday (T362868), and I might break things... is anyone actively using toolsbeta this week? cc Raymond_Ndibe Rook 
[17:30:45] <stashbot>	 T362868: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868
[17:31:50] <Rook>	 Safe from my perspective
[17:52:04] * dcaro off
[17:52:08] <dcaro>	 cya tomorrow!
[20:06:33] <arturo>	 dhinus: good finding re:kernel errors puppet race