[08:16:23] <dcaro>	 Oh wow, hughops to all involved
[08:40:09] * volans errand to run, bbiab
[08:45:02] <dcaro>	 dhinus: when you are around, can you double check that you are ok with the naming in https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/192 ?
[08:45:44] <dhinus>	 dcaro: I'm here but was reading the backscroll from the incident :)
[08:46:01] <dcaro>	 no rush, just fyi
[08:47:08] <dhinus>	 looks good, approved!
[08:47:34] <dcaro>	 ack
[09:40:47] <dhinus>	 dcaro: I noticed that components/gen/toolforge_models.py still has the old syntax (enabled instead of notify)
[09:41:55] * volans back
[09:43:04] <dcaro>	 dhinus: oh, it should not, let me check
[09:43:29] <dcaro>	 oh yes, that might be true, it generates the models from the public api endpoint openapi.json, not the code itself
[09:43:40] <dcaro>	 so until it's deployed, it does not fetch the "new" openapi.json
[09:44:13] <dcaro>	 it does not use those models though, as it has the actual original ones instead
[09:44:21] <dcaro>	 (for the components-api side of things)
[09:44:40] <dhinus>	 I see!
[09:44:54] <dcaro>	 I guess we could try to filter the models that are generated somehow (did not look into it), and avoid generating the ones we don't need
[09:45:34] <dhinus>	 not urgent
[09:46:23] <dcaro>	 today I created T439108 also
[09:46:23] <stashbot>	 T439108: [components-api] create a ci scheduled pipeline to update the toolforge models - https://phabricator.wikimedia.org/T439108
[13:47:53] <dcaro>	 dhinus: are you going to rejoin the call?
[13:49:22] <dhinus>	 dcaro: I opened a thread in slack, the mic is no longer working, I even tried the discord app instead of in-browser :)
[15:33:58] <taavi>	 andrewbogott: fwiw I just fixed T379550 for the seanleong-wmde account, in case you feel like poking into the keystone logs in case there's something useful there
[15:33:59] <stashbot>	 T379550: openstack: keystone may be failing to add users to the bastion project in Keystone and/or LDAP - https://phabricator.wikimedia.org/T379550
[15:34:21] <taavi>	 also, note to self: the openstack CLI doesn't give any useful warnings if you try to remove a typo'ed role from an user
[15:34:28] * andrewbogott scowls at that bug
[15:34:50] <taavi>	 not sure if the initial addition that failed was before or after the logstash fix, though
[15:35:47] <andrewbogott>	 happen to know when the...
[15:35:52] <andrewbogott>	 ok, that answers that
[15:36:30] <andrewbogott>	 first mention I see is from 14 minutes ago
[15:36:53] <taavi>	 that's probably me trying to fix it?
[15:36:57] <andrewbogott>	 yeah
[15:37:28] <taavi>	 this is from the support query in -cloud from yesterday that I missed during the codfw saga
[15:40:45] <andrewbogott>	 they were already in there on 2026-05-07
[15:40:58] <andrewbogott>	 so the actual failure is probably lost to time
[15:41:04] <andrewbogott>	 but thank you for fixing!
[15:49:33] <aputhin>	 can someone take a look at this one? T438255 (looks to be potentially tool-specific, but would be good to make sure) 
[15:49:34] <stashbot>	 T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255
[15:50:34] <taavi>	 aputhin: on a quick look that goes in the giant 'we should have larger workers so that we can better schedule large things' pile
[15:52:34] <andrewbogott>	 taavi: is there anything stopping us from having bigger workers other than someone getting around to running the cookbook?
[15:53:12] <taavi>	 andrewbogott: I think we want someone to look at the data (that g.odog collected?) to figure out what kind of a CPU:RAM ratio we want for those
[15:55:28] <andrewbogott>	 ok
[15:58:29] <taavi>	 bliviero: ^ this is one of the things I was forgetting on Tuesday on what we should work in Q2
[16:06:16] <bliviero>	 taavi: noted
[16:15:59] <bliviero>	 for the immediate issue that was filed (T438255) what should our response be?
[16:16:00] <stashbot>	 T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255
[16:18:50] <dcaro>	 the cluster seems pretty busy (compared to my memory of some time ago at least)
[16:23:19] <bliviero>	 are there servers that are depooled that could be brought back in for capacity?
[16:24:35] <andrewbogott>	 no, but we could create more workers (of the existing size) pretty easily. dcaro, taavi, any objection to me adding another 6 workers? Or X workers?
[16:25:07] <taavi>	 nope
[16:25:20] <dcaro>	 not really no +1
[16:26:36] <andrewbogott>	 dcaro: you were looking at the capacity section on https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?from=now-12h&to=now&timezone=utc&var-cluster_datasource=P8433460076D33992&var-cluster=tools ?
[16:26:44] <andrewbogott>	 Looks like 50% usage to me but maybe I'm misreading
[16:26:51] <andrewbogott>	 actual RAM  usage I mean
[16:27:04] <andrewbogott>	 Oh, I guess for scheduling new jobs it's the other panel that matters
[16:27:18] <andrewbogott>	 190%
[16:27:21] <dcaro>	 yep, I was looking at the reservations
[16:27:32] <andrewbogott>	 does that cap out at 200%?
[16:27:33] <dcaro>	 yep, so we might want to tweak that again
[16:28:45] <taavi>	 for scheduling, the main thing that matter is the largest free slot on any one node. we used to have a dashboard with that visible but I don't immediately remember which one
[16:29:05] <dcaro>	 yep, not finding it either :/
[16:32:34] <dcaro>	 hmm....
[16:32:36] <andrewbogott>	 'largest free slot' is still governed by the overprovision number right?
[16:32:47] <andrewbogott>	 So if we moved from 2x to 3x...
[16:33:01] <dcaro>	 all seem to have >5G memory free
[16:33:06] <dcaro>	 https://www.irccloud.com/pastebin/4b8yoteF/
[16:33:37] <dcaro>	 (unless I'm reading that wrong)
[16:33:47] <taavi>	 dcaro: the logs in the task say 'Insufficient cpu', not memory
[16:34:03] <dcaro>	 oh, that's interesting
[16:34:20] <dcaro>	 (oh, too, that's free memory, not unreserved memory I think)
[16:34:58] <dcaro>	 cpu is less reserved than memory even :/
[16:35:10] <andrewbogott>	 in that dash, is 'limits' the max allowed reservation
[16:35:12] <andrewbogott>	 ?
[16:35:17] <andrewbogott>	 Because if so, we are nowhere close
[16:35:37] <andrewbogott>	 (and not close for RAM either)
[16:36:24] <dcaro>	 it's the requests
[16:36:39] <dcaro>	 the limits is the "maximum memory/cpu allowed to be used" sort of thing
[16:37:07] <dcaro>	 the request is "minimum that my pod will need allocated to run"
[16:37:22] <andrewbogott>	 why do 'limits' change over time?
[16:38:09] <dcaro>	 it's the sum of all pods that are currently defined in the cluster
[16:38:19] <taavi>	 cronjobs starting and stopping, I assume
[16:39:57] <dcaro>	 there was a weird jump this morning (not that it should start making things break though) https://usercontent.irccloud-cdn.com/file/cTrD3A9I/image.png
[16:40:16] <dcaro>	 that was yesterday night
[16:40:58] <dcaro>	 in any case, more workers should be ok, make sure to use the bigger images though (2x the current one iirc is what we said?)
[16:41:00] <andrewbogott>	 I don't see anything that suggests we have a capacity crunch. So... why can't that user scheule?
[16:41:27] <dcaro>	 it needs to have a node with >5G 'requestable'
[16:42:08] <dcaro>	 and 3 cpu
[16:42:17] <dcaro>	 (at the same time)
[16:42:23] <dcaro>	 not sure if there's one, maybe?
[16:43:59] <andrewbogott>	 on average there should be quite a few
[16:45:09] <dcaro>	 interesting, this is giving me quite different numbers
[16:45:30] <dcaro>	 dcaro@tools-bastion-15:~$ kubectl-sudo describe nodes | grep -A5 "Allocated resources"
[16:45:47] <dcaro>	 https://www.irccloud.com/pastebin/LEuMrilt/
[16:46:03] <dcaro>	 cpus are almost saturated
[16:46:05] <volans>	 sorry I'm busy with production's repooling, but have you already compared reserved vs allocated resources?
[16:47:15] <dcaro>	 on it yes, I have the suspicion that the graphs we have are not correctly reflecting the actual reserved amounts
[16:47:23] <andrewbogott>	 allocated is going to be less (maybe much less) than reserved, right?  So 'reserved' is a good upper bound
[16:47:35] <dcaro>	 yep
[16:47:49] <dcaro>	 reserved is what the scheduler uses to decide where/if to put the pod in the node
[16:47:51] <volans>	 yes and the scheduler looks at reserved IIRC
[16:48:30] <dcaro>	 all but the control nodes show >90% cpu reservation using that command
[16:49:13] <andrewbogott>	 but we allow up to 180% right?
[16:49:17] <andrewbogott>	 Or do you mean 90% of allowed?
[16:51:12] <dcaro>	 yep, 90 of allowed, as in, only 10% is left to be considered when putting a new pod in it
[16:51:40] <andrewbogott>	 wow, very different from what grafana says!
[16:51:56] <andrewbogott>	 Ok, most workers right now are 8cores/16GB, I'm going to make more workers that are 16/16
[16:52:14] <dcaro>	 grafana might be averaging all the workers? but even then, there's many >90% (only 6 control nodes)
[16:52:25] <andrewbogott>	 We still need to do some real number crunching and graph fixing but that that should help for now.
[16:52:42] <andrewbogott>	 And I assume we're only really talking about nfs workers today?
[16:54:25] <dcaro>	 I think all of them, but let me double check
[16:57:13] <dcaro>	 andrewbogott: you mean 16cores/32G ram?
[16:57:27] <andrewbogott>	 no, 16/16
[16:57:32] <andrewbogott>	 Since we seem to have plenty of RAM
[16:57:48] <andrewbogott>	 and also because that's a flavor that exists, otherwise I'll have to start with a tofu patch
[16:58:32] <dcaro>	 non-nfs also have capacity issues yep
[16:59:06] <andrewbogott>	 hm, actually openstack is lying to me about available flavors...
[16:59:47] <andrewbogott>	 yeah, have to make a tofu patch regardless.
[17:00:08] <andrewbogott>	 So, do you think better to skip ahead to 16/32?
[17:00:56] <dcaro>	 that should be ok yes, or even more cpu too (given that it seems our assumption that we use more ram than cpu might not be correct)
[17:01:54] <dcaro>	 the VMs having idle memory is not an issue right? (openstack will overallocate?)
[17:01:56] <andrewbogott>	 At some point it gets  hard to schedule giant VMs due to roundoff on the hypervisors...
[17:02:05] <andrewbogott>	 No, we never overprovision RAM, only CPUs
[17:02:24] <andrewbogott>	 but we have lots of RAM so it is usually not an issue.
[17:02:31] <dcaro>	 okok, good to know, so then probably more cpu to ram ratio is better (we can double check later too, once we have good data)
[17:04:08] <dcaro>	 https://www.irccloud.com/pastebin/HKaQsoKs/
[17:05:42] <andrewbogott>	 dcaro, taavi, https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/364
[17:07:14] <dcaro>	 LGTM, feel free to make them smaller if you think they will bring problems on the openstack side
[17:07:54] <andrewbogott>	 Should be OK. I wouldn't want to do 128 though :)
[17:09:58] <dcaro>	 xd ack
[17:10:35] <dcaro>	 I'm going to take 15min before the discord call to stretch my legs, ping me if you need any help, we should follow up on the stats though
[17:11:52] <andrewbogott>	 sure. It's just the long wait for cookbooks to complete now.
[17:25:00] <andrewbogott>	 dcaro: oops, I forgot about quotas.  https://phabricator.wikimedia.org/T439157
[17:26:24] <dcaro>	 andrewbogott: got the meeting it 6 min, just +1d, please self-provision xd
[17:27:08] <andrewbogott>	 thanks, I will
[18:37:52] * dcaro off
[18:37:55] <dcaro>	 cya on monday!
[19:20:41] <dhinus>	 I rotated the etherpad and will cover clinic duty tomorrow, then I'll hand it over to r.aymond on monday
[19:20:53] * dhinus off