[07:42:44] <dcaro>	 Morning!
[08:24:13] <dhinus>	 morning!
[08:44:51] <godog>	 greetings
[10:19:13] <taavi>	 anyone in the mood of dealing with the tools nfs space alert?
[10:25:04] <dcaro>	 andrewbogot.t mentioned that he'd be handling it iirc? I'm sure he will be happy though if anyone wants to tackle it instead xd
[10:53:34] * dcaro lunch
[13:19:52] <andrewbogott>	 I ran the standard truncate commands and got it /just barely/ under the alert threshold but now it's back again.
[13:20:22] <andrewbogott>	 I can take another pass today
[13:22:17] <andrewbogott>	 dcaro, ever seen this health notice before?  https://phabricator.wikimedia.org/T434402
[13:33:10] <dcaro>	 andrewbogott: it does not ring any bells no
[13:33:22] <dcaro>	 that's the output of ceph status?
[13:33:29] <andrewbogott>	 health detail
[13:35:10] <dcaro>	 we are storing the db in the same device as the data, splitting them might help (though then we need 2x the devices), it matches the other task though (drives busy sound like a likely cause of it)
[13:35:31] <andrewbogott>	 yeah, I agree that it probably just means 'osd is busy'
[13:36:19] <dcaro>	 oh, it's in many
[13:36:31] <andrewbogott>	 another ceph question: I can find close to 0 docs about configuring ceph via ceph.conf, it's all about the live 'ceph osd set <whatever>' config
[13:36:36] <dcaro>	 wait no, the many is the other slow ops
[13:36:39] <andrewbogott>	 dcaro: no, that's the bluestore thing you're seeing which is a different issue
[13:37:09] <andrewbogott>	 1) Do you know of anywhere I can find actual complete docs about the syntax/options for ceph.conf?
[13:37:27] <andrewbogott>	 2) Is it possible that some config is /only/ manageable via live config and can't be configured in the static file at all?
[13:38:55] <dcaro>	 2) I think it might be, there's things you can only do in ceph config (filters, targets for config, etc.), would not be surprised if some special config was only available there
[13:39:08] <dcaro>	 1) looking (I remember something, but not 100% sure)
[13:41:00] <andrewbogott>	 If 2 is true that's bad news for tracking/maintaining our config :/
[13:44:47] <dcaro>	 I'm starting to suspect that the list of all possible config values might only be gotten with `ceph config ls`, but not sure if there's a way to verify if all can be set in config
[13:45:11] <andrewbogott>	 dcaro: if you're doc-searching, my immediate question is "Can I set bluestore_slow_ops_warn_threshold in ceph.conf and if so where and what is it called?" The merged patch doesn't work but I have live-tested several other versions without results.
[13:46:01] <dcaro>	 so far it seems doable
[13:46:03] <dcaro>	 https://www.irccloud.com/pastebin/ZExnFFXl/
[13:46:45] <andrewbogott>	 that doesn't reference the .conf does it? Or say what section it would be in?
[13:46:47] <dcaro>	 can you see the option you set in `ceph config show`?
[13:47:14] <andrewbogott>	 yes, it's always 1 no matter what I do in the config
[13:47:20] <dcaro>	 it does not no, there's the mention of runtime: true, but might not mean that it's ceph.conf-able
[13:47:24] <andrewbogott>	 although there's always the chance that I'm missing the obvious
[13:48:10] <dcaro>	 hmmm
[13:48:22] <dcaro>	 did you try to set it in codfw?
[13:49:46] <dcaro>	 this worked
[13:49:47] <dcaro>	 https://www.irccloud.com/pastebin/Zu1XKApu/
[13:50:46] <dcaro>	 oh, you are setting in on the mon section?
[13:50:56] <dcaro>	 I think it should be in the osd one
[13:51:04] <andrewbogott>	 I've tried mon, osd, default
[13:51:06] <andrewbogott>	 no effect
[13:51:09] <andrewbogott>	 but please check my work!
[13:51:09] <dcaro>	 :/
[13:52:02] <dcaro>	 okok, let me play a bit in codfw with the config
[13:52:30] <andrewbogott>	 ty
[13:52:57] <andrewbogott>	 I'm not a huge fan of the docs saying "eh, no one really uses .conf anymore, we're not going to bother to document it"
[13:53:10] <dcaro>	 yep
[13:53:18] <andrewbogott>	 but I guess we could follow their lead and script everything instead >:|
[13:53:21] <dcaro>	 same for cephadm/vs anything else (maybe except rook)
[13:53:45] <dcaro>	 https://www.irccloud.com/pastebin/8i8GTZQu/
[13:53:57] <andrewbogott>	 what? Where did you put the config line?
[13:54:01] <dcaro>	 https://www.irccloud.com/pastebin/c3fUMv7Q/
[13:54:10] <andrewbogott>	 what the heck?
[13:54:13] <dcaro>	 just moved it under osd, and restarted the daemon
[13:54:13] * andrewbogott tries it again
[13:56:58] <andrewbogott>	 ...I still can't make it work.
[13:57:02] <andrewbogott>	 https://www.irccloud.com/pastebin/ZGQCjLTi/
[13:57:12] <andrewbogott>	 oh wait!
[13:57:14] <andrewbogott>	 Now it picked it up
[13:57:23] <andrewbogott>	 for some reason it took 30 seconds after restarting the daemon
[13:57:29] <andrewbogott>	 so that's why I thought it wasn't doing anything...
[13:57:42] <andrewbogott>	 Anyway, this is great, thank you! I will fix that patch.
[13:59:25] * andrewbogott wonders why some of this config uses 2-space indent and some of it 4-space indent
[14:01:12] <dcaro>	 I think it does not matter for the file itself, might be related to old puppet template spacing maybe
[14:01:58] <andrewbogott>	 yeah, I'm sure it doesn't matter, just looks weird.
[14:01:58] <andrewbogott>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324319
[14:27:44] <dcaro>	 +1d
[14:30:27] <andrewbogott>	 thx
[14:32:51] <andrewbogott>	 godog: I like this dashboard! One thing bliviero and I talked about when looking at a different capacity dashboard is that if we're committed to having rack redundancy we need to start saying "100% full" when we fill 3 racks of hosts rather than 4 racks. But I'm not sure if that's feedback for this dash or for a different future dashboard.
[14:33:09] <andrewbogott>	 (The one we were looking at is https://grafana.wikimedia.org/d/000000579/wmcs-openstack-eqiad-summary?orgId=1&from=now-7d&to=now&timezone=utc&var-hypervisor=$__all&refresh=1m  which the top line ought to have the same stats as yours, I think?)
[14:35:28] <andrewbogott>	 dcaro: take two for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324319
[14:36:13] <godog>	 andrewbogott: interesting, yes same metrics although I don't see the oversubscription ratio in the expression. re: effective capacity and rack redundancy yes we'll have to define exactly "capacity left" means in that context
[14:37:17] <andrewbogott>	 godog: also someday soon I'd like to detect when/how often oversubscription is actually throttling cpu usage on VMs.
[14:37:30] <godog>	 that'd be useful to know for sure
[14:37:55] <dcaro>	 andrewbogott: LGTM, you can run a ppc if you want more certainty
[14:38:01] <godog>	 I'll keep poking a little at the placement capacity overview dashboard and see if we can get some signal
[14:40:58] <andrewbogott>	 The other thing about oversubscription that makes me nervous is that we should audit and make sure that we're consistent with enabling hyperthreading on cloudvirts since that is effectively also an oversubscription factor.
[14:41:58] <andrewbogott>	 the best practice for hyperthreading with kvm is: no one can decide
[14:53:13] <godog>	 the "point in time" audit for consistency is actually easy, and the answer is that we're consistent: cumin 'cloudvirt*' 'cat /sys/devices/system/cpu/smt/active'
[15:00:02] <andrewbogott>	 oh good
[16:21:54] * dcaro off
[16:21:56] <dcaro>	 cya tomorrow!
[16:36:12] * dhinus off
[17:02:06] <andrewbogott>	 taavi: for the k8s upgrade, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1324360
[17:02:28] <andrewbogott>	 includes some bullseye cruft that we may want to clean up elsewhere
[17:26:02] <andrewbogott>	 taavi: also, is it just coincidence that we're running stern 1.32 with k8s 1.32 or is there a reason for them to be in lockstep?