[07:11:23] <dcaro>	 bd80.8: nice :), we should definitely have it in the radar, and help with it
[07:17:12] <dcaro>	 artur.o: I have returned to using terminator xd
[07:28:36] <arturo>	 :-)
[07:52:51] <dcaro>	 it's pretty easy to create url handlers :)
[07:52:53] <dcaro>	 https://usercontent.irccloud-cdn.com/file/5FPOvlm5/image.png
[07:53:29] <dcaro>	 https://www.irccloud.com/pastebin/Q8L2jryh/
[07:55:06] <taavi>	 the stuck procs check is flapping on tools-k8s-worker-nfs-29, does anyone want to have a closer look or should I just reboot?
[07:56:18] <dcaro>	 it went down already is it recovering?
[07:57:39] <dcaro>	 oh wow, I was not seeing it in the graph because the color for it that was auto-selected was dark brown on a black background xd
[07:58:03] <dcaro>	 it looks steadily high yep, probably something got stuck
[07:58:48] <dcaro>	 since yesterday morning more or less (been piling up D processes)
[07:59:38] <dcaro>	 anyhow, feel free to reboot
[08:28:20] <arturo>	 dcaro: I might copy that plugin!
[08:29:34] <dcaro>	 feel free :), and share any you might come up with too! ;)
[08:30:20] <arturo>	 I'm using layouts a lot lately:
[08:30:22] <arturo>	 https://usercontent.irccloud-cdn.com/file/Fwv1ZEjA/image.png
[08:32:00] <arturo>	 for example the reimage ones open a couple of ssh sessions to cumin servers AND open the mgmt password
[08:41:35] <arturo>	 I'm seeing policyReports similar to yesterday's, but related to runAsGroup this time
[08:49:23] <dcaro>	 hmm, maybe it just stops at the first issue, so getting rid of one just lets it get to the next?
[08:49:41] <dcaro>	 blancadesal: have you deployed the 'tool in the path only' version of the envvars on toolsbeta?
[08:50:34] <dcaro>	 (/me getting things like Jun 26 08:48:19 toolsbeta-nfs-3 uwsgi-toolsdb-replica-cnf-web[246473]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.svc.toolsbeta.eqiad1.wikimedia.cloud:30003/envvars/v1/envvar/TOOL_TOOLSDB_USER)
[08:51:01] <blancadesal>	 dcaro: if you mean https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/33, then no
[08:51:19] <dcaro>	 yep that, okok, looking
[08:55:11] <dcaro>	 oooohhhh, some change in toolforge-weld broke the replicacnf code (it expects to catch an HTTPError when an envvar is not there)
[08:55:32] <dcaro>	 this might not be breaking in tools because we might not upgrade the toolforge-weld package there
[08:55:53] <arturo>	 I think I know what is happening with the policyReports
[08:56:33] <arturo>	 when we create a policy for Pods, kyverno autogenerates policies for Deployments and other pod-generators
[08:56:44] <dcaro>	 wait no, scratch that, looking
[08:57:03] <dcaro>	 (for the replicacnf)
[08:59:20] <dcaro>	 blancadesal: hm... I think that the post endpoint for envvars might already be broken
[09:00:15] <dcaro>	 wait no, the issue is that the interface changed
[09:00:38] <dcaro>	 now to create envvars you need to pass `{"value": ..., "name": ...}` and not put the name on the path :/
[09:06:32] <blancadesal>	 dcaro: do you mean maintain-dbusers should have been updated at some earlier change of the envvars service but wasn't?
[09:06:54] <arturo>	 dcaro: could you please +1 here? https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/46
[09:07:01] <dcaro>	 blancadesal: I think so, looking in tools, it might have been broken for a while
[09:08:45] <arturo>	 thanks!
[09:12:41] <dcaro>	 blancadesal: hmm... on tools using the envvar name in the path works :/
[09:14:20] <taavi>	 possibly stupid question: what's calling this in toolsbeta? I thought toolsbeta users don't have replica access
[09:14:48] <dcaro>	 taavi: nothing, just the tests
[09:15:19] <dcaro>	 blancadesal: in toolsbeta we have version 0.0.50, in tools 0.0.49 of envvars-api
[09:16:27] <dcaro>	 yep, the difference is https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/commit/42829b67a8c43dd119ec4ed247bacd964ffc5b01
[09:16:37] <dcaro>	 that's the commit that removes the endpoints I think
[09:17:14] <arturo>	 heads up, setting kyverno policies in tools to Enforce
[09:18:11] <blancadesal>	 dcaro: not sure why raymond didn't deploy that in tools. also, I think I remember there was some issue with that patch to begin with. It was rolled back the first time
[09:19:00] <taavi>	 related: T367961
[09:19:00] <stashbot>	 T367961: envvars-api 0.0.50 depends on unreleased envvars-cli changes - https://phabricator.wikimedia.org/T367961
[09:19:56] <dcaro>	 I see, yep, that's what the replicacnf is having on toolsbeta, it needs changing
[09:20:51] <dcaro>	 toolforge-deploy says 50 is the version, I'll revert that while we fix the replica_cnf code to work with the new endpoints
[09:23:07] <taavi>	 re-opened the task
[09:23:46] <dcaro>	 oh, just created a new one xd
[09:23:49] <dcaro>	 it's slightly different
[09:23:50] <dcaro>	 T368516
[09:23:50] <stashbot>	 T368516: [envvars-api] version 0.0.50 introduces breaking changes that need adapting for replica_cnf service - https://phabricator.wikimedia.org/T368516
[09:23:58] <dcaro>	 happy to merge if you prefer
[09:24:08] <dcaro>	 revert mr https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/352
[09:24:15] <taavi>	 those are two slightly different bugs I think
[09:24:34] <taavi>	 so I marked yours as a subtask since that needs to be fixed before rolling forward
[09:25:10] <dcaro>	 ack
[09:25:42] <dcaro>	 xd
[09:25:47] <dcaro>	 blancadesal: do you want to work on this with me?
[09:26:11] <dcaro>	 (feel free to say no)
[09:26:29] <blancadesal>	 dcaro: yes :)
[09:26:48] <taavi>	 dhinus: is T368066 on your radar? just needs the add-wiki cookbook to be ran, I saw someone ping about that on -data-persistence
[09:26:49] <stashbot>	 T368066: Prepare and check storage layer for btmwiki - https://phabricator.wikimedia.org/T368066
[09:27:02] <dcaro>	 👍 we need to make some changes to the replica_cnf, can be rolled up with the changes to use the `tools` in the url too
[09:27:08] <arturo>	 mmmm
[09:27:11] <blancadesal>	 dcaro: I'd like to finish up a couple things I was doing first. do you want to pair after lunch?
[09:27:18] <dcaro>	 now we have functional tests in toolsbeta so that's nice to test :)
[09:27:28] <dcaro>	 blancadesal: sure, works for me
[09:27:28] <arturo>	 just noticed my announcement email about today's operation never made it to the mailing list because requiring approval?
[09:27:37] <taavi>	 arturo: I approved it yesterday
[09:27:41] <blancadesal>	 arturo: I got it
[09:27:45] <arturo>	 oh, ok, thanks
[09:27:49] <dcaro>	 arturo: I got it too yep
[09:27:52] <arturo>	 I think we should figure out my permissions
[09:28:00] <taavi>	 and I think fixed the permissions to not hold your messages again
[09:28:09] <arturo>	 taavi: ok, thank you
[09:28:10] <taavi>	 but it's certainly visible at https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/
[09:28:56] <arturo>	 great
[09:29:13] <blancadesal>	 dcaro: sent a meeting invite
[09:29:21] <dcaro>	 blancadesal: thanks!
[09:29:30] <blancadesal>	 👍
[09:39:19] <dcaro>	 arturo: maintain kubeuser hang alert fired, are you doing anything with it?
[09:46:40] <dcaro>	 oh, it's updating all the poilicies
[09:48:05] <arturo>	 yeah
[09:48:08] <arturo>	 it is kind of expected
[09:48:29] <arturo>	 run just finished
[09:48:30] <arturo>	 2024-06-26T09:47:41.454079741Z finished run, reconciled 22 admins, 3311 tool accounts
[09:49:20] <dhinus>	 taavi: no I didn't see T368066, but I randomly noticed there was a new wiki because it triggered the "private data" alert a few days ago
[09:49:20] <stashbot>	 T368066: Prepare and check storage layer for btmwiki - https://phabricator.wikimedia.org/T368066
[09:49:35] <arturo>	 dcaro: taavi: any concern with me roll-rebooting k8s workers to try surface pod policy problems?
[09:49:56] <dcaro>	 arturo: you can first try with a single tool you control
[09:50:13] <dhinus>	 taavi: I think I can claim T368066 and run the cookbook. thanks for the pointer
[09:50:13] <dcaro>	 to avoid breaking many
[09:50:14] <taavi>	 yeah, let's start smaller scale please. individual tools, maybe an individual worker, etc
[09:50:17] <dcaro>	 (in the worst case)
[09:50:42] <arturo>	 ok
[09:51:06] <dcaro>	 you can use wm-lol, if you want (I have that one, not sure if it has a continuous job though, it has a webservice)
[09:53:45] <arturo>	 I will rollout-restart the pods for openstack-browser
[09:53:55] <arturo>	 the deployment resource was created from an old webservice-cli version
[09:56:32] <arturo>	 done, no problems
[09:56:50] <arturo>	 this confirms that new pods generated by old deployments can work with policies in Enforcement mode
[09:57:57] <arturo>	 to reduce the noise of the PolicyReports, I would like to merge this
[09:57:58] <arturo>	 https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/51
[10:03:13] <arturo>	 dcaro: could you please approve?
[10:04:11] <dcaro>	 will that allow to create bad deployments though?
[10:04:54] <arturo>	 kyerno will not enforce anything on the deployments
[10:05:40] <dcaro>	 don't we want that?
[10:05:47] <arturo>	 it should not matter for deployments created via toolforge-jobs or webservice-cli
[10:05:56] <arturo>	 because they already contain the right pod templates
[10:06:14] <dcaro>	 (not taking bugs into account I guess)
[10:07:11] <arturo>	 kyverno will still enforce pods normally. Not having the autogen rules is actually more closely resembling what PSP was doing
[10:08:27] <dcaro>	 so any deployment that is not passing the validation, will generate failing pods?
[10:08:52] <dcaro>	 (but the users will only notice when the deployment generates the pods, not when trying to create/change the deployment?)
[10:08:53] <arturo>	 no, because kyverno is mutating pods resources, then validating them
[10:10:51] <dcaro>	 hmm, that's not clear to me, so there's no way a deployment can generate a pod that does not pass the validation after the mutation itself?
[10:11:08] <arturo>	 correct
[10:11:13] <arturo>	 this is all a bit redundant
[10:11:34] <arturo>	 I corrected the templates on toolforge-jobs/webservice-cli to generate the right attributes
[10:11:44] <arturo>	 then we have a kyverno policy to mutate and inject the same if not present
[10:11:57] <arturo>	 then we have a kyverno policy to validate the config is correct
[10:12:07] <arturo>	 if we remove the first step, then later 2 still apply
[10:12:12] <arturo>	 I'm a bit worried about cronjobs
[10:12:22] <arturo>	 because they have an intermediary indirection
[10:12:25] <dcaro>	 what happens if the fields are already present?
[10:12:28] <dcaro>	 in step 1
[10:12:29] <arturo>	 cronjob -> generates a job -> generates a pod
[10:12:47] <arturo>	 if the fields are already present, nothing happens, mutation will do nothing
[10:12:50] * arturo brb
[10:13:13] * arturo back
[10:15:07] <dcaro>	 as long as the pods get validated we should be ok security-wise I guess
[10:16:02] <arturo>	 exactly
[10:18:42] <dcaro>	 I don't fully grasp how that will reflect on user's perspective though, but we will get rid of kyverno anyhow in the next upgrade, so maybe not worth the effort on making it supernice
[10:20:07] <arturo>	 I don' thing the patch has any user-visible effect
[10:20:23] <arturo>	 but it will greatly reduce the PolicyReport noise that we are currently getting
[10:22:48] <dcaro>	 well, to be fair, it's "real" noise, as in those objects are not passing the validations so they should be changed
[10:23:03] <dcaro>	 but I see your point
[10:28:56] <arturo>	 thanks, deploying
[10:51:52] <arturo>	 I would like to merge https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/49 next
[10:52:11] <arturo>	 i.e, drop the PSP config for each account
[11:08:26] <dcaro>	 maybe wait for a day?
[11:08:33] <dcaro>	 just to make sure nothing breaks with the new stuff
[11:08:49] <arturo>	 ok
[11:54:41] <dcaro>	 thanks
[12:23:20] <arturo>	 opinions? https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/52
[12:25:00] <taavi>	 hmm, I wonder if that should be a replicaset instead of a deployment
[12:25:20] <taavi>	 if not that seems ok
[12:27:24] <arturo>	 all upstream docs about replicaset refer to using deployment instead
[12:47:26] <dcaro>	 hmpf my reimage of cloudcontrol1006 failed, can I just use `--now` to retry?
[12:48:05] <taavi>	 why are you reimaging cloudcontrols?
[12:48:37] <dcaro>	 ooops that was cloudcephosd1006
[12:48:39] <dcaro>	 not cloudcontrol xd
[12:48:46] <taavi>	 phew
[12:49:06] <taavi>	 for --new: usually yes, but it depends on how it failed
[12:50:39] <arturo>	 yeah
[12:54:52] <dcaro>	 it got hung on the install screen, when I connected it just showed the install menu on the top and a grey screen
[12:57:10] * arturo food
[12:57:19] <dcaro>	 oh, now it showed something, network configuration failing (dhcp not found)
[12:59:27] <taavi>	 a good hack is to try to switch to an another screen window after attaching to the console of a a running reimage
[13:00:12] <taavi>	 that forces it to re-draw the entire screen
[13:01:46] <taavi>	 dcaro: hmm, which part was complaining about no dhcp leases? the bios or debian-installer?
[13:02:20] <taavi>	 if the former, I'd check it's configured to boot off of the correct interface, as both of its ports are showing as down on the switch side https://phabricator.wikimedia.org/P65477
[13:03:20] <dcaro>	 looking
[13:03:24] <dcaro>	 https://usercontent.irccloud-cdn.com/file/vnqxacfD/image.png
[13:05:12] <dcaro>	 I think that the interface changed names after the reinstall
[13:05:55] <taavi>	 the linux-side interface names changing is not a problem
[13:06:32] <dcaro>	 the installer sees enp175s0f0np0 and enp175s0f0np0, netbox has a different name
[13:06:42] <taavi>	 if you switch to a shell in the installer, is it showing all the interfaces? it should have two built-in 1G nics and then two additional ones on the 10G cards
[13:07:57] <dcaro>	 yep, sounds like it
[13:07:59] <dcaro>	 https://www.irccloud.com/pastebin/JTTVBdFg/
[13:08:29] <taavi>	 yeah, that looks correct
[13:14:25] <dcaro>	 hmm, topranks do you remember if there's any special config/manual change needed to reimage cloudcephosd nodes?
[13:15:10] <topranks>	 dcaro: no 
[13:15:20] <topranks>	 not that I recall 
[13:16:00] <topranks>	 newer OS might use different interface naming but the cookbooks will just use the "primary_ip" attribute for the DHCP part regardless of name 
[13:16:11] <topranks>	 and then should import the interfaces (with new names) once installed 
[13:16:28] <topranks>	 dhcp failed i see hmm 
[13:16:32] <taavi>	 so I wonder if it's trying to DHCP off of the "secondary" port
[13:16:36] <topranks>	 what host is it?
[13:16:46] <dcaro>	 cloudcephosd1006
[13:17:05] <dcaro>	 taavi: with the name change I'm not sure which one would be the secondary port
[13:17:53] <topranks>	 hmm I wonder 
[13:18:04] <topranks>	 dcaro: I assume the cookbook has now failed?
[13:18:06] <dcaro>	 probably enp175s0f0np0 would be ens3f0np0 in netbox, but not sure, is there a way I can check which link a port is connected from the machine itself?
[13:18:18] <dcaro>	 topranks: I have not retried
[13:18:20] <dcaro>	 should I?
[13:18:41] <topranks>	 perhaps, I was just checking, I see on the install server there is no specific config for that host right now 
[13:18:56] <topranks>	 but that makes sense if the cookbook has failed/finished 
[13:18:56] <dcaro>	 okok, let me rerun with `--new`
[13:19:30] <topranks>	 in terms of the names you are right, enp175s0f0np0 == ens3f0np0, just look at the numbers at the end, f0, p0 or f1,p1 
[13:19:32] <dcaro>	 oh, new prompt, puppet version should be 7 right?
[13:20:09] <topranks>	 that depends if the role has been updated to work with puppet 7
[13:20:16] <topranks>	 most have 
[13:21:47] <dcaro>	 should be ok yes (cloudcephosd1001 is running puppet 7)
[13:21:53] <topranks>	 the NIC firmware on that host is on  21.60.2 
[13:22:18] <topranks>	 that is slightly lower than perferred, 21.85, but afaik does not have the driver issue which causes the link to not work inside the debian installer (which we have in 22.x)
[13:22:31] <topranks>	 dcaro: yeah if others are puppet 7 go with the same 
[13:23:25] <dcaro>	 okok, running
[13:23:36] <dcaro>	 dgcp config passed now
[13:24:08] <dcaro>	 topranks: is there anything I can do to upgrade it at the same time?
[13:24:35] <topranks>	 dcaro: the firmware cookbook can be used but let's leave it if you have reimage 
[13:24:56] <dcaro>	 okok, I might want to do that as part of the reimages for the rest of osds too
[13:24:58] <topranks>	 21.6 is good afaik so I don't think urgent.  but just to say I checked and it's not 22.x, which causes a problem very like what you had 
[13:27:29] <dcaro>	 it's now trying to configure the network with dhcp
[13:27:35] <dcaro>	 (does not say which interface it's using :/)
[13:27:50] <topranks>	 hmm yeah, the speculation it's using the wrong one could be right 
[13:28:18] <dcaro>	 and failed
[13:29:18] <dcaro>	 when retrying it says it's 'detecting link on enp175s0f0np0', is that the correct one? (it's the one in nb with the tagged interface + ip)
[13:29:36] <dcaro>	 hmm... why is it tagged?
[13:29:48] <dcaro>	 are we configuring vlans at the OS level?
[13:29:49] <topranks>	 tha'ts the right one 
[13:29:55] <topranks>	 yes 
[13:30:03] <topranks>	 but the issue here is that the link is down....
[13:30:07] <dcaro>	 okok
[13:30:07] <topranks>	 https://www.irccloud.com/pastebin/x8izv4H6/
[13:30:16] <dcaro>	 taavi said the same yes
[13:30:27] <dcaro>	 should I try to upgrade the firmware?
[13:30:40] <dcaro>	 I can also try configuring it manually, see if it comes up
[13:30:51] <dcaro>	 oh, someone is connected to the console too?
[13:31:36] <dcaro>	 topranks: does it show up now?
[13:32:20] <topranks>	 that's me 
[13:32:30] <topranks>	 yeah it shows up after "ip link set up"
[13:33:22] <topranks>	 https://www.irccloud.com/pastebin/WyB6Cgns/
[13:33:26] * dcaro snooping on your debugging :)
[13:33:37] <topranks>	 I am really not sure what is going wrong in that case......
[13:33:49] <topranks>	 you'd think the debian-installer is doing something very link those commands 
[13:33:55] <topranks>	 bring link up, run dhcp client 
[13:34:08] <dcaro>	 ip a
[13:34:11] <dcaro>	 oops
[13:34:24] <topranks>	 at least it's not the root password :P
[13:34:32] <dcaro>	 hahahaha
[13:34:58] <dcaro>	 that's better yep
[13:35:29] <topranks>	 this AI stuff is amazing 
[13:35:35] <dcaro>	 xd
[13:35:35] <topranks>	 it's helping me fix it on the console :P
[13:36:08] <dcaro>	 yep, network looks good now no?
[13:36:11] <topranks>	 I've two ideas, but neither are very scientific 
[13:36:28] <dcaro>	 I'm open to trying things
[13:36:42] <dcaro>	 oh, now it passed
[13:36:52] <topranks>	 ugh what the hell 
[13:36:56] <topranks>	 my two ideas were 
[13:37:14] <topranks>	 1) shut down the second port on the switch side, to force it to use the right one, just in case that is what's causing it 
[13:37:23] <topranks>	 (but it doesn't seem like it as it tries all 4 ports)
[13:37:29] <topranks>	 2) upgrade firmware 
[13:37:44] <topranks>	 (but seeing as it came up when we forced from shell doesn't seem like firmware/driver incompatibility)
[13:37:58] <topranks>	 3) leave this for now see if it completes?
[13:38:03] <dcaro>	 sure
[13:38:24] <dcaro>	 I'll have to reimage a bunch more, so if it's a common issue it will pop up again
[13:38:28] <dcaro>	 oops, another error xd
[13:38:46] <topranks>	 hmm 
[13:39:00] <dcaro>	 this one's different
[13:39:03] <dcaro>	 https://www.irccloud.com/pastebin/i0IdKGOT/
[13:39:26] <topranks>	 I'm lost on that one 
[13:40:16] <dcaro>	 I'll keep digging :) thanks for the debugging session ;)
[13:40:22] <topranks>	 maybe something in the DHCP stuff wasn't passed properly cos I ran it manually 
[13:40:32] <topranks>	 you could try if it happens again 
[13:40:36] <topranks>	 drop to a shell 
[13:40:39] <dcaro>	 it seems to get there, but it's getting the wrong content
[13:40:47] <topranks>	 do the "ip link set dev <int_name> up" 
[13:40:53] <topranks>	 same as I did 
[13:41:08] <topranks>	 but don't run the DHCP from the shell, exit once it's up and see does it continue 
[13:41:16] <topranks>	 but tbh none of that should be needed 
[13:42:52] <dcaro>	 hmmm, maybe dns
[13:42:55] <dcaro>	 yep
[13:42:59] <dcaro>	 wget: unable to resolve host address 'mirrors.wikimedia.org'
[13:43:21] <dcaro>	 should the dns servers be pingable?
[13:43:23] <dcaro>	 https://www.irccloud.com/pastebin/Z1fHwEfX/
[13:48:36] <dcaro>	 topranks: ^?
[13:49:08] <dcaro>	 from other nodes it seems it should
[13:49:10] <dcaro>	 https://www.irccloud.com/pastebin/zeG64BcM/
[13:49:18] <dcaro>	 maybe vlans are off somehow?
[13:53:46] <dcaro>	 oh, no, something happened, now it does not ping to the gw anymore
[13:53:49] <dcaro>	 https://www.irccloud.com/pastebin/2wBSKlFO/
[13:54:50] <dcaro>	 and going back and reconfiguring the network on the install screen seemed to work :/
[13:54:56] * dcaro confused
[13:55:59] <dcaro>	 passed the debian mirron step now
[13:56:02] <dcaro>	 *mirror
[14:00:31] <topranks>	 perhaps the busybox dhcp client didn't set the dns 
[14:00:53] <dcaro>	 I got it from the resolv.conf file, it was set there :/
[14:44:51] <dcaro>	 reimage of cloudcephosd1006 finished \o/
[15:01:28] <taavi>	 dcaro: puppet versions fixed
[15:01:42] <dcaro>	 taavi: thanks!
[15:01:51] <taavi>	 now it's failing with 'Notice: /Stage[main]/Profile::Cloudceph::Osd/Interface::Ip[osd-cluster-ip]/Exec[ip addr add 192.168.4.6/24 dev ens3f1np1]/returns: Cannot find device "ens3f1np1"', which sounds like hiera file needing update for the new interfaces
[15:01:52] <dcaro>	 just downgrading the puppet client?
[15:02:02] <taavi>	 that, and then refreshing the certs for the puppet 5 ca
[15:02:07] <dcaro>	 taavi: yep, I have to fix that on hiera as the interfaces changed
[15:18:07] <topranks>	 dcaro: crazy, not sure what went wrong there with the reimage/dhcp 
[15:18:21] <topranks>	 I guess we'll see on the others if the same happens, or if it was just some one-off ghost in the machine 
[15:34:58] <dcaro>	 yep, it was pretty weird :/
[15:35:02] <dcaro>	 fyi. the ceph alerts is mi
[15:46:58] <dcaro>	 &me
[15:52:35] * arturo off
[16:20:08] <hashar>	 grafana.wmcloud.org only has 30 days of data nowadays hasn't it?
[16:21:43] <dcaro>	 looks like it (for cloudvps stats at least), I think we had set it up to limit by space, not date, but maybe we changed it
[16:22:40] <hashar>	 not a big deal
[16:22:50] <hashar>	 we just lost the ability to compare the state of things over a long trend ;]
[16:23:24] <taavi>	 yeah, retention there is 730h it seems
[16:23:46] <dcaro>	 it seems we have that yep
[16:23:46] <dcaro>	 --storage.tsdb.retention 730h 
[16:24:07] <taavi>	 iirc the prometheus option to do reliable disk usage-based retention is a relatively new thing, and the metricsinfra setup predates that
[16:24:12] <hashar>	 ah that explains it
[16:24:33] <dcaro>	 maybe I did it in cloudmetrics yep
[16:24:35] <hashar>	 I came asking cause we had a regression which I tracked to May 13th (more than 30 days)
[16:24:44] <hashar>	 and we thus don't have a baseline to compare against
[16:24:47] <hashar>	 too bad :]
[16:24:50] <hashar>	 but not a big issue
[16:25:07] <hashar>	 I guess 30 days is fine. Thank you!
[16:25:15] <taavi>	 also currently that 730h means about 70G of metrics, so could relatively easily bump that number to be higher (but not retroactively for already deleted bits of course :-)
[16:32:35] <andrewbogott>	 those emails/alerts about cloudvirt1063 are vriley moving it to a different rack -- no VMs there.
[16:37:51] <dcaro>	 ack
[16:38:00] <dcaro>	 I was just going to ask xd
[17:08:18] * dcaro off
[17:08:22] <dcaro>	 cya tomorrow
[17:37:45] <andrewbogott>	 everyone: I just rebuilt the codfw1dev bastion, so expect host key warnings next time you ssh there