[00:14:59] <bd808>	 andrewbogott: they are self-serve somewhere in your Phab settings. I’m away from my laptop or I’d give you better directions. 
[00:15:07] <bd808>	 Using strikerbot’s is fine too
[08:04:29] <arturo>	 I'm reading the g4 flavors announcement, and suddenly I had a question
[08:04:59] <arturo>	 what if a bunch of folks decide to migrate and we don't have enough HVs migteated yet to accommodate all the new VMs
[08:05:08] <arturo>	 migrated*
[08:30:20] <taavi>	 arturo: you mean migrate from buster to newer OSes?
[08:32:14] <taavi>	 we have 3 ~empty HVs on OVS atm
[08:51:57] <arturo>	 ok
[08:52:06] <arturo>	 I meant migrate to OVS
[08:53:06] <taavi>	 migration is not self-service
[08:55:49] <arturo>	 ok -- re-reading now: only for new VMs
[08:56:17] <arturo>	 apparently I read only a fraction of the email :-)
[08:56:24] <arturo>	 is super clear now
[10:13:53] <arturo>	 am I reading right the timestamps of these two log entries?
[10:13:59] <arturo>	 Jun 12 17:03:51 tools-k8s-control-7 kubelet[760]: E0612 17:03:51.972023     760 kubelet.go:2427] "Error getting node" err="node \"tools-k8s-control-7\" not found"
[10:13:59] <arturo>	 Jun 12 13:45:31 tools-k8s-control-7 kubelet[495]: I0612 13:45:31.401570     495 reconciler.go:352] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-proxy\" 
[10:14:24] <arturo>	 the second one has an earlier timestamp, but is listed later in the log
[10:14:56] <arturo>	 oh, wow, the kubelet log has plenty of this
[10:15:05] <arturo>	 I imagine they could be cause by the overload
[10:26:12] <blancadesal>	 super-quick review here would be much appreciated: https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/40
[10:27:26] <arturo>	 👀
[10:28:45] <arturo>	 blancadesal: LGTM
[10:30:13] <blancadesal>	 arturo: thanks!
[12:07:23] <taavi>	 [1/4, retrying in 3.00s] Server is in unexpected status: Server status is 'VERIFY_RESIZE', not in any of S, H, U, T, O, F, F
[12:07:25] <taavi>	 bah
[12:07:33] <dhinus>	 FYI, clouddb1018 failed to reboot for some hardware issues, I filed T367499
[12:07:33] <stashbot>	 T367499: hw troubleshooting: server fails to reboot for clouddb1018.eqiad.wmnet - https://phabricator.wikimedia.org/T367499
[12:24:25] <taavi>	 looking for code review: https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1043744
[12:47:19] <taavi>	 andrewbogott: ok to re-enable Puppet on cloudcontrol2004/6-dev?
[12:50:04] <andrewbogott>	 Yes please
[12:57:42] <taavi>	 done
[13:23:18] <taavi>	 reimaging cloudvirt1034 to OVS
[13:37:25] <taavi>	 andrewbogott: not sure what you did yesterday but now the scheduled os-deprecation run fails with 'ModuleNotFoundError: No module named 'arrow''
[13:38:27] <andrewbogott>	 Yeah :/ I'll be back at my keyboard in 30 or so
[14:32:18] <andrewbogott>	 taavi: fixed now (I just needed to rebuild the venv)
[14:54:00] <andrewbogott>	 taavi: my plan was to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1043203 and then use cumin to touch (or remove) *zed* cloud-wide so that the update fires
[14:54:38] <andrewbogott>	 I was worried that adding that flag to every 'apt get update' everywhere would add some security risks for other repos
[14:54:38] <taavi>	 why are we provisioning files for zed in the first place?
[14:54:48] <andrewbogott>	 zed is the latest release packaged for bullseye
[14:55:03] <taavi>	 hmmm I see
[14:55:12] <taavi>	 and we install osbpo by default on all the VMs
[14:55:13] <taavi>	 ?
[14:55:16] <andrewbogott>	 We could remove everything and fall back on the debian-packaged clients but they're even more out of date (V, I think?)
[14:55:48] <andrewbogott>	 Yes, at the moment we do. Because we need those libraries for some monitoring things iirc
[14:57:14] <taavi>	 hmh. would be great if we didn't do that, but i don't think we can change that retroactively on vms that already have them (so before trixie in practice)
[14:58:07] <taavi>	 I don't think we need to patch apt::repository if openstack::clientpackages::vms::bobcat::bullseye has its own apt-update that runs first?
[14:59:37] <andrewbogott>	 I tried that already, it doesn't fire at the right time
[14:59:48] <andrewbogott>	 dhinus: suggests that we try https://gerrit.wikimedia.org/r/c/operations/puppet/+/1043806 first
[14:59:59] <andrewbogott>	 Which I think won't solve the issue but is correct in any case
[15:00:09] <taavi>	 we need that anyway, but it won't solve the issue
[15:02:57] <taavi>	 ok, +1'd both
[15:03:49] <andrewbogott>	 thx taavi 
[15:04:11] <andrewbogott>	 now I just have to remember the cumin syntax for all VMs
[15:07:57] <andrewbogott>	 grrrrrrr
[15:08:13] <dhinus>	 andrewbogott: I think your patch got lost and must be reapplied to cloudcumin
[15:08:21] <dhinus>	 I added some details at T346453
[15:08:21] <stashbot>	 T346453: [cumin] [openstack] Openstack backend fails when project is not set - https://phabricator.wikimedia.org/T346453
[15:09:14] <andrewbogott>	 sure would be nice if that got fixed in upstream cumin, huh?
[15:09:29] <dhinus>	 it's still in my radar but I already lost too much time on it :)
[15:10:00] <andrewbogott>	 I can write another six alternative solutions but I'm sure they'll just be ignored too
[15:10:06] * andrewbogott stops complaining and hotfixes again
[15:16:55] <andrewbogott>	 dhinus: actually my hack is still there, so something else interesting is happening :/
[15:17:02] <dhinus>	 hmmm interesting
[15:18:42] <dhinus>	 did something change in bobcat maybe? I don't think I used cumin after the bobcat upgrade
[15:18:49] <andrewbogott>	 likely project name/id confusion
[15:18:54] <dhinus>	 :(
[15:20:24] <dhinus>	 in /var/log/cumin/cumin.log there are some details
[15:23:16] <andrewbogott>	 yeah, I'm not sure what's happening yet. We also have a new domain (toolsbeta) so that could be part of it
[15:26:40] <andrewbogott>	 dhinus: fixed; novaobserver didn't have permission to see the new domain
[15:27:02] <dhinus>	 ah that makes sense!
[15:27:21] <dhinus>	 I think it's looping and trying to get a new token for each domain
[15:28:00] <andrewbogott>	 I think it needs to
[15:28:53] <dhinus>	 I think you can potentially create a globally-scoped token, but then maybe nova won't accept it? I did kind of understand the scoping at some point, but I forgot :)
[15:29:13] <dhinus>	 if it works, I think it's good enough for now
[15:29:38] <andrewbogott>	 globally scoped token would work for user with that access but I think we were trying to avoid assuming weird user setup
[15:30:11] <dhinus>	 I see what you mean yeah
[15:30:31] <dhinus>	 there are potentially infinite combinations of domains/users/RBAC rules :)
[15:31:54] <andrewbogott>	 yep
[17:33:29] <andrewbogott>	 taavi: Is there anything I can do to help you wrap up your day? And is there anything in particular I shouldn't touch while we're mid-ovs-migration?
[17:35:06] <taavi>	 andrewbogott: the main thing you should be careful with at this point is draining cloudvirts still on linuxbridge. I've so far only ran the flavor cache fixing script just before draining hypervisors since I'm still a bit scared by it just poking the database
[17:37:32] <taavi>	 the other thing in my mind at this exact moment is fixing the flavors for a) the canary VMs on OVS cloudvirts and b) for the one toolsbeta node that was accidentally migrated but does not have a matching g4 flavor yet
[17:39:33] <andrewbogott>	 taavi: for canaries it's easy enough to just delete them all and rebuild
[17:39:37] <andrewbogott>	 after https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1043149 is merged
[17:40:06] <andrewbogott>	 & I will definitely not drain or rebuild any cloudvirts over the weekend
[18:03:59] <andrewbogott>	 taavi or dhinus: how does this look to you?  https://etherpad.wikimedia.org/p/busterreminder
[18:22:34] <dhinus>	 andrewbogott: looks good, I would s/those that are using/those that are still needed using/, because on first read I didn't parse that sentence 
[18:24:08] <andrewbogott>	 'Please take some time to delete VMs that are no longer needed, and rebuild those that are still needed with a more modern release, ideally Debian Bookworm.'
[18:26:45] <andrewbogott>	 ok, sent. thanks!
[21:59:31] <dhinus>	 andrewbogott: I saw you deleted the clouddb-services project, I believe it hosted the DNS zone for db.svc.wikimedia.cloud
[22:01:35] <dhinus>	 I still see the zone in "openstack zone list --all-projects", and the project is listed as "clouddb-services", so maybe it's a zombie project now? :P
[22:08:09] <dhinus>	 left a comment in T365975
[22:08:10] <stashbot>	 T365975: [cloud-vps] migrate DNS zones away from deprecated clouddb-services project - https://phabricator.wikimedia.org/T365975
[22:08:24] * dhinus off
[22:54:45] <andrewbogott>	 argh
[22:55:52] <andrewbogott>	 dhinus: do you know what IPs those entries should point to?
[22:56:35] <dhinus>	 hmmm vaguely :P
[22:57:12] <dhinus>	 I mean I think we can find them in a /reasonable/ time
[22:57:46] <dhinus>	 I can still see the zones with "wmcs-openstack zone list --all-projects"
[22:57:56] <dhinus>	 so maybe you can still dump the records?
[23:01:16] <dhinus>	 yes I can dump them, I will paste them to the phab task
[23:07:32] <andrewbogott>	 actually I think it's fine, I'm going to transfer them to cloudinfra
[23:07:40] <andrewbogott>	 can you stand by for 5 minutes and then confirm that it's working?
[23:08:46] <dhinus>	 sure
[23:09:03] <dhinus>	 I have a full dump in the meantime on my computer in the meantime
[23:09:19] * dhinus cannot type at 1 am :P
[23:09:41] <andrewbogott>	 ok, those domains are now owned by cloudinfra and look to still be populated as before.
[23:09:45] <andrewbogott>	 Look that way to you too?
[23:09:53] <dhinus>	 checking
[23:11:00] <dhinus>	 yep those look fine, but I found 4 more zones :P
[23:11:03] <dhinus>	 "openstack zone list --all-projects |grep clouddb-services"
[23:12:24] <andrewbogott>	 those are just the standard set of project-associated domains... do  you think there are still refs to them other places?
[23:12:30] <dhinus>	 hmm maybe not
[23:12:40] <dhinus>	 the only ones I know about are the 3 ones you migrated
[23:12:59] <dhinus>	 maybe "svc.clouddb-services.eqiad1.wikimedia.cloud"?
[23:13:45] <dhinus>	 there are only ns records though in that one
[23:13:50] <andrewbogott>	 looks empty to me, just the SOA
[23:14:01] <dhinus>	 yep
[23:14:05] <dhinus>	 I think we're good
[23:14:15] <andrewbogott>	 So I'm going to delete those 4
[23:14:19] <dhinus>	 sgtm
[23:14:27] <andrewbogott>	 And continue to think about making a project deletion cookbook because there are always too many parts
[23:14:34] <andrewbogott>	 Sorry for whatever temporary panic I caused you!
[23:14:41] <dhinus>	 np
[23:15:41] <dhinus>	 I randomly spotted your change
[23:15:59] <dhinus>	 and thought it was better to double check now rather than risking an outage later :P
[23:16:36] <andrewbogott>	 glad you noticed
[23:23:57] <dhinus>	 all looks good to me
[23:25:34] <dhinus>	 thanks for the fix, out of curiosity, do you think if you didn't move the zones they would keep on working indefinitely, and were just invisible in horizon?
[23:35:54] <dhinus>	 anyways, not an important q :) I'm off to sleep
[23:55:33] <andrewbogott>	 I don't know, probably they would live on for a while but might get lost during upgrades or similar