[00:21:40] * bd808 off
[09:44:18] <dhinus>	 I have made a few improvements to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Openstack_upgrade and I'm now ready to proceed with the upgrade
[09:45:38] <dhinus>	 I think it makes sense to set the maintenance mode for Horizon, as recommended by the wiki
[09:46:12] <dhinus>	 taavi do you agree and would you also send an email to cloud-announce or elsewhere?
[09:47:37] <taavi>	 dhinus: yeah, a short notice that horizon will be unavailable and the apis will be unstable would not hurt
[10:40:13] <dhinus>	 million dollar question: is it worth disabling puppet on all/some cloud* hosts before merging this? https://gerrit.wikimedia.org/r/c/operations/puppet/+/978636
[10:40:46] <dhinus>	 I'm tempted to say "no" as most config files will only be read when a service is restarted, but that is a big guess
[10:43:00] <dhinus>	 Puppet might actually restart services because it finds a config change
[10:43:24] <taavi>	 dhinus: I would disable
[10:43:53] <dhinus>	 on everything (cloudcontrols, nets, virts)?
[10:44:01] <dhinus>	 I guess I can use cumin
[10:44:11] <taavi>	 yep
[10:44:25] <dhinus>	 sounds good, I'll also add the Cumin command to the wiki
[10:46:16] <dhinus>	 this seems to return what I need: sudo cumin 'P{C:openstack::serverpackages::zed::bookworm}'
[10:47:10] <dhinus>	 I could add the cluster to make it reusable for future upgrades on codfw
[11:14:12] <dhinus>	 puppet is disabled on all hosts to be upgraded, and I have merged the patch
[11:28:20] <dhinus>	 now upgrading cloudcontrol1007
[11:45:36] <dhinus>	 cloudcontrol1007 is done, now I've started cloudcontrol1006
[11:48:02] <dhinus>	 cloudcontrol1007 has an alert "HAProxyBackendUnavailable"
[11:53:45] <taavi>	 that seems to have cleared by itself?
[11:54:31] <dhinus>	 yep
[12:01:59] <dhinus>	 cloudcontrol1006 upgrade completed
[12:03:50] <dhinus>	 starting the upgrade of cloudcontrol1005
[12:21:21] <dhinus>	 cloudcontrol1005 completed
[12:23:27] <dhinus>	 I'll take a break for lunch, then continue with the other nodes
[13:40:30] <dhinus>	 I'm back, starting the upgrade on cloudnets, but first checking the network tests
[13:41:56] <dhinus>	 network tests are passing
[13:42:26] <dhinus>	 I forgot to set_maintenance on horizon before starting with the cloudcontrols :/ doing it now
[13:48:23] <dhinus>	 hmm the set_maintenance script doesn't seem to work
[13:48:38] <dhinus>	 "Hosts cloudweb[1003-1004].wikimedia.org now in maintenance mode."
[13:48:45] <dhinus>	 maybe horizon has moved to different hosts?
[13:49:24] <dhinus>	 the cookbook derives the hostnames dynamically from Cumin
[13:50:12] <taavi>	 no, it is running on cloudwebs
[13:50:35] <taavi>	 the most disruptive part (cloudcontrols) were already upgraded, so setting things to maintenance now seems unnecessary
[13:52:53] <dhinus>	 agreed
[13:53:09] <dhinus>	 we still need to figure out why the cookbook is not working, but we can do it later
[13:53:28] <taavi>	 i think it's not been adapted to the containerized horizon deploy
[13:53:36] <dhinus>	 ah yes that must be it!
[13:54:48] <dhinus>	 upgrading cloudnet1005, which is the standby host
[14:07:49] <dhinus>	 cloudnet1005 is upgraded, some alerts have not cleared yet but I think they will soon
[14:08:28] * andrewbogott here, reading backscroll
[14:09:27] <andrewbogott>	 Yeah, could be that maintenance mode doesn't work post-dockerization
[14:11:36] <dhinus>	 as soon as the cloudnet1005 alerts clear, I will upgrade cloudnet1006 which will cause a Neutron failover and according to the wiki a brief network outage
[14:12:27] <andrewbogott>	 yep! short enough that it shouldn't timeout connections
[14:14:06] <dhinus>	 I think the alerts will take a few more minutes to clear because of how they're designed (using min_over_time)
[14:14:14] <dhinus>	 the systemd units are up and running so I think it's safe to proceed
[14:16:34] <dhinus>	 started the upgrade on cloudnet1006
[14:17:29] <dhinus>	 andrewbogott: the wiki has no information on cloudrabbits, I assume they can be upgraded with the same cookbook?
[14:18:21] <andrewbogott>	 good question!  I don't think of them as needing to be in sync with openstack services since rabbitmq isn't an openstack project
[14:18:28] <andrewbogott>	 but if there are new package versions in the bpo...
[14:18:42] <dhinus>	 let me check
[14:40:17] <dhinus>	 maybe it was in bullseye?
[14:40:26] <dhinus>	 and we now fetch it from the official debian repo?
[14:40:48] <andrewbogott>	 that's definitely what /we/ do :)  It might be that zigo doesn't package it in the bpo because it's the same
[14:40:59] <andrewbogott>	 let's see if I can find him to ask
[14:42:33] <dhinus>	 I'm still confused by the 3 (leftover?) packages installed with bpo11: https://phabricator.wikimedia.org/P54014
[14:43:27] <andrewbogott>	 hm. harmless but weird
[14:43:55] <dhinus>	 yep, it would be nice to remove them, so we make it explicit that cloudrabbits don't need the bpo repo
[14:44:41] <dhinus>	 I find it useful to check the presence of the bpo repo on a host as an indication of what needs to be upgraded for a new openstack version
[14:45:20] <andrewbogott>	 despite what apt said, when I did 'apt install python3-cinderclient' it installed bpo12 version
[14:45:49] <andrewbogott>	 I suspect that sometimes we'll want the bpo present
[14:45:54] <andrewbogott>	 but let's see if zigo has a clear answer
[14:46:44] <andrewbogott>	 apt-get upgrade --dry-run shows a lot of things that it would like to upgrade
[14:47:25] <andrewbogott>	 nothing I care about especially. I wonder if the openstack client packages are installed there by accident, or as a side-effect of me trying to get the right apt repo present
[14:47:37] <andrewbogott>	 cloudvirts doing OK?
[14:47:46] <dhinus>	 I've done two with only a canary
[14:52:43] <dhinus>	 I think the "bpo11" instead of "bpo12" is just a glitch in the naming, for some reason the bpo packages in "bookworm-zed-backports" retained the "11", but should really have "12" in the name
[14:54:03] <dhinus>	 so the 3 bpo packages in cloudrabbits are indeed coming from the correct "bookworm-zed-backports", and can be updated to "bookworm-antelope-backports"
[14:54:35] <dhinus>	 andrewbogott do you mind doing the apt upgrades on all cloudrabbits nodes, until we find out if we can remove those packages?
[14:54:52] <andrewbogott>	 you think I should do a full 'apt-get upgrade'?
[14:55:01] <andrewbogott>	 seems harmless since it doesn't affect rabbit packages
[14:56:07] <dhinus>	 makes sense
[14:56:59] <dhinus>	 I'm now upgrading cloudvirts106[0-5] with a for loop that will do those in sequence
[14:58:12] <andrewbogott>	 dhinus: I'm starting a meeting but ping me if anything bad happens :)
[14:58:53] <dhinus>	 ok!
[15:32:59] <dhinus>	 cloudvirt106[0-7] have been upgraded. now doing cloudvirt105[0-9]
[15:42:23] <taavi>	 I need to go get something done during normal people hours, talk to you tomorrow
[16:06:02] <andrewbogott>	 dhinus: Zigo says "I stopped doing this, as it's not really needed, and it's annoying to upgrade rabbitmq or OVS between Debian releases."
[16:06:28] <andrewbogott>	 Which seems sensible to me! I'll see if we can remove that whole bit from those servers.
[16:10:19] <dhinus>	 great, thanks!
[17:08:45] <dhinus>	 quick link to see the status of upgraded vs to-upgrade cloudvirts: https://debmonitor.wikimedia.org/packages/nova-compute
[17:08:50] <dhinus>	 31 done, 17 to go
[17:09:16] <dhinus>	 I've added some quick checks I discussed with andrewbogott to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Openstack_upgrade#Upgrading_cloudvirt_nodes
[17:10:26] <andrewbogott>	 I get a 500 from that debmonitor link (well, and apparently all debmonitor links)
[17:15:51] <dhinus>	 hmm seems to work for me
[17:17:59] <andrewbogott>	 that's odd
[17:19:15] <andrewbogott>	 Rook: is quarry sufficiently delegated to volunteers that we should stop seeing/caring about alerts?
[17:38:08] <dhinus>	 andrewbogott: I've upgraded cloudvirtlocal*, and I'm doing cloudvirt-wdqs* -- I'll let you handle the remaining 9 regular cloudvirts (1031 to 1039)
[17:38:46] <dhinus>	 if debmonitor is not working you can also check "sudo cumin 'P{C:openstack::serverpackages::zed::bookworm}'"
[17:39:21] <andrewbogott>	 ok!
[17:41:21] <dhinus>	 shall we send an update to cloud-announce saying that the upgrade is completed?
[17:42:05] <andrewbogott>	 yep!
[17:42:27] <dhinus>	 do you want to send it after you complete all the cloudvirts?
[17:43:20] <andrewbogott>	 you can go ahead and send it now, the cloudvirt upgrades won't affect users at all
[17:44:57] <dhinus>	 ok!
[17:48:18] <bd808>	 If anybody has some time to help debug email sadness, JSherman is asking about T347512 in -cloud
[17:48:19] <stashbot>	 T347512: Some emails from The Wikipedia Library aren't being received as expected - https://phabricator.wikimedia.org/T347512
[17:49:20] <dhinus>	 andrewbogott: sent
[17:49:33] <andrewbogott>	 great!
[17:49:45] <dhinus>	 bd808: I might have some time tomorrow if nobody finds the answer before that
[17:53:50] <dhinus>	 clloudvirt-wdqs* are all upgraded, only cloudvirt[1035-1039] are left
[17:54:06] <dhinus>	 I'm logging off shortly
[17:56:04] <Rook>	 andrewbogott: yes*
[17:56:04] <Rook>	 *not really sure what the * is, but it feels like there is one
[17:56:35] <andrewbogott>	 ok! I'll continue to ignore for now and will maybe see about getting them off the board entirelh
[17:56:37] <andrewbogott>	 entirely
[17:56:47] <Rook>	 Seems reasonable
[18:19:28] * dhinus off
[18:41:41] * bd808 lunch
[18:55:25] <taavi>	 Rook: andrewbogott: I updated the config, quarry alerts should no longer be sent to us
[18:55:34] <andrewbogott>	 thanks!
[18:57:10] <Rook>	 👍
[20:28:29] <andrewbogott>	 taavi: the app creds for eqiad1 on tf-bastion.tf-infra-test.eqiad1.wikimedia.cloud have stopped working. Would you guess that they simply expired, or that the upgrade this morning somehow broke all app credentials everywhere?
[20:28:37] <andrewbogott>	 (I'm making new ones for that one use-case in the meantime...)
[20:29:00] <taavi>	 andrewbogott: I have no idea. I'm not aware of any application credential expiry mechanism
[20:29:08] <andrewbogott>	 hrm ok.
[20:29:11] <taavi>	 does it say 'expired' in an error or what?
[20:29:36] <andrewbogott>	 I'm not any deeper than just seeing the TF error "│ Error: Error creating OpenStack container infra client: Authentication failed"
[20:29:40] <andrewbogott>	 Investigating more now
[20:44:08] <andrewbogott>	 yep, I think they're just expired. You can definitely flag a set of credentials with an expiration date, and new ones work fine.
[21:45:26] <andrewbogott>	 I'm installing laptop updates so out for now.
[22:56:14] <bd808>	 The problem with the wpl project outbound emails in T347512 turns out to be the sender address they are using (noreply@wikipedialibrary.wmflabs.org) not having SPF and Google rejecting the messages as a result. Because the sender address itself is also bogus there wasn't anywhere for mx-out03 to send a bounce notification to.
[22:56:15] <stashbot>	 T347512: Some emails from The Wikipedia Library aren't being received as expected - https://phabricator.wikimedia.org/T347512
[23:29:22] <andrewbogott>	 zero team-wmcs alerts!
[23:44:18] <bd808>	 zarro boogs found? nice :)
[23:45:17] <bd808>	 https://en.wikipedia.org/wiki/Bugzilla#Zarro_Boogs for those lucky enough not to know the reference.