[06:28:36] morning [06:28:45] is anyone looking at the puppet failures on cloudbackup1002-dev.eqiad.wmnet? [06:29:14] `Could not find class ::openstack::serverpackages::antelope::bullseye` [07:11:19] morning [07:11:23] I can take a look [07:12:50] I see no alerts [07:13:50] but I see the error, looking [07:17:47] it's just not there, we don't have a bullseye version of the serverpackages for antelope [07:18:57] the repo is missing also on https://mirrors.wikimedia.org/osbpo/dists/ [07:26:47] hmm, that is just a mirror from upstream, so maybe the packages are not built upstream yet [07:28:13] it synced last time yesterday morning, so seems fresh [07:53:00] maybe dhinus or andrewbogot.t have started to work on it for the upgrade? [08:14:05] oh, it seems toolsbeta-harbor is failing to connect to it's db [08:14:26] blancadesal: ^ are you doing anything there? [08:15:08] 2023-10-18T08:14:10Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /service/notifications/tasks/47 request in transaction failed: failed to connect to `host=ttg4ncgzifw.svc.trove.eqiad1.wikimedia.cloud user=harbor database=harbor`: dial error (dial tcp 172.16.5.95:5432: connect: connection refused)"}]} [08:19:02] hmm.... the /var/run/postgresql directory was owned by root, making the database container fail to start as it runs as `database`, chowning the directory allowed it to start :/ [08:19:22] dcaro: nope, nothing beyond the upgrade-harbor MRs [08:19:45] I suspect that might happen every time the VM is restarted (was one of the affected VMs from the cloudvirt1051 going down yesterday, so it got restarted somewhere else) [08:19:58] the VM being the database VM, not harbor [08:39:54] o/ let me check if we tried to upgrade that host and when [08:46:12] looking at the IRC log, I have reimaged cloudbackup1001-dev on Sep 18th, but I probably forgot to reimage 1002, which is still on bullseye. antelope is only available in bookworm, so that's why Puppet is failing. [08:46:20] reimaging 1002 to bookworm should fix it [08:46:42] ack :) [08:47:17] quick review to fix tests on builds-cli (not sure how I got a broken commit merged) https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/20 [08:47:55] the reimage of cloudbackup1001-dev is also tracked here https://phabricator.wikimedia.org/T345810#9174746 [08:48:04] I will start the reimage of cloudbackup1002-dev now [09:01:36] thanks, no rush [09:21:08] could I get a quick review of https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/18? It's not a big change and I'd like to deploy it together with the previous MR [09:35:43] do you want me to test it locally also? [09:40:12] nm, done [09:47:27] dcaro: sorry, distracted with something else. Thanks! [09:55:47] quick review also https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/22, making releasing easier [10:05:22] lgtm [10:30:09] * dcaro lunch [10:50:30] prepping the toolforge k8s upgrade (starting in 10 minutes) here: https://etherpad.wikimedia.org/p/toolforge-k8s-upgrade-1.23 [11:01:25] ok I'm starting [11:02:27] alertmanager downtime set [11:03:33] running prepare_upgrade cookbook [11:04:52] filed T349195 [11:04:52] T349195: cloud/instance-puppet.git updater is broken - https://phabricator.wikimedia.org/T349195 [11:06:36] running upgrade worker cookbook for control-4 [11:08:05] cluster upgrade starting [11:09:53] upgrading docker and kubelet on control-4 [11:10:45] rebooting instance, again some permission errors with locks [11:12:13] taavi: ask if you need assistance with anything [11:15:29] sure [11:15:46] the pods took a moment to restart after rebooting control-4 but seem ok now [11:15:50] moving on to control-5 [11:18:38] control-5 updating control plane components [11:20:00] upgrading docker and kubelet on control-5 [11:21:46] oh, mirrors.wikimedia.org seems down [11:22:31] that's why the apt update was taking a while? [11:23:11] maybe yes [11:25:34] it seems to be back [11:25:48] yep [11:25:52] continuing to control-6 [11:26:24] the k8s packages come from apt.wm.o not mirrors.wm.o, so apt update being slow is the main effect on the upgrade. still not ideal [11:27:06] filed T349197 too [11:27:07] T349197: Remove TTLAfterFinished from config - https://phabricator.wikimedia.org/T349197 [11:29:45] control plane upgrade complete [11:29:55] \e/ [11:30:05] verified that new pods are still getting scheduled and start fine [11:30:09] moving to first worker node [11:35:57] first 3 nodes upgraded, looks good [11:36:02] I'm starting the mass upgrade scripts [11:38:32] (by which I mean `cat ~taavi/nodes2.txt | xargs -L1 cookbook wmcs.toolforge.k8s.worker.upgrade --cluster-name tools --src-version 1.22.17 --dst-version 1.23.17 --hostname` on three different tmux tabs since the cookbook only has support for a single node at this point) [12:05:53] all nodes upgraded [12:06:11] the new cookbook seems to be much more aggressive about draining nodes compared to the old script, which is why it was this fast [12:11:55] dcaro: where was the third-party k8s component version information moved from https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Components? I don't see it on https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy [12:12:50] taavi: which component specifically? [12:13:18] I need an overview table which details which versions of various third-party components we use and what k8s versions those support [12:14:34] taavi: for that you have the components in toolforge-deploy, if you want the metrics related ones, they are defined in the helmfile for it [12:14:42] or the calico ones, are in the calico helmfile [12:15:02] there's no k8s versions support specified there though [12:15:56] yes, and having that data in a single place is required for k8s version upgrade planning [12:16:32] feel free to add it to the toolforge-deploy docs [12:17:19] that will have to be manually kept up to date though, as (afaik) there's no easy way to extract whit k8s versions each component supports [12:18:29] I would suggest to keep that info close to the version specification of each component, and extract it somehow for human consumption, so when someone changes the version of that component, will not miss updating that info [12:21:09] there's no single place with the versions of all of the components in toolforge-deploy, so I just re-added it to the wiki page [12:22:47] git grep chartVersion [12:27:35] includes a bunch of info that I don't need (in-house components), and the values files don't have all of the info I need [12:33:23] feel free to add it then if it's missing [12:34:28] hmm, it's a pity that not all implement https://helm.sh/docs/topics/charts/#the-kubeversion-field [12:34:55] cloudbackup1002-dev is reimaged and looking fine [12:36:02] I think we can generate that info parsing the helm chart's kubeVersion string if there, or adding it manually if not [12:37:05] the kubeVersion of the ones that have it does not seem good either though [12:37:19] dhinus: 🎉 [12:39:37] taavi: adding it to toolforge-deploy will allow also to check it at deploy time, and prevent or warn when you are trying to deploy a component that does not support the target k8s [12:39:50] and keep the single source of truth [12:44:59] if we had that data for everything, sure. but right now we don't and I don't have the time to fix it completely, so I will continue to use the manually maintained but complete and very useful table [12:45:29] please move it to the docs at least [13:40:33] dcaro: do you know which postgres version we are currently using for harbor? [13:46:36] nm, it's 12.7 [13:48:12] I'm going to miss the meeting today but I have 45 minutes or so before chaos resumes. Please ping if there's anything I can help with/unblock. [13:50:53] andrewbogott: can you the canary for cloudvirt1051 back to 1051? I think I moved it to 1058 when evacuating all other VMs from it [13:51:15] sure [13:52:01] is 1051 back and healthy again? [13:52:56] what's the process for starting a canary VM btw? I could not find anything in wikitech [13:53:59] there's a cookbook which should go through and ensure that everything is as it should be [13:54:15] it's back up at least. I have not tested running anything on it yet, and the cause for the lockup is a mystery [13:54:19] that or create by hand on the cli [13:54:25] taavi: ok [13:56:50] taavi: in theory the move should look like this: [13:56:52] openstack server migrate --live --host cloudvirt1051 0f6b34aa-cef3-4c06-977d-2dc868dcf16d --os-compute-api-version 2.30 [13:57:01] but that's not working for weird reasons, I'm investigating [13:58:32] dcaro: fyi i had a look through the incident re isc-dhc-client. i can't think of a way we could really detect this at the puppet level. i guess the advice is dont be explicit and use ensure_packages($package) in every class that needs it. [13:58:44] ill keep mulling it over but nothing comes to mind right now [14:07:37] * dcaro paged [14:07:58] cloudvirte1058 nova proc, looking [14:09:26] that was me, sorry [14:09:32] already resolved, I just restarted the service [14:09:37] and got unlucky with the check [14:09:45] dcaro: ^ [14:10:03] ack [14:10:24] jbond: thanks [14:10:43] Nova is saying "Compute host cloudvirt1058 could not be found" when I try to move hosts off of 1058. Seems to be unique to that server. [14:11:00] Of course nova also reports that that nova is up on that host, and notices when it's down. [14:12:34] you mean "to move VMs" right? [14:13:05] we messed up with the database yesterday when moving VMs out of 1051, maybe something got out of place? [14:14:34] yes, moving vms [14:14:44] could be although the complaint is about 1058, not about 1051 [14:16:14] yes, but the instances were moved to 1058 [14:16:33] ah, ok [14:16:38] so maybe we forgot to update some entry somewhere, and openstack is unable to link it [14:16:38] which db? i can look [14:16:58] there's some info in the ticket https://phabricator.wikimedia.org/T349109#9258647 [14:17:18] the db was nova_eqiad1 [14:17:43] and neutron [14:17:50] cinder seemed to get updated by itself [14:19:06] taavi: ^ fyi [14:29:58] definitely possible the db is not in sync somewhere, I was following some clearly outdated upstream docs in a state of panic [14:32:40] Running "update instances set node='cloudvirt1058.eqiad.wmnet' where host='cloudvirt1058';" which may help or hurt [14:32:44] sorry about the panic :( [14:33:52] that seems to have helped. [14:34:02] Seems like those two fields are redundant, or at least redundant for our purposes [14:36:39] and now the canary is back where it belongs [14:37:56] btw taavi did you try 'openstack server evacuate'? In theory that's the command to move a server off of a failed hypervisor [14:38:16] no, I had no clue that even exists [14:38:19] I can add that to the docs if you link me to the runbook you were following [14:38:40] ok :) I think the docs say "evacuate the VMs" without specifying that evacuate is a literal thing [14:39:15] there was no runbook except the very generic https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown [14:39:56] oh yeah, I guess it's hard to link to a specific runbook from a hostdown alert... [14:40:57] yesterday you quoted "restore the server or evacuate manually the VMs on it" I was thinking I'd update the string wherever it is [14:42:21] that's what the prometheus alert says https://gerrit.wikimedia.org/g/operations/alerts/+/1f77dd6f684fb5211736172f70d006c627d15b02/team-wmcs/general_nodes_down.yaml#37 [14:47:42] oh, so I really can add a custom runbook. Let's see if I can do that before I have to go [14:48:17] thx taavi (and also thanks for dealing with the outage while I was dealing with other things) [14:49:40] hm, no such luck, gotta go in a minute. I'll work on the runbook later [15:48:10] anyone up for reviewing https://gerrit.wikimedia.org/r/c/operations/puppet/+/966871? I have no clue why it worked before, but it certainly does not work anymore [15:52:16] dcaro: dhinus: sorry I forgot to ask this during the meeting, will you let brett know we selected our victims for the incident review ritual? or should I? [15:52:27] taavi: done :) [15:52:37] I asked him to add the team to the invite too [15:52:39] awesome, thanks [15:52:58] ah I just asked lmata as well :) [15:53:03] in the -sre channel [15:58:13] added build and envvars services to the toolforge sidebar https://wikitech.wikimedia.org/w/index.php?title=Template%3AToolforge_nav&diff=2121081&oldid=2116117 [16:01:37] thanks! [16:02:49] I'll also rewrite the leads for both of the articles, to make them bit less technical for those who don't care about toolforge internals [16:05:16] I wonder if we should list buildservice and envvars also in https://wikitech.wikimedia.org/wiki/Help:Toolforge#Main_features_of_Toolforge [16:13:46] taavi: created https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/119 with the kubernetes compatibility info, nothing automated yet though [16:14:40] I added it to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Quickstart#Build_and_host_your_first_tool [16:14:43] (build service) [17:11:50] * dcaro off