[06:49:31] greetings [07:46:19] morning! [07:47:27] oh, I see the k8s uprgade did not go smooth, looking [07:47:31] (toolsbeta) [08:47:46] morning [10:45:56] * dcaro lunch [11:23:07] bliviero: the bullseye spreadsheet you linked in the wmcs meeting etherpad is read-only for me, is that intentional? [12:11:39] dcaro: can you tell what's up with toolsbeta? I got it to a point where k8s reports that things are basically happy but 1) ingress is broken 2) I don't understand why there's no api server running on control-10 [12:12:19] andrewbogott: oh hey, not really, I was not sure what was the current status (got confused by the -13 control node and such), I can give it a look [12:12:54] so everything should be in place then rigth? using the 10/11/12 control nodes? [12:13:05] -13 was a desperate attempt to get quorum so I could move forward, it should now be out of service. [12:13:10] yep, that' sright [12:13:15] ack [12:13:29] -13 got me unstuck when I needed it :) [12:14:14] looking then control 10 [12:14:21] And integration tests were passing until the very end, I think things broke when I upgraded gateway nodes. [12:15:18] I'm pretty sure that the rest of the noise was due to an unworkable disruption policy (budget 0) for typha [12:15:29] andrewbogott: the point of a testing environment is that you can spend time debugging issues instead of rushing to unstuck things with hacks :-) [12:15:51] trust me, I did plenty of that :( [12:16:32] well, /etc/kubernetes/manifests/kube-apiserver.yaml does not exist on control-10 [12:16:45] iirc the uprgade process does mess up a bit with the yaml to update the image [12:16:56] or was kubadm directly? [12:17:08] it was the cookbook initially. [12:17:20] andrewbogott: do you have the output of the initial upgrade cookbook on -10? [12:17:27] I tried 'upgrade node' after the fact but I think it declined to do anything. [12:17:51] I would recommend playing with it if you want, but rebuilding a new control node after just to make sure [12:18:55] taavi: no, but I can tell you what it looked like: it got to the 'evicting static pods' step and hung (or appeared to, maybe I should've given it another 20 minutes) [12:19:58] Where does /etc/kubernetes/manifests/kube-apiserver.yaml come from? Is that something that's removed and then replaced by the cookbook? [12:20:07] managed by kubeadm [12:20:22] andrewbogott: 'evicting static pods' is a kubeadm step? or the cookbook? [12:20:49] I'm not sure, I assume kubeadm [12:22:14] we might still have the cookbook logs no? [12:22:23] did you run it in cloudcumin1001? [12:22:45] the cookbook logs will be there but don't contain actual command output iirc? [12:23:50] 🤦‍♂️ yep, though we might see if it was kubeadmn or not [12:25:00] dcaro: andrewbogott: unless you can think of a reason not to, i'm going to restore the apiserver manifest to -10 from the pre-upgrade backup kubeadm keeps in /etc/kubernetes/tmp, looking at an another node the version bump is the only change in the manifest so easy to do by hand [12:25:32] 👍 we can try doing a diff of the files, maybe others got broken/missing too [12:25:49] taavi: is that a manual copy of that file from a different control node, or some kubeadm command? [12:26:17] same node, under a tmp dir that kubadm created [12:26:29] andrewbogott: manual copy from a backup just before the upgrade was attempted [12:26:32] oh, that's handy [12:26:48] is that kubeadm creating it or is it us? [12:26:49] So -- if it's not a problem to leave toolsbeta broken for another day, I'd rather do the restore myself for practice. [12:27:11] What would I do after restoring the file to get the pod going? [12:27:12] dcaro: kubeadm feature I believe [12:27:38] andrewbogott: no problem for me, it's not broken enough to block deploys or testing other (non-so-infra) things [12:27:42] dcaro: seems like the version bump was the only change in the manifests https://phabricator.wikimedia.org/P96060 [12:27:44] as in not k8s itself [12:28:21] taavi: 👍 [12:28:38] just putting the (version upgraded) file in /etc/kubernetes/manifests is enough. I just did that to test that it works, but can revert if andrewbogott wants the practice [12:28:52] k8s is just fine with a single apiserver down, so that's not a major concern [12:30:20] And the pod just appeared as soon as the file was there? [12:30:37] yep [12:31:11] ok! [12:31:56] So... it's also the case that ingress is broken, right? That's the failure that's more worrying to me since there was no warning of problems from the cookbooks [12:32:14] what do you mean by the ingress being broken? [12:32:25] https://admin.beta.toolforge.org/ and others seem to load just fine [12:33:01] when I gave up last night basically all the integration tests relating to creating and testing a tool were failing. let me see if that's still happening... [12:33:28] for example... [12:33:30] https://www.irccloud.com/pastebin/8b0IgJ51/ [12:36:44] is that from now, or from last night? [12:36:59] that error suggests a timeout when trying to talk to jobs-api, which at least now works fine [12:37:46] hmm, I deployed a few clis today without issues [12:38:31] from last night [12:39:00] ok, so the failure was because the tests were trying to hit -10 specifically? That doesn't really explain the timing of when they started failing, but... [12:39:03] I'm rerunning now. [12:39:20] no, tests would not be hitting an api server specifically [12:39:38] I wouldn't have thought [12:39:41] was that before or after upgrading the worker nodes? [12:40:12] at the very end of my day, after everything was upgraded, nodes all showed 1.33.13 and all pods (except that one api server) showed as up [12:40:42] should've been the exact same state you were looking at 10 minutes ago [12:41:57] I'm re-running the tests but will probably have to go before they finish. [12:42:11] do you have any more idea of the time when that happened? trying to match it to the logs from that time [12:42:47] was it only that test? or various others as well? [12:45:32] I don't have a good timestamp. It would be less than 12 hours ago [12:45:42] And that run had just the one failure, the run before it had 2. [12:45:55] well, and 96 skips [12:47:16] looks like the other failure was... [12:47:19] https://www.irccloud.com/pastebin/ahKmUb4P/ [12:49:08] yep, the tests once one of them fails, create a skip file so no more tests run [12:49:31] Sorry, I'm leaving the house in 10 and all over the place. I have a follow-up question about getting the api running... if the issue was a missing file, is that file literally what tells the scheduler that the api server should be running? 'missing file' sounds to me like it should show up as crashloopbackup rather than as the pod just not appearing in 'show pods' at all. [12:49:53] yep, that file is what starts the api pods [12:50:22] it's a static pod https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/, if the file does not exist, there's no pod to start [12:50:54] ok! [12:51:14] I'm surprised by that but will read up on static pods [12:51:14] that's to avoid depending on k8s api itself [12:51:25] (to start the k8s api xd) [12:51:28] integration tests are all passing now, dammit [12:52:00] ok, that makes sense. I spent some time yesterday googling "api server chicken egg problem" but that was not the right way to ask this question which you have just answered [12:57:40] thank you for the education dcaro and taavi! I'm still upset that things were failing last night and seem to have spontaneously fixed themselves but I will read more logs when I'm free. [12:58:24] yw! [12:59:04] Oh, one action item for someone: the upgrade doc has a process in it for ingress nodes (which we no longer have) but no mention of gateway nodes (which we do have). If it's as simple as 'just use the worker upgrade script to upgrade gateway nodes at the end' I can make that edit myself. But that is of course the part that I thought had gone terribly last night. [12:59:04] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes [12:59:08] now have to run :/ [13:00:16] it is almost that, except that until T426948 happens one needs to be rather slow and careful with it [13:00:16] T426948: wmcs.toolforge.k8s.reboot needs to be slower with Gateway nodes - https://phabricator.wikimedia.org/T426948 [13:52:15] taavi: the original spreadsheet is owned by Komla - I opened up editor access to it for more team members assuming he's ok with that [14:05:01] bliviero: it's totally fine! Movement comm says it contains private info so it should remain WMF-only access. [14:16:13] komla: what exactly there is private information? migration details are public on https://os-deprecation.toolforge.org/, as are the related phabricator tasks [14:19:25] taavi: I'm not entirely sure myself, but Quiddity mentioned that. [14:45:56] taavi: how brief is the blip in T426948? Too brief to make my tests fail I'm guessing... [14:45:58] T426948: wmcs.toolforge.k8s.reboot needs to be slower with Gateway nodes - https://phabricator.wikimedia.org/T426948 [14:46:48] andrewbogott: in the order of a minute or two, and it would not cause the 504 you pasted earlier [14:46:54] ok [14:47:38] a normal worker node being rebooted at the exact moment that test were running would cause it, but that's true of any service really [14:49:46] * andrewbogott nods [15:02:37] taavi: the spreadsheet that Komla is maintaining has tools + owners - and although the list of owners is likely obtainable, perhaps the concern is that it's not ideal to make such a compendium public? [15:03:32] komla: oh I might've misunderstood. From memory, maybe it was just the annotations that I assumed were private? Your call! [16:19:35] * dhinus off [17:00:52] * dcaro off [17:00:55] cya on monda! [17:00:58] *monday xd [22:16:01] and my funky finding of the evening is the recent cloudvirt machines do not use intel_pstate power/scaling/cpu freq governor but intel_cpufreq schedulutil [22:16:16] why older ones have intel_pstate powersave [22:18:20] context is I found out the cloudvirt 1077 - 1080 (Xeon Gold 6530) perform the CiviCRM jobs slower (25 minutes) than other hosts (15 minutes). And the uninformed non-CPU engineer I am tend to suspect the power management could be a cause ;) [22:19:08] the task is https://phabricator.wikimedia.org/T434024#12213140 (the heatmap of CPUs frequencies is probably misleading cause the 6530 host was mostly idle while the other host had load) [22:19:26] I'll poke about it again next week. I just wanted to brain dump :) [22:19:59] the puzzle is why those new hosts apparently underperform [22:29:21] hashar: are the CPUs taxed significantly during these CiviCRM jobs? Or is that workload mostly I/O or memory bound? [22:32:33] and can you provide names of other hosts where you are not seeing this problem [22:33:02] oh i see it in the phab ticket, nvm [22:35:56] h.ashar has diagnosed more hardware/software perf problems in the OpenStack deployment over the last 12 years or so than all of WMCS SREs combined. CI is a system stress factory, and he has the focus to just keep digging. [22:36:16] ^ [22:36:17] :) [22:37:18] bliviero: I haven't looked at the CiviCRM payload. It is on my list to have a look at what it runs, but I assume it is CPU intensive, or at least there is a report of slow tests showing that the tests are indeed taking 40/50% longer which tends to lean toward the CPU is too slow [22:37:48] it is going to take a while to figure out for sure! [22:44:22] * hashar sleeps