[07:46:20] morning! [11:22:57] * dcaro lunch [15:08:56] andrewbogott: are we doing the scratch nfs flip? [15:09:20] yes. I'm just running an epic cumin command to get the baseline before I break things... [15:20:12] cookbook is done, now waiting for clients to refresh dns and cope with the change... [15:25:10] hm, in my tests the VMs had recovered by now. But I'm seeing lots of hangs... [15:26:34] maybe I didn't account for active file use, but surely not all 100+ VMs have that happening [15:27:37] taavi: do you have an opinion about whether we should go ahead and reboot tools-nfs workers, or give them more time to recover? [15:28:25] andrewbogott: might as well get the cookbook running already if that's what we have to do, it'll take a while anyway [15:28:33] :( ok [15:29:03] that's wmcs.toolforge.k8s.reboot_stuck_workers ? [15:29:30] just wmcs.toolforge.k8s.reboot with --all-nfs-workers [15:29:45] i'm fairly sure stuck_workers was for a problem that no longer exists [15:29:52] ok [15:30:03] have you seen the new mount work on a single VM so far? [15:30:46] yes, a few [15:30:54] I'm rebooting tools-k8s-worker-nfs-10 by hand to make sure that a reboot is sufficient [15:33:36] your suspicion is right though, something is wrong on the new server... [15:33:41] scratch-2 doesn't have a working exports file [15:33:47] Aug 03 15:33:40 scratch-2 nfs-exportd[227980]: 2026-08-03 15:33:40,936 WARNING Retrying mwopenstackclients.Clients.novaclient in 11.807941882189265 seconds as it raised NotFound: Could not find project: wikiqlever. (HTTP 404) (Request-ID: req-8c0571f9-830f-4929-8c02-b37656017e25). [15:34:30] yep, looking at that now [15:34:46] found the issue, manually hacking around it for now [15:35:29] andrewbogott: try now? [15:35:42] yep, helped [15:35:47] was it the qlever thing? [15:36:08] yeah, it could not get an exports file out due to that [15:36:16] it's the project name/id difference hitting us [15:37:11] so https://gerrit.wikimedia.org/r/c/operations/puppet/+/1320181 [15:37:39] wait a second [15:37:53] i need to check how it's keying the data on the client side [15:37:55] sure [15:38:00] ` Warning FailedMount 93s (x6 over 11m) kubelet MountVolume.SetUp failed for volume "scratch" : hostPath type check failed: /data/scratch is not a directory` [15:38:02] from toolforge posd [15:38:04] *pods [15:38:31] is that on -10? [15:38:42] tools-k8s-worker-nfs-54 [15:38:50] I'm worried that even with that exportd fix we'll still need to reboot... [15:39:11] I'm inclined to start the reboot script now, you agree? [15:39:24] yeah, client accesses that map as `$wmcs_project`, so it'll need a bit more involved fix. on it [15:39:26] andrewbogott: go for it [15:39:57] ack, let me know if you need anything, for now just hanging around [15:40:08] ok, it's running [15:40:17] (assuming you have a working single test VM already) [15:40:20] taavi: thank you! [15:40:39] dcaro: is this what's affecting the bastion login that was reported in -cloud channel? [15:40:46] Also, why did things work with the old scratch server? Or have we really not built any new tools workers since the qlever change? [15:41:02] bliviero: very likely. [15:41:17] bliviero: I think so yes, it's not the bastion login but starting any pod (webservice shell starts a pod and execs into it) [15:41:44] (*any pod that uses nfs/scratch) [15:46:44] I have a list of hosts that mount that scratch volume (excluding tools and toolsbeta worker nodes) [15:47:00] We could just reboot... all of them (excepting tools things which probably need delicate handling) [15:47:01] https://etherpad.wikimedia.org/p/nfsscratch [15:47:37] Unfortunately I think it requires a hard reboot because the shutdown locks up on nfs client things. So we can't really reboot things with cumin... [15:48:11] should I declare an incident? [15:48:22] yes [15:48:46] ack [15:48:49] on it [15:48:50] * dcaro ic [15:48:51] thoughts about starting reboots for that list? In theory those hosts aren't actually broken, only things that access scratch are [15:48:56] ty dcaro [15:49:29] all nfs workers seem to get stuck (did not check all, just a few), and regular pods don't start [15:50:09] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1320183 [15:50:20] why are there NFS hosts that have NFS mounted? [15:51:09] andrewbogott: `umount -l /mnt/nfs/secondary-scratch && mount -a` seems to be enough [15:51:26] really? I've been getting busy warnings when I try that. [15:51:37] but maybe that timed out... [15:51:53] just worked on tools-mail-4 at least [15:51:56] let me try it somewhere else [15:51:58] andrewbogott: which hosts did you run into that problem on? maybe taavi can look at that [15:52:02] yeah, and worked on the video host [15:52:11] I'm going to run that on all the hosts in my etherpad list [15:52:40] worked fine on a random k8s nfs worker too [15:53:22] your etherpad includes hosts without any nfs mounts as well, for example dumps-nfs-2.dumps.eqiad1.wikimedia.cloud [15:54:16] * andrewbogott fighting the F syntax [15:54:24] sudo cumin F{"/home/andrew/hostlistscratch.txt"} [15:54:40] taavi: that is the list of places where 'ls /data/scratch/' succeeded [15:54:48] prior to the switchover [15:55:43] huh, did the hostlist cumin thing regress? [15:55:57] i'm running the command on all tools k8s nfs workers [15:55:59] I guess I could run it on every VM everywhere and it'll just fail where scratch isn't mounted... [15:56:02] great [15:56:11] taavi: does that help for pods? [15:57:38] fixed launching new pods at least [15:57:43] ok [15:57:56] anyone know what became of F{} ? [15:58:01] Or if I'm just messing up the syntax? [15:59:40] has that ever worked on the cloudcumins? I only see that plugin installed on the per-project cumin master profile (which was also used on the in-cloud cloud-wide cumin masters before we got the cloudcumin VMs) [16:01:00] maybe not [16:01:05] anyway, I'm doing it the hard way [16:01:09] which is not all that hard [16:01:30] any reason to not run that remount command on all tools hosts? [16:01:45] I think taavi is doing that? [16:02:09] he said just nfs workers before, I think [16:02:27] go for it [16:02:37] ah, okok [16:02:52] great, doing for tools and toolsbeta [16:03:28] but like, it's almost all of toolforge that is already done, i would focus on the rest of the projects [16:03:52] or just run it cloud-wide with an grep condition against /etc/fstab including the magic line [16:03:53] rest of projects are done [16:03:56] and now everything is done [16:04:00] cool [16:04:10] re-checking to make sure it actually fixed things everywhere, that'll take a few minutes [16:04:30] thank you. did anything hang? i guess you'll know as a result of what you are checking now.. [16:04:36] and of course nfs worker reboots continue, which will force all pods to get rescheduled. [16:04:41] you can ^C that [16:05:09] really? Don't we want to force everything to reschedule, for pods with busted mounts? [16:05:32] are the mounts busted inside the pods? [16:05:46] I don't know, I just assumed they were since they were busted everywhere :/ [16:06:32] bah, you're right, they are [16:06:40] ok, so we'll let this continue then... [16:06:44] yeah [16:07:35] but VM/OS level things are looking good. So I think we're all clear except for the pod reschedules. [16:07:47] Not sure if that means the incident is over or if the incident stays open until the reboot cookbook finishes [16:07:52] dcaro can decide :) [16:08:09] I'd wait until users are not affected [16:08:34] (and I understand that until the reboots they still might?) [16:08:43] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1320183 is still open [16:11:21] taavi: LGTM [16:11:29] proposed followup email [16:11:30] https://etherpad.wikimedia.org/p/scratch_followup [16:14:47] andrewbogott: the message looks good. I hope you can document the testing that had been done prior to this downtime. [16:14:53] (in the phab ticket) [16:15:05] sure [16:16:24] dcaro: want to make an incindent doc, or just track things on the task? [16:16:45] things seem to start getting through on toolforge right? [16:16:51] https://docs.google.com/document/d/1xdDfrE97Mvivwhz3Y12Y2xEfeWgWav8Zo7QIiR2lz8w/edit?tab=t.0 [16:16:57] I thought I had shared xd [16:22:10] ok, email sent, now I'm just filling in things on the report [16:22:33] So the current status is that we are waiting for all the toolforge nfs workers to reboot, to get all the pods that were already running unstuck right? [16:22:37] but also, there's a training at the SRE meeting that I want to attend so there will be a delay before I do the rest of the documentation. [16:22:39] dcaro: that's right [16:22:45] ack [16:23:00] the training is off (trainer has wifi issues) so this work can continue [16:23:04] andrewbogott: that's ok, I'll update the status when the reboots are done and declare the incident over [16:23:07] xd [16:23:10] dcaro: You should go back to not working as soon as you like [16:23:27] andrewbogott: any idea on when the reboots will complete [16:25:05] I haven't run this cookbook in a while; taavi do you remember how long it takes? 90 minutes? [16:25:20] andrewbogott: ack, got still ~30 min :), I'll let you handle though [16:25:38] oops either the training didn't happen or I'm thinking of the wrong meeting. [16:25:42] So, I will write! [16:26:22] dcaro: when you say 'declare the incident over' you're just talking about the gdoc and the irc notice right? [16:26:32] yep [16:27:03] mainly to tell users that things should be back to normal, and new errors should be considered issues (not part of the incident) [16:27:08] andrewbogott: it'll take on the order of a couple of hours at once [16:28:06] yep, it's not half way there yet [16:28:07] andrewbogott: just to clarify, the training was supposed to happen in the SRE call earlier today but didn't, because the trainer had wifi issues. so you didn't miss anything [16:28:49] ok! ty [16:30:00] I guess you said that already, I'm reading too many windows at once :) [16:30:02] Raymond_Ndibe: thilp fyi. when an incident is still ongoing, refrain from deploying stuff [16:30:43] thanks! I was precisely thinking of delaying https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/361 :) [16:31:12] 👍 [16:53:15] I have not opened tasks yet but have written all I know to write on the incident doc. [17:33:22] * dcaro off [17:33:30] let me know if you want me to review/add stuff tomorrow [17:33:31] cya!