[04:46:43] For the Europeans when you wake up: cloudvirt1071 froze up around UTC 04:05. I rebooted it 15 minutes later, drained it, left it in maintenance, and send a cloud-announce email about affected VMs (there weren't many). [04:48:16] This is all tracked on https://phabricator.wikimedia.org/T431374 -- I'm not seeing any useful explanation but if anyone has a curious morning mind please have a look! [08:01:07] morning! [13:40:43] * andrewbogott waves [13:41:43] godog or taavi, if you have a minute to look at cloudvirt1071, I'd appreciate a second opinion. As it is I don't know if it's safe to re-schedule VMs there... this is the second time it has crashed but 7 months between crashes. [13:48:44] andrewbogott: a cooling issue perhaps? the core temperatures I see in Grafana are significantly higher than other cloudvirts I checked [13:48:46] so https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&from=now-30d&to=now&timezone=utc&var-server=cloudvirt1071&var-datasource=000000026&var-cluster=wmcs&refresh=5m&viewPanel=panel-25 [13:51:07] hmmm that graph shows two big steps down... the first step down /should/ be from the cookbook reboot but that doesn't really line up [13:51:17] but doesn't that show it as having been cooler than ever right before the crash? [13:51:49] thermal throttling of some sort? would explain why it froze up [13:52:09] I see this in the kernel log: https://phabricator.wikimedia.org/P94751 [13:52:48] * andrewbogott checks if cloudvirt1072 is next to cloudvirt1071 [13:53:30] nope [13:53:49] 1072 shows that same weird step-down in temp yesterday, but other cloudvirts don't show it as much [13:53:52] or at all [13:55:18] Do you happene to know where the temp probe is for that metric? [13:55:28] I'm wondering if we should suspect fan, or thermal paste. [13:57:41] oh, the metric is labeled 'core' so probably it's actual cpu temp [14:01:54] * andrewbogott makes T431429 [14:02:14] ty taavi [14:32:42] I'd appreciate thoughts on https://etherpad.wikimedia.org/p/paws-cleanup [14:33:05] I'm not detailing the specifics of the cleanup algo there because I don't want to train the enemy [14:59:59] andrewbogott: regarding paws cleanup, could we consider allowing the volunteers to proactively contact us with information about files that are typically large but still legitimate? i.e., allow 1 week for them to proactively flag data that should not be removed (with an explanation of what it is), then implement the cleanup with those exceptions [15:01:03] I suspect that there are 0 legit use cases that will trigger the scanner. I'd like to proceed on that assumption for starters and if we run into legit large-file cases we can add a simple allow-list to the scan. [15:03:11] um... I'm not 100% opposed to doing things in the reverse order, it's just that in my experience even if there are legit users with giant storage needs, it won't occur to them to ask permission ahead of time. [15:04:13] are there specific users we could proactively contact? I realize this (what you propose) would go faster [15:04:46] hmmm [15:05:48] I tend to treat paws users as a black box (since they present as a numeric ID on nfs rather than as a username). So it would be a bit of work to a) correlate humans with IDs and b) identify legit power users [15:05:56] possible! But not work we've already done [15:07:39] Unrelated note: I'm about to unceremoniously reboot clouddumps servers for T430915, brace for impact. [15:09:05] do we typically do this (remove, and wait for complaint) and has this stance ever been met with constructive feedback from volunteers? curious [15:10:57] Yes, historically we've moved large files out of the way and then later deleted them. When we move them we leave a note in their place saying to contact us for file recovery, and no one has ever asked. [15:11:10] So the thing I'm announcing today is the same as that, just more aggressive. [15:12:07] ok [15:13:20] thilp: heyo, :), we might have to coordinate a bit as I have some mergable MRs on jobs-api too, you just got an MR merged right? [15:13:48] I did! does it need to be deployed though? It’s a unit test change [15:14:02] oh, we reformatted too [15:14:18] well, it can be deployed with others, but usually we deploy the changes one by one [15:14:41] (as you say, if you have a few MRs changing tests, you can merge them all together, and then deploy all together) [15:15:07] sounds good. Should I merge https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1321 then? [15:15:20] We do avoid having the upstream code ahead of what's running in production (as in, even if it's a test, if it got merged, we don't let it hang for days without deploying it) [15:15:30] yep, makes total sense [15:15:42] thilp: nope, we have to deploy it first in toolsbeta and then in tools, do you have access already? [15:16:10] this is the cloudcumin part, right? [15:16:19] yep :) [15:17:29] not yet (just tried ssh to tlepage@cloudcumin1001.eqiad.wmnet with the prod key) [15:18:10] okok, do you want me to do it? (we can do in a video call if you want, should go ok though) [15:18:24] did your puppet patch get merged though? [15:18:56] I think it did? see second-to-last update in https://phabricator.wikimedia.org/T431010 [15:19:59] re: want me to do it? — yes cause I wouldn’t want to block the queue! Happy to watch if you have time 🙏 [15:20:18] oh, I think they merged the change in "approver" and bliviero approved it, but it's missing the other patch adding you to the groups and such [15:20:26] (I think, let me double check) [15:21:36] yep, you are missing ssh keys, groups and such, so let's wait for that, let me open a meet [15:36:25] taavi, godog, as predicted the clouddumps reboots were a total non-issue for toolforge as far as I can tell. [16:57:37] 🎉 [16:57:41] that's good [17:14:26] * dcaro off [17:14:29] cya tomorrow!