[06:52:02] greetings [08:00:13] Morning! [09:04:28] is anyone doing anything to cloudvirts 1040/1041? just saw a few alerts going to the email [09:04:34] (flapping) [09:07:54] it seems like the maintain-dbusers alerts are due to the refreshed clouddb servers coming online, and the cloning of existing hosts related to that [09:12:27] interesting, as in they are failing to create accounts on those as they are not ready yet? [09:13:08] that and also because cloning a new host involves briefly stopping mariadb on an existing host to copy the data over [09:17:34] hmm, is there an easy way to programmatically check if a host is being migrated? [09:17:47] (or not that easy too xd) [09:19:17] given that it does not happen often, might be not worth the effort to automate if it's not simple enough, and instead coordinate better to silence the error [09:56:03] opened this do discuss the systemd unit alerts in toolforge vms T431261, I think it falls more on the infra side [09:56:04] T431261: [infra,o11y] Fix/remove all failing systemd units from all VMs and enable alerts on failed unit - https://phabricator.wikimedia.org/T431261 [09:56:11] (happy to discuss though) [09:59:18] yeah, and makes sense to do i think [10:09:37] * dcaro lunch [10:12:23] * godog too [11:45:43] FYI I got a 500 when trying to live-migrate tools-redis-6, investigating [11:58:53] filed as T431282 and investigating [11:58:54] T431282: nova.exception.GroupAffinityViolation while live-migrating tools-redis-6 - https://phabricator.wikimedia.org/T431282 [11:59:52] hmmm, anti-affinity seems to be blocking the move? [12:01:28] looks like it [12:01:40] "to host None"?? [12:03:05] I'm guessing that's what it says when you don't select explicitly a host? but yep, might be a bug [12:03:22] yeah I think so too, no explicit destination host [12:08:00] ok I think what I'll try next is reset the vm error state and ask nova to try again, if that fails too I'll live-migrate to a destination host explicitly [12:14:51] second time around migration worked [12:15:57] :l [12:16:19] it-crowd-roy-off-and-on-again.gif [12:17:43] lol [12:26:23] heheh it is funny because it is true [12:38:12] I'll silence NeutronAgentDown for a little bit btw, it is just noise [12:39:13] or rather, the silence doesn't seem long enough ? [12:41:17] yeah, seems like it removes the downtime directly after the host comes back up [12:48:06] indeed [13:36:35] taavi: not sure anymore it is a race, I filed T431300 [13:36:35] T431300: NeutronAgentDown firing during cloudvirt.safe_reboot despite silences - https://phabricator.wikimedia.org/T431300 [13:39:48] or rather, a race in the cookbook that is, surely a race somewhere [13:53:19] mmhh ok this instance https://horizon.wikimedia.org/project/instances/a27581db-fb90-4235-8b94-97248e74a519/ is suspended and wmcs-drain-hypervisor won't act on it, what's the SOP in this case ? [13:56:39] 'suspended' is such an edge case that we should maybe just remove the feature. [13:56:56] I would just unsuspend, repeat the migration, and then re-suspend. [13:57:23] And I guess open a task to figure out how to handle it properly in the cookbook someday [13:57:32] SGTM, will do [14:00:49] godog: other thing you might run into: 1) VMs that have huge ram allocations sometimes can't live-migrate within the allocated timeouts; the only good solution there is to keep retrying or stop them briefly during migration 2) sometimes cinder mounts get weird and prevent live migration (although I /think/ I full resolved this one). [14:42:10] andrewbogott: ack, thank you, good to know [14:46:18] re: deprecating VM suspend, that's now T431307 [14:46:19] T431307: Consider deprecating the 'suspend' VM feature - https://phabricator.wikimedia.org/T431307 [14:47:43] apparently we already disabled it a decade ago https://gerrit.wikimedia.org/r/c/operations/puppet/+/228288 [14:48:58] maybe we suspended the suspend [14:57:56] I always waffle about how much to customize or not customize our deployment :/ [15:17:06] andrewbogott: I'm signing off for today, cloudvirt.safe_reboot is still running on cloudcumin1001 root screen and hopefully won't need any more manual intervention [15:17:32] ok! I'll attach after you go and keep an eye out. [15:17:51] cheers [17:23:25] * dcaro off [17:23:28] cya tomorrow [19:12:49] bd808: at your convenience, I'd appreciate your thoughts on T431340 [19:12:50] T431340: Should we ban use of 'freeroot' on PAWS? - https://phabricator.wikimedia.org/T431340 [19:14:43] andrewbogott: I don't know if I should write this on task, but I would actually be ok with dropping the shell access in PAWS entirely and making them just use normal workbooks. [19:16:19] ok, so looking at this... [19:16:34] https://usercontent.irccloud-cdn.com/file/kn6On99m/paws.png [19:16:53] that would mean axing the whole 'Console' group and also all the Bash types? [19:17:00] Or are you suggesting something even broader than that? [19:18:04] I assume that a sufficiently-determined user can install a complete linux OS with pywikibot :/ [19:18:14] but that doesn't mean we shouldn't make things harder [19:19:46] andrewbogott: probably bash and teminal, but I don't know enough about it off the top of my head be certain about this being a reasonable idea. :) [19:20:42] ok [19:22:01] I'm sure that 'bash' icon acts as a big sign reading "free compute for t33ns" which is not ideal. But I don't think i have an easy way of asking paws users to justify themselves. [19:22:13] We could probably add a banner to a talk page...