[00:24:25] FIRING: SystemdUnitFailed: pt-heartbeat-wikimedia.service on db1125:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:24:25] FIRING: SystemdUnitFailed: pt-heartbeat-wikimedia.service on db1125:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:54:07] db1125 is a test nod for Manuel so I guess we should let it aside to avoid loosing his work? [05:54:12] node* [07:24:34] If it's not being used for anything prod-ish, although it's a long time to leave a node unhappy. [07:26:15] I've downtimed the host while we decide [09:02:07] heads up I'm upgrading all clouddb* hosts from mariadb 10.6.18 to 10.6.19 (T365424) [09:02:08] T365424: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424 [09:02:41] * volans runs away :D [09:04:09] * arnaudb is in a meeting x) [09:15:43] Nah. These updates are minor 😁 update it to 11 because I'm ooo [09:16:58] :D [10:44:48] 4 out of 8 clouddbs are upgraded, I'll do the rest after lunch [10:47:29] there's a small gotcha in my upgrade+reboot process, maybe volans has ideas: I have to disable puppet before rebooting, but the reboot-single cookbook doesn't re-enable and this causes icinga alerts to fire immediately after the reboot [10:47:41] this is the process I'm following https://wikitech.wikimedia.org/wiki/MariaDB/Rebooting_a_host [10:48:23] I wonder if we could add another option to the reboot-single cookbook (there's already --enable-puppet but that enables it _before_ rebooting I think) [10:49:31] puppet runs @reboot, so there is no way to enable it before that run unless enabling it before shutting it down [10:50:18] but it could enable it after it detects the boot? [10:50:36] sure but you're still open to race conditions that might trigger the alerts [10:51:30] hmm but the cookbook could then wait_for_optimal like it does if puppet is enabled [10:52:07] I'm also confused because I add a 1-hour silence before rebooting, but that's also getting cleared at some point [10:52:19] yes at the end [10:52:35] with self.alerting_hosts.downtimed [10:53:20] should it delete just its own silence, and not other existing silences? (not sure if it's possible with icinga) [10:53:43] correct, alertmanager deletes the silence ID with icinga the whole host is cleared [10:54:05] as there is no easy way to identify specific downtimes [10:54:08] ack [10:54:22] why do you need puppet disabled? [10:54:50] because I'm running "umount /srv" and I don't want puppet to create files in there [10:55:00] there are plans to work on a specific reboot+upgrade cookbook for databases but not yet there [10:55:14] maybe we need a custom cookbook yeah [10:55:31] why the umount? [10:55:35] * volans trying to get more context [10:56:06] m.anuel was worried of some data corruption I think, let me see if I find the details [10:56:32] "If you don't umount /srv manually, there is a risk that systemd does not wait for the umount to complete and that can lead to data corruption." [10:56:56] (this is in https://wikitech.wikimedia.org/wiki/MariaDB/Rebooting_a_host but came from some old wikitech page, not sure if it's still true) [10:58:11] there 2 reasons [10:58:26] first, preventing forgetting about stopping all processes on /srv [10:58:50] which had happened before (I stopped a mysql instance, but left another running) [10:58:54] so that would fail [10:59:11] the other is that sometimes there can be a lot of caching on memory [10:59:43] and that basically syncs disk before attempting the reboot, which usually makes the reboot faster [11:00:06] faster as in, after you ask for a reboot [11:01:00] wouldn't "sync" suffice instead of umount? [11:04:31] * dhinus lunch [11:07:20] AFAIK puppet doesn't mess with any mysql data no? [11:08:13] * volans about to go for lunhc [11:08:17] *lunch [12:00:19] arnaudb: jynus are you planning to do any further maint on s2 or s3? I want to test circular replication in one of those [12:00:38] (or s5/s6 if you're done with the switchover) [12:03:09] Amir1: could you please wait to talk to volans ? [12:03:56] sure, I will do it tomorrow but I'm in a conference these days and won't be around much [12:04:48] I think you should be doing those coordinating with others, if you are not going to be around much [13:01:16] volans: afaict puppet only creates the empty dirs in /srv, so it's not a big issue if it does it on an unmounted volume... but still not ideal [13:01:57] it's in modules/mariadb/manifests/instance.pp [13:02:47] that's why i was asking if we could prevent from umounting in the first place if the only outcome we want is to flush teh cache to disk, "sync" should be enough, but ofc should be tested [13:05:24] Amir1: s2/s3 are paused already on my end, I still want to try and do s8/s7/s5 as they have 3 nodes total and I've already swapped some of those codfw masters [13:05:33] T367781 is up to date [13:05:36] T367781: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781 [13:09:15] volans: I guess the "answer" might be to automate the reboot procedure in a db-specific cookbook [13:09:41] * arnaudb loves this ↑ [13:09:44] even if we fix the alerts, it's still sub-optimal using the reboot-single cookbook because it will wait forever for puppet to succeed, and you have to start the db manually in a separate window [13:10:34] dhinus: of course we need a dedicated cookbook, but automating it doesn't remove the race condition so I was trying to avoid it all together removing the need to disable puppet in the first place [13:11:37] I agree if we could remove the "puppet-disable" it's nice, but why do you think a dedicated cookbook does not remove the race condition? [13:12:30] the cookbook could handle the reboot internally and wait before removing the icinga silence [13:12:43] (migrating the alerts to alertmanager would also help :P) [13:15:20] :D [13:15:25] sure sure [13:17:42] the good news is that these alerts are not paging I think [13:18:00] you'll see a few more today :) [14:11:03] volans jynus not urgent, but I would like your review on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1072755 [14:12:11] dhinus: the cookbooks related to those hosts are in the prod or wmcs cookbooks repo? [14:12:25] to understand if those can still be run or need to be moved [14:12:25] prod [14:12:28] ok [14:13:51] I had an idea to move them, but I closed that as declined T347977 [14:13:52] T347977: cloudcumin: allow wmcs-admin to run wikireplicas cookbooks and scripts - https://phabricator.wikimedia.org/T347977 [14:14:23] I think we can keep them on prod cumins for now [14:15:25] question inline [14:18:08] dhinus: tecnically it seems safe, I don't know policy-wise [14:20:13] volans: replied [14:20:43] jynus: policy-wise it should be at least safer than the current situation, i.e. it reduces the number of people who have access [14:21:31] dhinus: yeah, not blocking on that regard. What I meant is that I added some worries to the ticket, but didn't ask for that solution [14:21:36] thx, +1ed [14:21:51] as in, "it has to be like that" [14:22:14] I did the same [14:22:55] jynus: yep I know, I also think there might be "better" solutions, but given there was no clear consensus on the ticket, I think this at least makes things more consistent. we can revisit it in the future [14:23:03] +1 [14:23:14] thanks both :)