[08:41:50] I'm seeking an easy +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/900238 [08:50:03] claime: <3 <3 thank you sir [09:58:24] kamila_, hnowlan, apergos: head's up for tomorrow eqiad row B maintenance, most of the core plaftorm table is empty in https://phabricator.wikimedia.org/T330165 [10:00:31] I can put a no action needed on all my ones, we'll lose wikidata entity dumps this week but there's no way around that really [10:00:38] the rest are in between runs [10:01:59] {{done}} [10:02:50] hey godog o/ do you need a hand with switching alert to overlay2 https://phabricator.wikimedia.org/T329939 ? I've prepared https://gerrit.wikimedia.org/r/c/operations/puppet/+/902318 so we can force docker config [10:02:56] XioNoX: thanks for the heads-up! [10:03:08] thanks! [10:03:19] hnowlan, apergos I think your edit collided :) [10:03:42] you fixed it! cool! [10:04:53] jayme: yo! thank you, I haven't looked yet into what needs to be done but I'm game, also none of the docker users on alert hosts is in production yet so we can flip at any time [10:09:53] godog: The process, in case no container is running is: "systemctl stop docker.*; rm -rf /var/lib/docker/*; systemctl start docker.*" - as long as the overlay module is not in the blocklist [10:10:00] which it still is on alert [10:10:59] jayme: ack, so flip hiera first, puppet run, stop the containers and run your commands above ? [10:11:37] godog: plus "modprobe overlay" [10:11:58] ah yeah of course [10:13:58] jayme: one extra thing that I did in the past was umounting explicitly container leftovers, when the rm -rf /var/lib/docker/* didn't clean up all the subdirs [10:14:27] hm..that should only happen in case there are containers running [10:14:29] I gues... [10:14:46] yeah I know, it definitely happened after the drain and after stopping dockr [10:19:35] ah, I see [10:23:42] godog: I'll go for lunch - let me know if you'd like some help [10:25:23] jayme: thank you! yeah same here, I'll ping you after lunch [11:05:42] anyone know who would be a good contact to chat and change phabricator permissions? [11:05:51] for a bot account [11:18:18] is the SRE Team Vacations calendar automatically constructed from Out Of Office events? I'm not sure if it's not showing my leave because it thinks I know about it already, or if I've done something wrong [e.g. 2023-05-29 ] [11:24:00] Emperor: it checks the text in the event and matches a bunch of obvious OOO names, IIRC it's not in real time, so might take a bit to get it [11:24:18] cc jobo if it doesn't work for you :) [11:26:27] jbond: surely Andre, not sure if there is a phab tag to use for that specifically [11:27:21] ack cheers ill ask Andre [11:28:08] Yes, that is correct and run is set to 1h. Emperor please let me know when you’ve set up the time off and if it was more than 1h ago please let me know which day is it for and I will check it. [11:45:39] jobo: seems to have picked up everything correctly now, thank you! [11:46:50] 👍 [12:16:34] jayme: basically https://gerrit.wikimedia.org/r/c/operations/puppet/+/903238 and the operations we discussed earlier [12:20:57] godog: +1ed [12:21:02] neato, thank you [12:34:43] jayme: alert2001 done, will do alert1001 [12:34:53] FWIW modprobe is taken care by puppet [12:38:18] ah, interesting [12:39:05] well, maybe it's actually docker [12:46:23] jayme: all good on my end [12:46:47] godog: sweet. We should be good do merge then [12:47:31] godog: if you have another minute to review https://gerrit.wikimedia.org/r/c/operations/puppet/+/902318 we could wrap it up right now [12:47:39] will do [12:49:13] cool, thanks! [12:49:29] sure np [14:15:44] <_joe_> dcausse: I made a change to the chart of flink-session-cluster that will only remove some redundant env variables; I won't deploy the change to production but just to staging though [14:15:56] <_joe_> just so you're warned next time you have to deploy [14:16:08] _joe_: thanks for the heads up! [14:16:15] inflatador: ^