[04:29:35] We are going to be switching s4 primary master in 30 minutes [06:37:00] good morning folks [06:37:45] are swift hosts in eqiad expected to have ~94% usage of data disks across most of the nodes? [06:37:52] in codfw it seems around 75% [06:38:09] there is still a lot of space on each node so not an immediate concern [06:43:27] ahh I guess that we are waiting https://phabricator.wikimedia.org/T285808 [06:45:28] not sure, will ask to godog later on :) [07:25:52] * arturo is back [07:28:58] welcome back arturo! [07:29:06] o/ [07:35:21] that's correct elukey [07:35:26] welcome back arturo [07:36:10] thanks :-) [07:48:12] godog, I am not seeing backups impact on swift- according to my calculations, worst case scenario I am only adding a 3% overhead in reads [07:49:02] jynus: very nice, thank you that's useful to know [08:05:29] welcome back indeed! [14:38:57] volans: sre.hosts.decommission - does it care if it's run before or after the references to a host are removed from puppet? [14:39:23] kormat: some yes some no, it will show you the references when you run it [14:39:43] volans: what version of the puppet repo does it grep against? [14:39:59] e.g. if something is submitted on gerrit, but not puppet-merged, what state will it show? [14:40:05] the idea is first run it then remove the role and stuff, no need to do it earlier, but in some cases (cluster and stuff) you might need to remove it from other hosts config first before it goes down [14:41:03] kormat: the puppetmaster in chargeof CA [14:41:11] so puppetmaster1001 as of now [14:41:27] gotcha [14:41:33] so puppet-merged [14:41:42] or otherwise pulled :) [14:42:11] volans: to rephrase my initial question: will anything _break_ if the host is already gone from puppet before the cookbook runs? [14:42:43] IIRC yes, it might not be able to find the host, but checking [14:43:00] what about if you attach a jpeg of the host in a rack? [14:43:16] lol [14:43:25] if it doesn't match it asks you to confirm [14:43:55] ok. that sounds low-impact enough. thanks! [14:44:06] but I'm not sure if other parts break [14:44:17] I'm testing it right now as I have another host from anothe rtask to look into [14:44:25] so give me 10 and I'll tell you [14:44:59] volans: FWIW, kormat is asking because I am running it now for pc1007 whose removal CR is merge but not puppet-merged [14:45:43] ack, lmk how it goes [14:45:50] No 🔥 yet [14:46:10] END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc1007.eqiad.wmnet [14:46:43] did you pass it a task? [14:46:47] I'd like to see the update there [14:47:09] I said "cookbook sre.hosts.decommission -t T289118 pc1007.eqiad.wmnet" [14:47:10] volans: https://phabricator.wikimedia.org/T289118 [14:47:10] T289118: decommission pc1007.eqiad.wmnet. - https://phabricator.wikimedia.org/T289118 [14:47:32] thx I was looking for https://phabricator.wikimedia.org/T289118#7334559 [14:47:33] all good [14:48:15] volans: thanks 💜 [14:48:57] +1 [14:49:52] thanks for checking! [14:50:38] anyway, worse case scenario, the cookbook fails, you revert the puppet patch, the cookbook fails again because in a weird state, ad then we fix it in some way :) [15:34:18] anybody familiar with wikifeeds?