[12:06:55] looks like pcc is broken? possibly because cloudelastic1005.eqiad.wmnet is still in puppetdb but has been decommissioned/is being decommissioned [12:07:18] I might delete it in puppet to unblock [12:32:49] ah I see https://phabricator.wikimedia.org/T380937#10461825 [12:52:46] volans, elukey: got a second for a poweroff failure in https://phabricator.wikimedia.org/T384001 ? [13:06:42] federico3: o/ sure, what's up? [13:07:31] it seems the IPMI shutdown failed, anything we can do about it? [13:10:00] federico3: in general, you can use https://wikitech.wikimedia.org/wiki/Management_Interfaces#Reset_the_management_card as possible list of scenarios/fixes [13:10:44] elukey: It was during the decommissioning process, so the host is hard down already [13:11:13] elukey: Should we just switch it off manually via IPMI? [13:12:33] marostegui: if the host is already down I'd say no, it is going to get decommed so we don't really care. The cookbook tries to shut it down IIRC, but if it fails the rest of the cleanup should proceed [13:12:48] so from my pov we are good, not sure if I am missing something though [13:13:10] if federico3 wants to play a bit with resetting the idrac etc.. it may be a good time to do it :D [13:14:20] idrac? [13:14:39] the BMC basically, it is called like that in Dell's land [13:22:44] sorry was at lunch [13:24:17] the decom cookbook doesn't do gentle shutdown, it pulls the plug via ipmi, we can try to reconnect to it, if it works, or if it's broken you can ask dcops tomorrow to power it off physically [13:31:48] federico3: I think we can just do that ^ [13:32:09] works for me [13:37:00] but isn't the host already shutdown? Or did I misunderstand? [13:43:27] I am a person guilty of this, but we should try to avoid acronyms when possible [13:43:57] IMHO (pun intended) [13:44:30] specially when they are vendor lingo or WMF-isms [13:53:06] marostegui, federico3: I can login fine on the web UI of the management console. Do you want me to power it off? [13:53:12] we're talking about https://netbox.wikimedia.org/dcim/devices/2211/ right? [13:54:38] yet the IPMI shutdown failed so maybe it's a glitch in the decommission script? [13:59:59] no, IPMI is not that reliable, in old hosts it might get broken for N reasons, hence the wikitech https://wikitech.wikimedia.org/wiki/Management_Interfaces page [14:00:19] that helps to diagnose what's the issue and how to fix it [14:01:01] but now that the decommission cookbook has gone through, it's all a bit more complex as the DNS records are gone, just the management IP is still live [14:05:47] lmk if you want me to shut it down or not [14:51:35] Hey guys... Joe Biden here. I've decided to step down from the White House to focus on other projects. Billionaires are a threat to democracy, so check out https://BidenCash.st to put them in the bullseye. Keep an eye on the CNN inauguration for a promo code! [15:17:02] volans: sorry, yes, please power it off [15:17:16] federico3: ^ [15:18:17] {done} [15:18:18] and logged [15:18:23] thanks [15:18:26] per off [15:18:30] *power off now [15:19:43] thank you volans [16:18:53] We need to update all SRE documentation linking developer account to Wikitech :-( [16:19:01] starting but there will be a lot of stuff outdated [16:24:28] also check_user no longer works because LDAP is no longer attached to wikitech