[09:31:23] hnowlan, bjensen o/ I just repooled pki1002 for https://phabricator.wikimedia.org/T434268 [09:31:47] if any puppet failure re-happens please just depool it, I am also working on adding more VMs and an LVS service [09:31:55] elukey: ack, thanks! [09:31:57] but now we have two pki nodes, that is better than one! :D [09:32:11] also telling it to myself since I'll be on-call after lunch [09:32:45] ty! [09:39:59] Just a reminder, we're beginning to decommission the remaining parts of the graphite infrastructure from today. if you see any alerts about it that's probably related. given how long graphite was around, there's a chance it might break things in an unusual manner so let us know [10:09:50] wow the moment has finally come! [10:10:00] IIRC graphite was already there when I started lol [10:22:33] the carbon receiver has been firewalled off for a year so it's been effectively dead for a while :D [14:22:45] If no one objects, I'm going to do a couple of rounds of rolling restarts on sessionstore. No impact expected. [14:23:07] clarification: rolling restarts of Cassandra (not the sessionstore service) [15:11:35] about to disable the graphite web interface- I've put in a silence but just a heads up elukey, Raine [15:14:11] ack, thanks [15:21:58] graphite web is dead [15:23:23] long live graphite we—er, prometheus+grafana? [15:27:03] yep [15:43:14] {◕ ◡ ◕} [16:56:12] early handover for future on-callers - pki1002 showed some issues during the past days, it was depooled till this morning. I've upgraded BIOS and iDRAC as suggested by DCops, we'll see if this is sufficient. Due to how things are configured, we only have one host for eqiad and one for codfw, so if pki1002 fails, all hosts dns-mapped to it will fail (puppet failures etc..) [16:56:56] it is sufficient to depool it: `sudo -i confctl --object-type discovery select 'dnsdisc=pki,name=eqiad' set/pooled=false` on puppetserver1001 for example [16:57:15] if it happens, please log something here https://phabricator.wikimedia.org/T434268 [16:57:35] we have plans to make everything more robust via LVS, I'll keep you updated [18:43:21] Thanks e.lukey! [18:53:37] ty!