[06:46:05] what's the deal with db2117? [08:28:22] it does not seem reachable nor has metrics [08:29:09] yeah, I saw Amir1 depooled it last night but there's nothing associated to it so maybe it crashed? can you investigate/track it in a task? [08:29:24] it's been seen down since 6hrs ago for 24hrs [08:29:50] imho there is some testing going on, I'll check with Amir.1 when he arrives [08:29:55] thanks :* [10:15:49] I think my schema change broke it [10:15:50] :( [10:16:47] up to a point where SSH is gone? wow [10:17:49] in s6, I was just doing change of pk [10:19:48] https://www.irccloud.com/pastebin/gynuDVkp/ [10:19:54] yup, the schema change broke it [10:20:08] oof [10:20:53] can't ssh into it either [10:21:00] will check ipmi [10:41:31] let's create a task for tracking and follow up [16:01:37] arnaudb Amir1 we should probably downtime db2117 for a week or something, cause otherwise it might page during the weekend if the downtime isn't large enough [16:02:04] iirc Amir1 was disabling notifications [16:02:27] manually? [16:02:38] cause if done via puppet it won't work as puppet won't run on the host [16:02:45] oh [16:02:59] I'm not sure actually [16:03:03] will downtime it for the weekend, just in case [16:03:10] yeah, give it 3-4 days [16:03:29] done [16:03:42] thank you! [16:04:21] Ah. I did it with puppet [18:05:40] hello - T358892 [18:05:41] T358892: db1169 is lagged over 16000 seconds - https://phabricator.wikimedia.org/T358892 [18:05:47] db1169 is pooled, but is running a schema change atm [18:06:39] in SAL I see marostegui downtiming it but no depools [18:06:49] should I depool it? [18:11:37] That's weird [18:11:40] Please depool yes [18:11:45] Thanks [18:12:58] taavi: I'll check why it wasn't depooled automatically with the schema change script [18:13:02] When I get home [18:13:18] depooled. thank you! [18:15:09] Thank you!