[10:33:12] andrewbogott: cool :) [17:17:05] I just notice backup1008 was setup, so the extra backup servers should be ready to be put on service now [17:17:14] *ed [17:55:39] jynus: if you can give me a hostname that has SD card I can have a look at what shows up in the redfish api [17:56:03] sure, as long as it is safe :-) they are backup servers [17:56:09] let me double check the ticket [17:56:27] GET are always safe :D [17:56:39] not in mediawiki :-D [17:56:46] like in mediawiki, I was about to say :D [17:58:18] I am checking it, one second, I am still searching for the ticket [18:02:11] ah, I remember now, they were not backup servers [18:02:24] they were databases for backup sources, on reimage [18:04:13] https://phabricator.wikimedia.org/T280979#7498786 [18:04:26] it was db1139, among others, volans [18:04:46] db2100, too [18:05:02] they may be old hw, so maybe not worth it [18:06:16] and it happened after a motherboard change on both [18:09:39] x1 is scheduled maintenance, right? [18:11:01] I can see yes [18:12:54] yep: https://phabricator.wikimedia.org/T300965 [18:13:16] FYI, monitoring is creating a bit of noise, but only on mw2* servers FYI [18:15:10] weird, because the logs say "Wikimedia\Rdbms\LoadBalancer::pickReaderIndex: all replica DBs lagged. Switch to read-only mode" but codfw should be in read only mode to start [18:15:35] probably just a missleading error message [18:18:20] jynus: ack, thx, I can have a quick look tomorrow and see what come out, if anything [18:18:32] no rush, I thought it was on backup hosts [18:18:39] on those probably not a big deal [18:19:03] I was mostly asking if we keep buying different hw, how to handle different needs [18:20:36] as long as it something dependent on the hardware we can probably put the logic in the cookbook like we do now to pick which NIC enable PXE on [18:22:05] that was the only answer I needed thanks [18:22:17] if it's something logical per cluster or other, I'd say dpends on the need. For example for our future work on the raid part we'll need to have that config come from some source of truth place [18:22:43] jynus: I imagine that mediawiki is stepping in before the MariaDB read only level [18:22:45] we could use the same source of truth (TBD) for customized bios settings too [18:22:45] yeah, step by step [18:23:11] I just got excited by the possibilities, volans :-D [18:23:32] You'll have MariaDB read only which will be detected when a query happens, the lag detection automatic read only (that) and the $wgReadOnly config variable all at play [18:24:08] RhinosF1: my guess is that it says "switching to read only" despite being already setup at mw level in read only [18:24:49] :) [18:25:14] jynus: yep, it'll probably be getting confused between the numerous methods to make it read only [18:25:42] in any case, it is enforced also at db level to avoid mistakes [18:25:53] yeah