[09:31:08] it's been _years_ since I had to run an sosreport, that was some neurons I thought I'd long-since recycled :) [09:32:04] FIRING: [2x] SystemdUnitFailed: pt-heartbeat-wikimedia.service on pc1013:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:32:34] will downtime ↑ [09:32:46] good memory Emperor :D [09:56:46] I had to get through vet school somehow ;) [09:58:11] (though it is always a bit "forget important stuff, remember detailed account of ruminant digestive enzymes for evermore") [10:24:09] xD [11:24:31] Emperor: o/ the "urgent" hosts are thanos-be[12]005 right? [11:25:33] elukey: correct [11:25:57] ok so you will be surely pleased to know that 2005 has the wrong BMC firmware :D [11:26:22] I asked to DCops to upgrade it, I can't remotely [11:26:34] and now I am checking 1005 but I suspect it will have the same issue [11:26:45] I'll keep the rack tasks updated [11:28:14] yes same issue sigh [11:28:18] dear vendors, please to be shipping the correct firmware :facepalm: [11:28:48] I'll flag that to willy as something we're obviously going to have to be more explicit about in the procurement process :-/ [11:29:07] * Emperor thinks calming thoughts [11:30:02] we already told them, I think it got missed due to the urgency of the shipment [11:30:15] we had a similar issue but with another model/motherboard [11:30:30] but all the ms-be ones have the right firmware [11:34:08] Emperor: do those nodes have 24 disks as well? [11:34:12] (just to confirm) [11:36:35] elukey: they should do, yes, they should be identical to the new ms-be* nodes [11:36:48] (and be likewise identically set up as JBOD etc) [11:38:09] all right updated all the tasks, the next step is to wait for dcops to upgrade the firmwares and hopefully that will unblock provision + jbod + reimage [11:38:29] Jesse may have found a solution for the reimage bug, so fingers crossed. [11:42:51] oooh :)