[07:58:47] elukey: FTR, you're up to 2.6T with docker_registry in apus [08:18:58] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s5.service on db2201:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:25:57] ^ downtimed, I am working on that host. [08:32:54] Emperor: o/ s3cmd -H showed 2TB, but calculating every bucket I see 2.3T, and the real size at this point is 2.6T [08:32:56] uff [08:33:22] I am working on cutting ~500G of space in these days, so we should get down to a more manageable level [08:33:31] I'll open a task to increase the 3TB limit [08:34:23] elukey: thanks. I am open to bribes in the form of help with T424895 ;p [08:34:24] T424895: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895 [08:39:48] Emperor: I was about to rant Supermicro-related things but with my great surprise those are Dells! [08:40:08] the error rings a bell though [08:40:23] it feels as if I already seen the problem [08:50:20] ahhhh yes got it [08:50:26] this is a subtle one [08:51:12] I already encountered the problem for idrac10 hosts, and indeed we have "'HttpDev1TlsMode': 'None'," in the Bios config changes [08:51:20] but ms-be1098 is a idrac 9 host [08:51:45] so this must be a recent firmware for idrac 9 [08:52:40] "yay" [08:53:06] [that at least sounds like it's fairly fixable, tho] [09:09:20] I am going to reprovision ms-be1098 [09:11:37] Updated value for attribute BIOS.Setup.1-1 -> HttpDev1TlsMode: OneWay => None [09:12:06] it is updated only if present, so we shouldn't affect the other deployments [09:13:57] 👍 [09:22:33] Emperor: done! Can you retry to reimage 1098? [09:23:26] yep, doing so [09:24:20] elukey: failed right away [09:24:45] elukey: 'spicerack.redfish.RedfishError: POST https://10.65.0.137/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset returned HTTP 409' [09:25:26] Response payload: {'error': {'@Message.ExtendedInfo': [{'Message': 'Server is al [09:25:26] ready powered OFF.', 'MessageArgs': [], 'MessageArgs@odata.count': 0, 'MessageId [09:25:26] ': 'IDRAC.2.9.PSU502', 'RelatedProperties': [], 'RelatedProperties@odata.count': [09:25:26] 0, 'Resolution': 'No response action is required.', 'Severity': 'Informational' [09:25:29] }], 'code': 'Base.1.12.GeneralError', 'message': 'A general error has occurred. [09:25:32] See ExtendedInfo for more information'}} [09:26:40] it feels as if the update is still being carried [09:26:51] but the cookbook finished, and the BMC reported the new value [09:27:13] marostegui: I don't see mention of backups in the DC switchover, but the processes write to the database. That said, I don't see mention of those clusters in the pre-switchover ticket (T436496), so are they excluded and backups are left running as-is? [09:27:14] T436496: Pre switchover: Database preparations - https://phabricator.wikimedia.org/T436496 [09:27:26] cezmunsta: Yeah, nothing to do there [09:27:27] elukey: FWIW serveraction powerstatus reports OFF [09:28:47] marostegui: *\o/* win #1 for the day :D [09:29:09] Emperor: maybe dcops turned it down? shall we turn it on and retry? [09:29:15] the new option is set in the bios, I checked [09:30:09] I've serveraction powerup'd and restarted the reimage cookbook [09:30:20] very weird though [09:33:56] now in the BIOS Configuration thingy, which looks to have completed OK [09:37:28] elukey: has started UEFI HTTP boot OK. Is whatever you did easy to do to the other two hosts in that ticket? ms-be1099, ms-be1100 [09:38:54] elukey: installer has failed because it can't find any disks, but I think that's a DC-ops issue most likely [09:39:12] elukey: but the not-booting-into-installer-at-all problem looks good [10:08:11] Emperor back sorry! So https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1341708 is out, what I did was simply to test-cookbook it, I can apply the new settings to the other two nodes now [10:08:24] sigh for the disks, lemme know if there are other problems [10:27:40] elukey: cool, thanks :) [10:28:13] Emperor: spoke too soon, ms-be1099 doesn't have any interface configured in netbox https://netbox.wikimedia.org/dcim/devices/6850/ [10:28:25] so I think dcops stopped when ms-be1098 didn't work? [10:29:07] elukey: yeah, looks like. But I think that means I can feed all this back to dcops, and then when they provision the other two it should Just Work (TM)? [10:29:14] I'll update the ticket shortly [10:30:21] Emperor: only after the code review is merged, they don't have access to test-cookbook [10:30:37] elukey: True, I'll note that too. [10:35:05] {{done}}