[06:41:00] greetings [08:37:27] howdy? [08:39:58] morning [08:55:35] I'm cleaning up old MRs, I got this one https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/10 that make k8s-status move to python3.13/poetry (and fixes ci) and such, any reviewers? (deployed already on toolsbeta) [09:05:12] dcaro: approved, I didn't test it locally, but if it works in toolsbeta that should be fine [09:05:28] it was hard to avoid moving ci to 3.13, and avoid upgrading the deps, and the other way around too [09:05:42] we can't move to buildservice directly yet, because of the custom deployment [09:05:51] (uses k8s directly, instead of webservice/jobs-api) [09:07:55] we have so many repos and things that it's hard to keep them all up-to-date :) [09:11:44] of course, there's some other error in prod xd [09:11:46] looking [09:15:10] it seems that the read_namespaced_pod function changed behavior (now returns 404 raising exception) [09:22:32] hmm.. it's failing on timeouts now "Liveness probe failed: Get "http://192.168.166.179:8000/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" [09:23:05] oh [09:23:06] https://www.irccloud.com/pastebin/3vAjCEC8/ [09:24:21] hitting 4G `k8s-status-5ccd86778c-54nkl 582m 4095Mi ` [09:25:59] oop, got there https://k8s-status.toolforge.org/ [09:26:03] nice! [09:27:15] yep, I think it was the initial fetching of everything :/ [12:51:28] just did a full run of https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/76 on toolsbeta, freed ~40G [12:52:05] let me know if anyone starts getting errors anywhere xd [14:47:49] andrewbogott: when working on https://phabricator.wikimedia.org/T419892 I came across https://phabricator.wikimedia.org/T439041 for the latest cloudcephosd hw and was wondering if going even denser with 12x SSD + 2 m2 internal for the OS was considered and/or if it is an option ? [14:49:18] I'm pretty sure that 10 SSDs + boss card is the most we can fit in a 1U server. [14:49:27] If 12 will fit then yeah we should get 12 [14:51:08] heh I'm not sure for dell, though from the sm specs at https://www.supermicro.com/en/products/system/clouddc/1u/sys-112c-tn it does seem available, that'd be 12x drive bays hot swap and 2x m.2 nvme internal for the os [14:51:25] or at least that's my reading [14:51:57] but yeah might as well go as dense as we can [14:52:00] oh yeah, I'm definitely thinking of dells. Of course we can't get competing bids if we adopt a sm-only config... [14:53:09] pinged Rob on the task, let's see [14:53:15] indeed, and thank you [16:28:25] * dhinus off [18:02:48] * dcaro off