[08:16:17] morning [09:22:27] morning [09:36:14] o/ [12:12:14] * arturo online now [13:22:47] there's many emails about admin.toolforge.org, and tools-static being down from the blackbox exporter side, have we changed anything on the network lately? (it seems to be unstable) admin has some errors in the logs too, but tools-static is on a VM itself, might be a coincidence, but well, just in case [13:23:48] I haven't changed anything that I can remember [13:25:29] the last semi-related change on ops/puppet seems to be https://gerrit.wikimedia.org/r/c/operations/puppet/+/1007566 [13:37:31] I'll treat it as separated things for now, the others only failed tonight also [14:09:32] * arturo food time [14:45:45] I'm deleting an (empty) cloud vps project, and following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Projects_lifecycle#Deleting_a_project [14:46:13] step 3 seems to be no longer valid "Make sure the project is not configured for NFS modules/labstore/templates/nfs-mounts.yaml.erb" [14:46:38] do we still have a list of NFS-configured proejcts somewhere? [14:46:45] quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/26 maybe @Raymond_Ndibe ? [14:48:19] dhinus: afair we are using the per-project nfs servers if needed [14:49:13] dcaro: thanks, I'll remove that line from the wiki [14:49:36] dhinus: it's now here https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/cloudnfs/data/projects.yaml [14:50:22] hmm ok so maybe I should just point to that list [14:50:26] and it seems we are still using it yep xd [14:50:37] but we have the nfs server of the project hardcoded there [15:06:47] dhinus: If the project doesn't appear in modules/cloudnfs/data/projects.yaml then it doesn't use nfs; I'm pretty sure that file is still used by projects that appear there though. [15:07:01] oops, I'm behind, you already updated the docs about that :) [15:11:17] ⚡ (lacking a flash superhero, seemed clearer than a camera flash) [15:35:30] can I get a +1 for T359061? [15:35:31] T359061: Request increased quota for mbh Toolforge tool - https://phabricator.wikimedia.org/T359061 [15:36:19] done [16:26:04] let me know if there's anything you want me to mention at the SRE meeting starting in 30 mins [16:31:47] successful migration to containerd (by taavi) :), if that was not already there [16:32:48] dcaro: that was part of the workers upgrade to bookworm? [16:32:57] yep, simultaneous [16:33:05] yes we did those two together [17:08:45] I checked and it's the first SRE meeting since we stopped all grid tools, I'll mention that [17:13:34] nice [17:24:15] arturo: you were asking about compute capacity; this dashboard (which I think you made) suggests that we're in good shape. https://grafana.wikimedia.org/d/000000579/wmcs-openstack-eqiad-summary [17:24:22] Like, maybe even /too/ good shape. [17:25:53] (also, thank you for the reminder to follow up on the hardware discussion!) [17:32:52] 👍 [17:40:31] quick review (breaking webservice for gridengine) https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/29 [17:59:00] how can I tell apt to select a local package over a remote one? [17:59:03] > Note, selecting 'toolforge-webservice' instead of './toolforge-webservice_0.103.3_all.deb' [18:00:40] can you just use dpkg instead of apt? [18:00:43] that message is misleading, but it is actually working with the deb you gave it [18:01:05] I'm not so sure, I don't see the changes I should be seeing :/ [18:01:07] if you're trying to install a new local .deb with the same version that you currently have installed, try --reinstall [18:02:00] I did, and apt remove + apt install [18:02:02] if not, please post the full output from apt [18:02:14] https://www.irccloud.com/pastebin/nhXZ3D1O/ [18:02:28] Installed version https://www.irccloud.com/pastebin/cKCO0ZRA/ [18:02:41] Version extracted from the .deb https://www.irccloud.com/pastebin/SU5hKtMW/ [18:03:23] * dhinus off [18:04:25] hmm, odd [18:04:31] `dpkg -i` worked [18:04:57] so if your local deb has the same version as what's in a repo, `apt install` will use the one from the repo [18:05:14] I did not have that issue with lima-kilo though :/ [18:05:52] maybe the debian version or some pinning config [18:06:12] anyhow, thanks :), `dpkg -i` worked good enough [18:14:51] * dcaro off [18:14:54] cya tomorrow [19:09:19] * bd808 lunch [21:21:20] * bd808 out for some long errands