[07:54:16] greetings [08:01:55] morning! [08:37:43] morning [08:56:28] https://docs.python.org/ seems to be down for me :/ [08:57:12] confirmed, I'm getting 503 [09:00:41] :hughops: [09:02:06] dhinus: just noticed that you are reviewing https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/380, but raymond did a pass too (and I just updated with the addressed comments), sorry for the mess. I'd appreciate though if you continue the review, would be good to merge/deploy today (and raymond will not be around until the afternoon) [09:03:27] dcaro: no problem, but yes I noticed the same, we need to find a way to signal when we're starting a review :) maybe adding oneself to "reviewer" in gitlab? [09:03:50] I think in this case as we're trying to push things out quickly it's actually fine having more than one reviewer [09:03:57] moving the task "should" be enough, but yep, that is more visible [09:04:07] I'm catching up with raymond's comments and will re-review [09:04:15] (I only started the review yesterday anyway) [09:06:19] ack, thanks [09:24:54] mmhh the prometheus-metricsinfra (at least?) datasource seems to be busted, this doesn't work for me https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board [09:25:59] toolforge dashboards too seem to be busted [09:28:00] taavi: does that ring a bell ^ ? [09:32:33] godog: not immediately [09:33:07] the grafana data sources tab is completely empty? [09:35:44] grafana was seemingly unattended-upgraded from 13.2.0 to 13.2.3 last night [09:37:30] I see a list of datasources but yeah each datasource is empty e.g. https://grafana-rw.wmcloud.org/connections/datasources/edit/PF356435408BB301C?orgId=1 [09:39:03] Oct 01 09:36:11 metricsinfra-grafana-2 grafana[3139952]: logger=plugin.backgroundinstaller t=2026-10-01T09:36:11.895913252Z level=error msg="Failed to install plugin" pluginId=prometheus version= error="unlinkat /var/lib/grafana/plugins/prometheus/4256.js.map: permission denied" [09:42:40] godog: fixed the permissions issue and it seems working now at least to me [09:43:34] taavi: confirmed! thank you very much [09:44:06] a little surprised I must say because IME grafana upgrades have been painless :( [10:19:35] dhinus: the logs-cli mrs are also ready (in the ready for review column) [10:19:59] dcaro: yes, I'll move to those in a sec... double checking one thing in !380 then that one should be ready to go [10:20:11] yep yep, no rush, take your time :) [10:23:33] on my side, I think the only one waiting for review is https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 [11:01:08] yep, I'm checking it out [11:20:57] does anyone have opinions on whether the requestctl web interface VM should live in cloudinfra or project-proxy? [12:02:38] if anything else than project-proxy uses or we think might use it (cookbooks?), better cloudinfra, no strong opinion either way though [12:52:27] dhinus: reviewed, there's some minor questions/nits that you can ignore if they don't make sense or the answer is "it's ok as it is" [12:57:47] dcaro: thanks, will look in a minute. I'll also be in the pairing meeting but I'll be 10 mins late [12:57:55] ack [14:35:39] there is an outstanding change for cloudcephosd1043's switch port. Removing the `cloud-private-c8-eqiad` vlan. Does that host need it or not? https://www.irccloud.com/pastebin/beMO4YqE/ [14:36:23] caused by some automation, dunno what triggered it though, maybe a re-image? https://netbox.wikimedia.org/dcim/interfaces/18846/changelog/ [14:57:01] XioNoX: I suspect that might be https://phabricator.wikimedia.org/T429387 cc andrewbogott (we're in a meeting) [14:59:21] andrewbogott: https://phabricator.wikimedia.org/P96544 [14:59:51] so this indicates that CAPI itself does actually store state about clusters? and doesn't that imply that we can't just simply throw the cluster away and build a new one when we want to upgrade it? [15:02:19] possible but I haven't had any ill effects from rebuilding. I'll do some tests. [15:03:15] The answer about certs is: the commercial companies that use magnum-cluster-api say that you need to manually renew your cluster every 12 months to keep certs up to date :/ So we need monitoring for that for sure! [15:31:42] dhinus: Raymond_Ndibe this stack is complete https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/12, I'll add functional tests for all that soon, but they can be reviewed, merged and deployed already [15:33:49] dcaro: nice, I think we can aim to review all the stack between today and tomorrow, and you can merge it on monday, wdyt? [15:33:58] 👍 [15:34:36] they are pretty simple mrs yep, feel free to update the patches if the fixes are minor issues/etc. [15:34:51] sounds good. Raymond_Ndibe: if you're working until later today, I'll let you do a first round of review today, and I can pick up tomorrow whatever's left [15:41:52] I think there's some issue with the ci images poetry version https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/9, we should probably update them soon-ish [16:32:44] dhinus: I found an "issue" with the alerts confiiguration. Currently we do not allow a toolforge config to not have any components configured, so validation will fail for tools that only want the alerts [16:33:04] (I was testing the docs process with an empty tool) [16:35:58] we might want to allow it now, to support this (and future) settings [16:36:14] dcaro: yes, I realized that but I think it's ok to require them to define the component as well... could be a small issue for people using just a webservice [16:36:31] or using pre-built images [16:36:32] in the discord call we also said "you need to be using toolforge config" [16:36:54] it does apply to all jobs though [16:36:59] be them in the config or not [16:37:01] yes [16:37:07] it's just a matter of config... [16:37:12] xd [16:37:46] that's also why I'd like to get the webservice support in the config as soon as possible :) [16:38:02] so we could tell people to define their webservice there, and then enable the alert [16:38:22] most webservices (I think) are pre-built images though [16:39:18] well, true, but I think it's fair to say "the alert feature is beta and only for buildservice images..." [16:39:47] might actually push people to explore how hard it is to migrate their webservice to a build image! [16:39:56] ack, I'm just making sure that it was intentional [16:40:08] yes, maybe not intentional, but I was aware of it :) [16:40:20] I added a note here but I should probably make it more explicit: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Alerts#Known_limitations [16:42:31] tomorrow I plan to merge & deploy the MRs you already reviews. then T439554 will be early next week, so we can discuss this again on monday. [16:42:32] T439554: Read the opted-in tools from components-api - https://phabricator.wikimedia.org/T439554 [16:42:50] *already rewiewed [16:43:40] do you need anything else before I go away? [16:45:04] btw. I did some minor changes to the wiki page, feel free to revert/etc. [16:45:40] no, I just checked your reviews to see if anything was not clear, but I everything was and I addressed all your comments [16:45:56] ack, in that case, cya on monday! \o [16:45:58] * dcaro off [16:46:01] thanks for looking at the wiki page! [16:46:09] and have a good weekend :) [16:46:23] np, it looks good (the page) [16:50:13] * dhinus off