[00:56:55] * bd808 off [08:52:57] arturo: I have a meeting in about an hour, do you want to do the kubernetes control node replacement before or after that? [08:53:36] morning [08:53:54] o/ [08:56:36] morning [09:05:12] taavi: good morning. I'm fine either way [09:05:31] I will just shadow you, so you decide! [09:06:09] ok, let's just do it now [09:32:50] arturo: https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1005440 [09:43:10] arturo: https://phabricator.wikimedia.org/T284656 [09:43:26] taavi: thanks [10:06:21] fyi. I'm going to start shuffling some tasks from the toolforge subprojects to the toolforge org, sorry for the spam [10:11:12] ack [11:43:20] FYI I'm about to drain/reimage cloudvirt1033 [11:46:36] we will see if https://gerrit.wikimedia.org/r/c/operations/puppet/+/1005065 makes any difference (hopefully it does) [12:16:17] I'd like to deploy pdns-recursor security (these are already live on our main DNS recursors and wikidough, wouldn't expect any issues), during the package update/daemon restart there will be a few seconds of unavailability, should I just go ahead or is there anything else to consider? [12:17:22] moritzm: I think that's fine. if you want to be extra safe stopping bird.service is an option too [12:18:15] ok, for the codfw1dev I'll just go ahead and for 1005/1006 I'll stop bird before the update and re-enable it when pdns-rec is updated [12:18:39] sgtm [12:21:39] 1005 was updated that way and seems all fine, will carry on with 1006 in a bit [12:23:23] * dcaro lunch [13:05:32] dcaro: moving to here to not spam the daily channel... what would be the way to start playing with this? can I just build my python server into an image manually, push it to harbor in lima-kilo, and use it in api-gateway? [13:07:27] btw, afaik the apis are not currently exposing their own OAI specs [13:08:30] I would create the boilerplate code first to build in CI and let it push to harbor directly, then use the toolforge_deploy_mr thingie to test the whole setup. If you want to edit the code while testing I would run the python script from within the lima-kilo (as the tool user, just `python script.py` or whatever, probably mounted from my home), so it has access to the APIs and any edits are seen directly [13:08:35] I thought we were :/ [13:08:36] we should [13:13:29] only builds-api does it [13:14:28] so envvars needs it [13:14:33] should be easy [13:17:46] on it [13:28:37] thanks :) [13:39:33] hmm, oapi-gen generated changes that are unrelated to the current patch are always a bit annoying. maybe we could get in the habit of running and commiting those before making the "real" changes in a separate patch? [13:42:57] I think I reimaged cloudvirt1033 with the wrong hiera config and I'll need to reimage again :-( [13:43:35] blancadesal: got an example? [13:45:12] I created a fresh branch off main, then ran `make gen-api` without touching anything else: [13:45:16] https://www.irccloud.com/pastebin/TCx3Nf2B/ [13:45:46] please +1 here: https://gerrit.wikimedia.org/r/1005513 [13:46:00] mostly, it can be annoying for reviewers [13:49:20] arturo: done [13:49:28] dcaro: thanks [13:50:46] blancadesal: refreshing the go.mod and go.sum is needed in case some dependent packages are added/removed when generating the new code (I think it might be updating deps nowadays only, as there's not that much code generated) [13:51:06] I think it's not preventable [13:51:22] we might be able to minimize it somehow to only new packages, not updates [13:52:47] is that the two that you were talking about? (there's nowadays a go.work.sum or similar too :/) [13:53:05] I don't think it's preventable, but if we commit the automatic changes first, then the "real" patch will be cleaner and easier to review, I think [13:54:17] it's two complete files, easy to ignore for me (and probably not deployable by itself, if it added/removed any deps, as it needs the code that uses them to go along with it) [13:55:07] go.mod and go.sum can be ignored, but having automatic changes intermingled with changes related to actual schema changes in gen/api.go and gen/models.go can be annoying sometimes [13:55:28] gen/* should never we changed manually [13:55:31] what do you mean? [13:55:35] i've already seen reviewers comment on automated changes thinking they were real/related to the patch [13:55:36] (it's only generated changes) [13:57:11] i know, but it still feels like clutter to me [13:59:21] okok, so can you send an example patch of what you want? (it's still not 100% clear to me, if it help I can try) [14:02:02] maybe we can flag the files somehow on gitlab (I think there was a way to tell it to hide/unflag generated files somehow) [14:03:49] https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/pipelines/42025 [14:04:02] have there been changes to the pipeline lately? [14:06:26] nope, but it's been a while, let me merge the one I have that has the fgix [14:07:08] blancadesal: done, you can rebase now [14:12:39] I closed it because the autogenerated changed it had were already included in your MR. [14:13:13] oh, sorry :facepalm: [14:13:34] xd [14:14:29] tl;dr nevermind, the idea was just to keep patches to "related changes", whether those are autogenerated or not [14:15:01] maybe in the next one you can show me, there's also one for builds-api that I have not merged yet [14:20:22] arturo: about the ceph errors on cloudvirt1033, is it continuously happening? Do you want some help? [14:33:51] quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/14/diffs [14:35:21] dcaro: done [14:35:28] thanks :) [14:44:28] dcaro: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/80 [14:45:31] 👀 there's only one commit [14:46:12] is it missing another one or something? [14:46:27] no, the idea would be to merge it, then do your real patch [14:47:00] but that merged commit will not be able to be deployed, given that you just changed the api definition no? [14:47:46] there's no change to api definition, no? [14:48:13] oh, I see, so I would do this MR before starting to change anything? [14:48:23] then I change the definition, then regenerate again? [14:48:29] YESS!!! [14:48:39] could that just be one commit in the MR instead? [14:48:58] (as in, first commit update deps, second commit update api def + gen, third commit implementation) [14:49:37] I guess yes? the idea is just for gitlab to not show you the changes mixed together [14:49:53] I say because the deps and such might depend on the time you run the generate/dep update [14:50:08] so even if you create the first MR, by the time you get the second reviewed, you might get more changes [14:50:23] (we are not specially fast reviewing) [14:50:40] that's right. I like your suggestion of 3 focused commits in the same MR [14:51:17] I'll try to do so then on the next MR [14:51:50] 👍 [14:53:07] cool! sorry about the whole nit pickiness of this; i've seen reviewers being confused/annoyed by this (incl myself obviously xd) [14:54:02] no problem, if it help and is easy to do why not [15:04:33] could I get a +1 here? T358023 unless we think there's something else they need to clarify? [15:04:45] https://phabricator.wikimedia.org/T358023 [15:08:37] Hm, I just looked at the cloud-vps roadmap, last updated in 2022, and we have actually implemented many of the things on the roadmap! https://wikitech.wikimedia.org/w/index.php?title=Portal%3ACloud_VPS%2FRoadmap&diff=2152238&oldid=1960679 [15:09:42] Nice to see that glacial progress is still sometimes progress [15:21:38] neat :-) [15:25:21] dcaro: I'm not looking at the ceph problem at the moment [15:25:47] blancadesal: done [15:28:01] blancadesal: I think the should consider Toolforge [15:29:43] they will be primarily generating/storing files it sounds like. isn't that problematic in toolforge? [15:29:56] if it's big files yes [15:29:59] (no real quotas) [15:32:03] quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/24 [15:32:41] done [15:33:11] thanks! [15:34:26] blancadesal: they should clarify. I see a comment `In general no media will be stored on this WMCS instance`. Maybe there is no big storage involved? [15:35:22] arturo, ok, let's see what they say [15:36:19] andrewbogott: nice! should we link that roadmap page from somewhere? I didn't remember it existed :) [15:36:50] also, in line with what is mentioned in the other comment, I have mixed feelings about a project in support of an external wiki like mdwiki [15:38:06] it's listed as a wikimedia thematic organization: https://meta.wikimedia.org/wiki/Wikimedia_thematic_organizations [15:39:43] I would say that means wikimedia recognizes it as helping the movement [15:42:34] the project relates to this: https://meta.wikimedia.org/wiki/Wiki_Project_Med [15:47:31] ok [16:05:57] dhinus: probably we should link to it from someplace but I'm not sure where [16:06:21] I've added a category [16:06:27] not great but better than nothing :) [16:06:43] thx [16:11:54] dcaro, fyi one of the wiki med folks (James Heilman) was on the WMF board of trustees for a while. So those folks are definitely wmf-aligned. [16:12:45] good to know [16:13:05] is it possible that openstack renamed the `Flavor Name` VM attribute to `Flavor` ? [16:13:05] can't tell from their description if it would work in toolforge though *shrug* [16:13:18] yep, not sure either [16:13:23] arturo: not that I know of, what are you seeing? [16:13:47] andrewbogott: a JSON like `'Flavor': 'g3.cores1.ram1.disk20'` from the API [16:13:58] andrewbogott: on the other hand, mdwiki exists purely because they want to host content incompatible with the WMF licensing policy [16:14:01] but expecting `Flavor Name` [16:14:45] Hm, I suspect that some apis return the name and some return the ID, and I wouldn't count on them being consistent. Which api call? [16:15:02] server list [16:16:33] Seems like at least the formatted output has been like that for a while (https://wikitech.wikimedia.org/wiki/Help:Using_OpenStack_APIs#OpenStack_Commandline_Interfaces) [16:17:17] ok [16:17:54] looks to me like the actual api returns a dict for flavor with the various attributes including name and id [16:18:07] lmk if you want me to dig deeper! [16:18:25] ok, thanks. I think I'll just send a patch for the error I'm seeing in the cookbook [16:19:15] andrewbogott: https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1005557 [16:20:00] Finished the first step of the toolforge backlog grooming \o/: https://phabricator.wikimedia.org/project/board/539/ renamed the old 'backlog' to 'ready to be worked on', and the old 'triage' to 'backlog', I'll start moving tasks from jobs/buildservice to 'ready to be worked on' and then start going through the 'triage' column [16:23:10] dcaro: I did not get a lot of phab spam because of the reorg. Not sure if a good or a bad thing :-) [16:23:56] I'm going to start moving those tasks now xd, I just finished with the "old" triaging xd [16:24:28] andrewbogott: thanks for pointing out how small improvements add up over time. :) We really do move things forward here nearly constantly, but because we also have high expectations for our work product and big desires it can be easy to lose track of how far we have come. [16:27:37] looking at the list, object storage stood out as something I don't remember promoting much. Then I found https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#Toolforge and realized why I haven't really played with it any myself. [16:28:59] yep, there's the idea going around of creating a service to be able to provision buckets on-demand for toolforge users so they don't have to create a whole vps project and such [16:29:21] (similar for travis...trello...trove!) [16:30:28] Hm... we could also just automatically create a keystone tenant to correspond to every ldap user [16:30:42] * andrewbogott tries to think of what would be bad about that [16:30:59] That would be very nice I think. I remember Magnus had some use cases for blob storage. Also tools like the image manipulation ones that Tim recently brought to k8s could use a way to avoid NFS [16:31:18] you mean for every tool, right? [16:32:01] a tool is a kind of ldap user [16:32:07] but yeah, we could restrict it to that [16:34:27] anyone want to come to check-in? [16:34:34] could we let maintain-kubeusers to do that provision? [16:34:41] coming [17:03:11] * arturo offline [17:15:17] * dcaro off [19:11:48] * bd808 lunch