[01:11:54] * bd808 off [09:31:03] morning [09:31:33] o/ [09:39:55] quick review appreciated: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/61 [09:48:52] * taavi grumbles about gitlab's 2fa [09:49:26] done [09:51:47] thanks! [09:55:38] dhinus: if you have a moment, I have a patch to make alertmanager be magically less spammy :D https://gerrit.wikimedia.org/r/c/operations/puppet/+/977741 [09:57:28] looking [10:04:09] +1'd [10:04:55] thanks! [13:12:50] btullis: do you have a moment to +1 https://gerrit.wikimedia.org/r/c/operations/puppet/+/976688? I'd like to move wiki replica traffic to use cloudlbs [14:08:02] taavi: Done, thanks. When do you expect the switch to happen. [14:08:05] ? [14:11:25] btullis: I was thinking of doing it, like, now unless you had any concerns about it [14:12:07] taavi: That works for me. Should we make an announcement anywhere, in case it blows up? [14:28:26] so far everything seems stable [14:32:43] dhinus: since we're de-meeting'd, want to upgrade some things? Or are you already in the middle of that? [14:40:46] I'm in the middle of figuring out KubeCon bookings/approvals :P I hope that will be sorted in ~30 mins [14:40:57] I haven't started any upgrade yet [14:41:10] ok! [15:13:26] jbond: I think I missed a ping from you about cloud-vps puppet upgrades last week. Did that get taken care of? Or can I help with that today? [15:15:15] andrewbogott: taavi: has been exploring that [15:15:41] ok -- please ping if there are things I can do! iirc I was in the middle of learning how to build a puppet7 standalone server, I'll have another stab at that [15:17:19] there's a WIP cookbook to migrate data from puppet5 to puppet7, I need to follow up on the reviews of that [15:19:54] taavi: what kind of data needs migrating? [15:20:21] andrewbogott: the private CA material, and local commits in puppet.git and labs/private.git [15:20:52] Oh right, it's in a different place now [15:21:40] yeah, I'm still not sure if they changed anything other than the location. there's a magic 'import' command but it's not entirely clear to me what it does [16:44:46] I wanted to ask if anyone is familar enough with dotNet or understands how to help Hawkeye7 on this ticket: https://phabricator.wikimedia.org/T311466. It would be nice to have a tool to point to that has completed the transition to buildpacks for the other mono/dotnet users. [16:46:50] balloons: my $0.02 is that Hawkeye7 is being deliberately obstructionist. "I have multiple C# projects in the one git repository. It was never intended to be built in this manner." are both true statements, but not practical reasons that change is impossible. [16:49:47] It seems they made some progress in the later comment after trying to get it to work? https://phabricator.wikimedia.org/T311466#9358026 [16:52:43] * bd808 makes a suggestion on the phab task [16:55:26] some time ago as a test, I built a hello world mono app with build service by installing the mono-devel package via the Aptfile. The fact that we don't have a mono buildpack shouldn't be a blocker [16:55:59] oh, we haven't put a mono pack in the stack at all? [16:56:22] is the current stack easily inspectable? [16:57:03] this is the builder we're using: https://devcenter.heroku.com/articles/heroku-22-stack#language-runtimes [16:57:27] + an apt buildpack that we're injecting manually [17:00:25] blancadesal: is your mono POC repo somewhere folks can look at it? [17:01:56] We really could use a mechanism for injecting a custom buildpack into the stack per project. [17:03:19] Or just back up and allow bring-your-own containers since we are not using buildpacks to maintain backwards compat with our legacy container systems anyway (ldap connections, runtime users, runtime paths, etc) [17:03:28] nope, it was a quick and dirty test on my remote server for thishttps://phabricator.wikimedia.org/T305780. I can make it available, but not this very moment because dinner, etc [17:04:37] *nod* the dual runtime container angle makes sense. [17:07:24] Figuring out what version(s) of dotnet can be installed via the Aptfile magic and showing that to Hawkeye7 would be helpful I think. This would likely leave things pretty DIY for building the code and mounting it into runtime container, but it would be better than nothing. [17:08:13] Vort was also recently successful with this mono-based tool migration https://phabricator.wikimedia.org/T320168 [17:11:51] bd808: noted, I'll create some docs and a proper test tool tomorrow [17:12:18] blancadesal, for my try I used paketo as they do offer a dotnet buildpack. Given everyone on grid is on an ancient version of mono, if apt installed mono is the limitation I'm presuming that it's not anymore of a limitation than exists on the grid. [17:12:34] At the same time, I know the upstream dotnet tooling was desired [17:13:19] Is it possible to use a paketo buildpack with the buildservice? [17:13:52] (I'm presuming the issue would be in compatibility with how heroku lays out there buildpacks and what our workflow would expect) [17:14:10] I think that would require that we allow choosing the stack somehow. [17:14:50] the buildpack part is a standard, but the base image varies by stack and likely affects which buildpacks can be used effectively. [17:21:17] we don't have a bookworm based mono container in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/toollabs-images/+/refs/heads/master yet. Maybe we should just make one and set it up to use the packages from ? [17:22:32] That will not solve the build service issues with mono/dotnet, but it may unblock Hawkeye7 [17:25:34] * bd808 re-reads ticket's history [17:28:34] balloons: https://elements.heroku.com/buildpacks/jincod/dotnetcore-buildpack looks like the buildpack that should work with our stack. I don't think we have built an equivalent of `heroku buildpacks:set` however that would let anyone change the stack for their tool. [17:28:53] afaik, the paketo buildpacks are not compatible with the heroku builder image [17:30:53] heroku (as a platform) allows third-party buildpacks, but for the beta we decided to go with just the classic stack. We still haven't decided how we want to move forward with allowing other builders/stacks [17:35:43] blancadesal, understood. So https://elements.heroku.com/buildpacks/jincod/dotnetcore-buildpack is a third-party/unofficial buildpack. Can you share a little what technical work exists in supporting the other buildpacks in heroku marketplace? [17:37:11] taavi: dcaro: just going through some old patches and wonder if this is of any use: [17:37:14] https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/663826/1/spicerack/openstack.py [17:38:11] bd808, for the bookworm mono container, it would be nice to see if we could ensure mono tools only have to migrate once, and to buildpacks. But yes, given we now have a shutdown timeline, it's nice to know options exist that allow a migration off the grid, even if it's not a buildpack [17:42:45] jbond: not really, the enc api now needs openstack keystone tokens for authentication so the module would need to be reworked to support that. we also have a module in wmcs-cookbooks.git now which wraps around the CLI too [17:44:15] balloons: Is that a vote for me (or someone else) to do work on a different mono base container or not? This is niche language stuff in our community today, but potentially because we have never given the runtime more than grudging support. [17:44:21] balloons: support for third-party buildpacks and multi-buildpack builds are platform-specific and not implemented by the buildpacks themselves. As bd808 pointed out, we don't have an equivalent of `heroku buildpacks:set`, or the ability to choose a different builder image. We do allow adding an apt buildpack and/or a nodejs buildpack in addition to a the primary language runtime, but that's a hack. We could [17:44:21] inject other buildpacks in the same way, but it'd be better if we though of a more sustainable and scalable way to expand the buildpack offering beyond what we have now [17:44:21] [17:45:08] this was something considered out of scope for the beta, though [17:45:17] taavi: i figuered it was probably outdated thanks for confirming :) [17:45:49] balloons: (not sure that answers your question?) [17:46:13] We sort of pushed past the beta into general release with the grid shutdown timeline announcement. I feel like this has not been fully explored. [17:46:38] The OS EOL is the tail wagging the dog right now. [17:47:32] indeed, most of this had decision-request-to-be status 😅 [17:50:43] I think I understand the gist of what everyone is saying. I'll add a topic for our team call tomorrow to discuss further and if we want to take any action. Thanks for the discussion! I know it's EoD for some of you [17:51:02] 👍 [17:52:12] bd808, for now, no action needed. But let's discuss specifically how to support the mono migrations. I believe many have occurred without us resolving the tickets raised [17:52:32] That can be tomorrow in our team meeting [17:56:49] "many" is likely just the "mbh" tool -- https://k8s-status.toolforge.org/images/docker-registry.tools.wmflabs.org/toolforge-mono68-sssd-base:latest/ [17:58:02] But we may only have 5-6 dotnet tools in the first place