[07:04:48] morning! [07:05:04] greetings [07:08:37] morning [07:08:58] welcome back volans \o/ [07:09:24] thanks! [07:17:25] This is cool https://docs.gitlab.com/user/project/merge_requests/stacked_diffs/ [07:17:54] https://usercontent.irccloud-cdn.com/file/TSUaJQwD/Screenshot%20From%202026-09-07%2009-11-12.png [07:50:41] dhinu.s: thanks for sending the hypothesis updates :) [08:37:00] dropping a couple of reviews pending kind souls to take a look https://gerrit.wikimedia.org/r/c/operations/puppet/+/1331487 https://gitlab.wikimedia.org/toolforge-repos/cloudvps-quota/-/merge_requests/3 [08:37:17] morning [08:38:00] that sounds a bit like a classified ad, but you get the idea [08:38:01] dcaro: thilp did most of the work, I only posted it to Asana :) [08:38:58] Better than clickbait xd "Look at this review, you wouldn't believe it!" [08:40:11] godog: did you try it on the nfs servers? do you have an example generated promfile or similar? [08:40:45] dcaro: good point yeah, I did try it and will post an example now [08:44:59] sigh I actually need to tweak the script types a little because clouddumps1001 has not been reimaged yet [08:45:05] so the script works on 1002 but not 1001 [08:52:38] {{done}} [08:52:59] ack 👀 [09:19:12] godog: +1d both [09:19:55] godog: for the cloudvps-quotas, you can deploy the new code to test before merging if you want [09:20:00] (build + restart webservice) [09:21:56] dcaro: thank you! appreciate it, yes cloudvps-quota supports a local testing mode with mock clients, I used that for testing [09:22:34] awesome, extra +1 then :) [09:33:26] \o/ \o/ \o/ [11:20:29] * dcaro lunch [12:58:59] quick review quick review https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/96 [12:59:20] I repeat myself, myself I repeat xd [13:01:43] gah I meant to approve and I merged [13:01:45] anyways, LGTM [13:01:56] obviously [13:02:08] thanks! [13:02:12] np [13:13:59] is anyone restarting cloudvirt1067? (or similar) [13:14:25] alerts going off [13:14:50] oh, 67 and 74 [13:15:00] yes cloudvirt10(6[567]|7[234] is filippo [13:15:07] ack [13:15:14] yes that's me [13:15:16] I'll silence [13:15:20] https://phabricator.wikimedia.org/T435921 [13:16:17] {{done}} [13:17:17] ack, was there a sal entry somewhere or something I missed? [13:18:51] no I just ran puppet with a new role, I'll log it too [13:19:37] ack, np, just wondering if I'm looking at the right places [13:20:57] totally, you are looking at the right places and I'm going a little against the grain [13:21:22] my understanding is that normally we reimage in this case, so that comes with silences for example [13:21:32] this case == "apply a new role" [13:23:07] ack, so there's some silences missing in the cookbook? [13:23:44] oh, sorry, usually you would reimage and that would silence, but that's not the current action taken (thus against the grain?) [13:23:52] yes exactly [13:24:34] ack [14:56:07] does anyone know why all the toolforge k8s etcd nodes have all the other nodes in their puppet cert SANs? [14:58:42] hmm, not from the top of my head [14:59:47] might be related to upgrade procedures? (using the certs to auth or something) [15:06:08] Raymond_Ndibe: were you able to do any investigation regarding the service manifest usage and migration plan? [15:35:03] taavi: could be a k8s thing? you can probably compare in prod a k8s cluster vs the other ones [15:45:54] hmpf... the deploy on toolsbeta failed twice with "branch not found bump_components-api", and now it's passing without me having done anything except ran the command to run the functional tests directly on the bastion [15:46:00] has that happened before? [15:46:07] https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1413 [15:55:49] dhinus: for T437208 I'm thinking whether we should send a follow-up to cloud-announce with a bit more detail on what tool authors need to do [15:55:50] T437208: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208 [15:56:34] taavi: yes I was thinking the same, I think we should at least mention they can start using the new hostnames, like we did when x3 was introduced [15:57:12] I've updated the docs (unless I'm forgetting something!), but that doesn't help when the original mail didn't point to any docs :P [15:57:49] thanks for updating the docs btw! I can send the email but I'm not sure if I'll manage before tomorrow [15:58:47] the timing of this was a bit surprising, honestly I would have expected a heads-up more then 4 days before :) [15:59:03] please do! tomorrow morning is probably just as fine as this evening would be tbh [16:01:40] anything that you would change compared to this old one we sent for x3? https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/2IEY34ZAUT3VHBBHZHWMQU3U33PI5ST5/ [16:02:40] > For more information, refer to the News page on Wikitech. [16:02:51] are we creating a News page for this migration as well? probably not a bad idea? [16:03:47] probably should mention somewhere about the tables that'll be on both the main and extension clusters. I think that's at least the page table and maybe something else I'm forgetting? [16:22:31] yeah a News page would be good, but I'm travelling tomorrow so I will have very limited time :/ when is the deadline for Tech News? [16:25:26] * dcaro off [16:25:41] re: pages in both clusters, two are mentioned in T398709 (page and redirect) but I'm not sure that's still up to date [16:25:42] T398709: FY2025-26 WE 6.4.1: Move links tables of commons to a dedicated cluster - https://phabricator.wikimedia.org/T398709 [16:42:56] I have to log off, I'll try to pick this up tomorrow during my travels (wifi/battery permitting :P)