[07:34:38] hello folks! [07:34:56] the mw deployments seem stuck, I think a new version of https://phabricator.wikimedia.org/T390251 [07:35:09] if anybody has time ping me, I am on #operations debugging [08:00:25] Daniel reverted their patch, no idea for future deployments [08:00:29] I added some info to the task [08:41:21] update - we are still stuck with deployments, the revert patch makes CI to fail, we decided to re-do the deployment. The image push went fine, but now helm seems to have emitted a big error (still unclear why, waiting for a paste) [08:43:12] Fyi, there are those 2 warnings when running the DNS cookbook [08:43:12] 2025-04-07 08:41:41,000 [WARNING] Device an-worker1202 of IP 10.65.3.18/16 with DNS name an-worker1202.mgmt.eqiad.wmnet not in devices, skipping. [08:43:12] 2025-04-07 08:41:41,031 [WARNING] Device sretest2002 of IP 10.193.3.49/16 with DNS name sretest2002.mgmt.codfw.wmnet not in devices, skipping. [10:40:55] status: deployments seem to be unblocked [13:41:34] sukhe: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/bird/anycast.pp#26 is there a specific reason for this check? for my use case (wmcs cloudlb) it'd be nice to add v6 addresses one by one instead of doing everything at once [13:46:03] taavi: the reason was that in production, there is only one service doing anycast v6 so far (Wikimedia DNS) and so it met the set of requirements at that time (that's still true today, everything is on v4 anycast) [13:46:52] in theory, you can provide the v6 address (address_ipv6) and not set do_ipv6? [13:47:56] if there is another use case, we can certainly relax this requirement. but the original intent was someone enabling this by mistake (it is in the docs as an example) and not setting the v6 anycast address [13:48:57] "everything is on v4 anycast" -> internal recursors, logging, NTP, is all v4, except Wikimedia DNS, which is both v4 and v6 [13:49:10] if i'm reading the correctly, setting up address_ipv6 without do_ipv6 will not really do anything? [13:50:16] no, it won't since v6 is conditional on that. I read the above "add v6 addresses one by one" as in that you want to add the config, not turn on the actual configs [13:50:26] in which case, yes, that will need to be updated [13:55:49] I am fine if you change it to a warning, as long as we have something there [14:01:04] sukhe: hmm, do you think a warning/error when none of the services have v6 addresses is enough? [14:03:46] taavi: not sure what you mean? "none of the services" in wmcs? [14:04:50] i mean do you want a warning when there's do_ipv6 => true but no services with address_ipv6, or when there's at least one service without address_ipv6? [14:09:37] since we are iterating over advertise_vips and the conditional is on do_ipv6, https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/bird/anycast.pp#27 fail() to warning() here should be enough? that way you can achieve your intended purpose? [17:56:46] this is possibly a very silly question but I will ask because I have never dealt with gitlab before and have only done gerrit. I guess, well deserved for breaking an existing workflow that was just fine for me :) [17:57:15] do I need to do something _special_ for it to apply patches in debian/? [17:57:46] it -> wmfdebci or any of the various abstractions underneath it :) [17:59:02] I just can't see what though and I promise I have RTFMed whatever I could. I also think this is something that sholud be already a default so I am having a hard time believing that I need to toggle this. [17:59:13] https://gitlab.wikimedia.org/repos/sre/openssl-ech/-/jobs/479286/raw [18:00:29] https://gitlab.wikimedia.org/repos/sre/openssl-ech/-/tree/bookworm-wikimedia/debian/patches?ref_type=heads also looks good, plus, the same thing builds with patches (gasp!) on the build host [18:00:35] any thoughts? [19:22:46] <_joe_> sukhe: in theory, you shouldn't need to do anything. it runs dpkg-buildpackage -uc -b [19:23:11] <_joe_> so if that does apply patches locally, so it should in CI [19:24:21] _joe_: yeah I am giving up and will file a task. not sure where to go from here anyway :) [19:24:44] <_joe_> sukhe: does it work on your machine? As in, is it installing the patches? [19:24:51] <_joe_> Emperor: ^^ [19:24:57] <_joe_> (for tomorrow morning ofc) [19:25:10] (tried a brand new repo, same issue, builds fine on build host with dpkg-buildpackage but doesn't pick up the patches on gitlab) [19:25:19] _joe_: yeah it builds fine on the build host [19:25:23] exact same repo [19:25:52] Empero.r: not urgent, let's look at it tomorrow [19:47:12] heads-up that I merged a change to cumin aliases: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1134078 . It looks like it's working for me. ping me if you notice any problems and I'm happy to roll back