[07:10:14] 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10Stevemunene) on the datahub charts we need to replace the jaas configmap with an oidc setup For the env variables we can avail them via ` {{- if .Values.auth.... [09:10:51] 10SRE-tools, 10DC-Ops, 10Infrastructure-Foundations: sre.hosts.reimage: fails to get uptime in debian installer - https://phabricator.wikimedia.org/T342345 (10Fabfur) Thanks @jbond I confirm the installer now can correctly install the base system without any error on lvs1016! The "minor annoyance" now is th... [09:14:00] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10collaboration-services, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) OIDC is enabled instance-wide now. >>! In T320390#9052117, @dancy wrote: > In https://gitlab.wikimedia.org/repos/releng/g... [09:17:51] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10collaboration-services, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10CodeReviewBot) jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/38 use provider openid_con... [10:11:27] 10SRE-tools, 10DC-Ops, 10Infrastructure-Foundations: sre.hosts.reimage: fails to get uptime in debian installer - https://phabricator.wikimedia.org/T342345 (10Volans) From the test made with @Fabfur I'm convinced we're hitting the timeout of spicerack checking the uptime, it runs: ` transports.Command("cat /... [10:50:27] 10SRE-tools, 10DC-Ops, 10Infrastructure-Foundations: sre.hosts.reimage: fails to get uptime in debian installer - https://phabricator.wikimedia.org/T342345 (10jbond) nice hopefully a simple fix >>! In T342345#9054715, @Volans wrote: > After d-i ssh is normally quick, so not sure if it's worth investigating.... [12:20:57] hi all im feeling quite light headed, tierd and dizzy. i think the sun over the last few days has caught up to me. im going to go lie down for a bit. i dont think ill be around for the meting but i sohuld be around later this evening if anyone wants anything [12:21:14] feel better jbond and stay hydrated [12:21:27] Hope you feel better in a bit jbond [12:22:10] thanks and definetly cdanis [12:22:21] take care, don't worry about the meeting [12:27:09] jbond: get well! [14:49:40] hi folks. possibly missing an understanding of the hook that triggers the debmonitor update but: https://debmonitor.wikimedia.org/packages/gdnsd [14:50:03] we are running 3.99.0~alpha2 already but debmonitor is not reflecting this [14:50:03] sukhe: what's up? [14:50:24] how did you upgrade it? do you have the output of any of the upgrades not reflected? [14:50:39] volans: apt update and then debdeploy [14:50:40] *but* [14:50:50] it says they are all on 3.99.0 [14:51:04] is that not correct? [14:51:31] previously to test it out, it wasn't in reprepro and we were doing dpkg -i [14:51:39] volans: it says that on the debmonitor page you mean or in general? [14:51:48] on https://debmonitor.wikimedia.org/packages/gdnsd [14:51:53] Version: 3.99.0~alpha2-1 14 [14:52:01] all 14 hosts have that version [14:52:11] right, but why is it saying 3.8.0-1~wmf2 upgrade [14:52:13] on some of them? [14:52:17] for some it shows 3.8.0-1~wmf2 as upgadable to [14:52:35] yeah that's what I was curious about [14:52:46] is if I am misunderstanding the "upgradable" part here [14:52:51] was apt-get update run? [14:52:55] yep [14:52:57] twice now [14:53:05] and policy on the host also says 3.99 is the right candidate [14:53:11] also additional check for my own sanity [14:53:12] dpkg --compare-versions 3.99.0~alpha2 gt 3.8.0 && echo true [14:55:28] I've run debmonitor-client on dns1004 to test and it disappeared there, so I'm wonderign if it was a race or it's a bug [14:55:31] I'll have to check [14:56:17] np, definitely not urgent fwiw [14:56:29] also happy to open a task but thought I should mention this here if that helps [14:56:32] thanks! [14:56:36] anyway it will go away in <24h with the daily run of debmonitor "full" to reconcile [14:56:54] ok that works [14:57:14] I guess I wanted to flag it because I noticed that the hosts that were (manually) updated to 3.99.0~alpha2 [14:57:23] debmonitor was saying that 3.8.0 is the higher version [14:57:45] wait a sec [14:57:57] the hosts where it shows the issue were treated differently? [14:58:06] let me check [14:58:24] These hosts are already up-to-date: dns[2004,2006].wikimedia.org (2 hosts) [14:58:31] so 2006 was different but so was 2004 [14:58:34] but 2004 doesn't show upgradable [14:59:05] These hosts are already up-to-date: dns3001.wikimedia.org (1 hosts) [14:59:11] ^ treated differently [14:59:15] different being dpkg -i :) [14:59:27] yeah I am guessing this muddied the waters somehow [14:59:48] anyway, this is not worth spending a lot of time into so if we see it again during the bookworm upgrade (and thus many packages to follow), I will note it here [14:59:51] I don't recall off the top of my head if dpkg -i bypasses the hook [15:00:09] any apt operation should trigger the hook [15:00:28] yeah so I thought and did an apt update on all 14 after too [15:00:35] see /etc/apt/apt.conf.d/10debmonitor [15:00:39] but I suspect dpkg -i might be the issue here (and which is why I haven't seen this before) [15:04:08] XioNoX: I wanted to talk about upgrading anycast-healthchecker from 0.8.2 (what we are running in prod) to 0.9.1 [15:04:51] yes it's most likely that (dpkg -i) as from a quick google I think the dpkg hook doesn't give you the package name so it would be useless for debmonitor, although we could make a full run in case dpkg is used [15:05:10] volans: ok fair enough [15:05:15] so not worth looking into right now I think [15:05:34] we could run it only for install/remove/purge [15:06:08] I'll check wih moritz on his thoughts [15:06:41] sukhe: Ar.zhel is out this week you might want to fallback to topranks maybe ;) [15:06:59] oh ok [15:07:12] yep happy to discuss [15:07:12] thanks, I will ping him when he is back [15:07:20] topranks: sure :) [15:07:48] I'm not overly familiar with the service, it's fairly simple from what I recall [15:09:45] yeah just the healthcheck services to generate the IPs to advertise [15:10:10] I guess I will collect my thoughts in a task [15:10:28] but looking to upgrade to 0.9.1 (https://github.com/unixsurfer/anycast_healthchecker/blob/master/ChangeLog) as part of the bookworm upgrade work we are doing [15:10:31] I'm assuming 0.9.1 has some new functionality that would be useful? [15:11:05] yeah two things I liked so far: [15:11:07] - * Exit if configuration file passed with '-f' doesn't exist [15:11:13] which is more on the ops side [15:11:15] but more: [15:11:18] https://github.com/unixsurfer/anycast_healthchecker/commit/c7af5e67eea51344429621de6af7b36e6cbfad9d [15:12:09] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10collaboration-services, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/38 use provider openid_con... [15:12:33] and I think in general, 0.8.2 is quite old and .9 is more recent, so might as well close the gap sooner than later [15:12:39] but anyway I write this up in a task [15:12:45] and add you and XioNoX [15:12:48] thanks! [15:14:21] cool sounds good [16:50:13] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (FY2023/2024-Q1): Allow wmcs cookbooks running on cloudcuminXXXX to write to the SAL - https://phabricator.wikimedia.org/T325756 (10Andrew) I don't think this is something we should implement right away, but I wonder if al... [17:01:14] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (FY2023/2024-Q1): Allow wmcs cookbooks running on cloudcuminXXXX to write to the SAL - https://phabricator.wikimedia.org/T325756 (10dcaro) From a conversation in a meet, in order to keep the ability to log messages when ru... [20:15:13] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team (FY2023/2024-Q1): tcpircbot: enable logging to #wikimedia-cloud-feed - https://phabricator.wikimedia.org/T342666 (10bd808) `lang=irc [20:09] < bd808> flags #wikimedia-cloud-feed logmsgbot_cloud +V [20:09] -ChanServ- logmsgbot_cloud is not r...