[07:57:21] how can I read about kerberos setup on apt.wikimedia.org, or does the analytics install apply? [07:59:02] jynus: o/ can you add more info? kerberos + apt.wikimedia is new for me [07:59:40] elukey, log into apt.wikimedia.org or install1003.wikimedia.org, you get the same banner than on analytics hosts [07:59:46] *as [08:00:03] (or at least I do) [08:00:12] that's being fixed: https://gerrit.wikimedia.org/r/c/operations/puppet/+/701512 [08:00:32] ah, I see [08:00:38] waiting for reviewers, but should be merged in the next days [08:01:13] "which are mostly part of the [08:01:13] Kerberos realm for remotely executed commands" -> that explains my doubts, thanks [08:02:04] I'll review the code change but we may have to think about a more maintainable solution for the defaults, we are adding more and more analytics-kerberos-specific settings to hiera that will be difficult to maintain [08:02:37] analytics slowly taking over SRE [08:03:21] the opposite, SRE taking over kerberos :P [08:07:24] :) [08:08:31] don't worry, the time for plugging Kerberos into some ML setup will come :-) [08:09:56] moritzm: commented, I'd say let's wait for Andrew,Razzi's option [08:10:31] the kerberos nightmare will be real again when the kubernetes clusters will need to authenticate via kerberos [08:10:40] https://engineering.linkedin.com/blog/2020/open-sourcing-kube2hadoop [08:10:43] * elukey cries in a corner [08:11:24] ouch ouch ouch [08:30:02] elukey: I posted that to you like 2 months ago [08:30:12] and you just chose to forget [08:31:06] joe: I was well aware of it :) [10:28:39] on a new machine, Failed to add the host to the list of known hosts .. /.ssh/known_hosts.d/wmf-prod. Looks like I have to manually create .ssh/known_hosts.d/, ecxpected? [14:27:34] Services switchover is starting now, can follow along in -operations [16:57:35] I just sent a calendar invite for tomorrow's switchover to sre@, mostly for awareness. thanks to sobanski for the suggestion :) [17:53:03] kormat: I'll deploy the --shard param today in about 1-2h after my meeting [17:53:29] It'll be inactive code at first. Will wait for you to decide together how/when to use it [21:12:22] I'm going to start a dry run and then live test of the MediaWiki switchover in a few minutes, `sudo -i tmux attach -rt switchdc` if you want to watch/follow along [21:27:32] sigh, x2 again [21:28:32] it matches A:db-role-master and A:db-core, except it's no longer in the spicerack list so it throws an error on the mismatch [22:30:42] ok, now that x2 is fixed via live hack, I'm going to actually do the live-test now [22:30:57] which will "switch" us from codfw to eqiad