[01:10:46] 10serviceops, 10Observability-Logging: Mutate mmkubernetes k8s fields into ECS fields - https://phabricator.wikimedia.org/T292881 (10colewhite) [01:13:30] 10serviceops, 10SRE, 10Thumbor, 10User-jijiki: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10AntiCompositeNumber) [08:44:58] <_joe_> jelto: helm 3 is so much faster [08:45:02] <_joe_> thanks for working on it [09:02:33] legoktm: thanks for the hint, I added screenshots of the linked dashboards [09:03:07] joe: I'm glad about this side-effect ;) [09:05:00] FYI: I'm going to shutdown kubetcd2005 and temporarily switch it to DRBD (i.e. increased latency) as part of the Ganeti update in codfw (since the virt node on which it currently runs will be reimaged) [09:34:58] 10serviceops, 10Patch-For-Review: Upgrade kafka-main nodes to buster - https://phabricator.wikimedia.org/T296641 (10elukey) Another interesting thing to consider is: ` ===== NODE GROUP =====... [10:12:48] akosiaris: is it okay to keep kubetcd instances in codfw on DRBD for the next ~ two weeks? I've switched kubetcd2005 to DRBD for the initial reimage, but ideally I'd only switch them back to "plain" once all codfw Ganeti nodes are reimaged to buster (since those instances will likely be migrated to new nodes multiple times when reimages progress) [11:51:14] 10serviceops, 10Dumps-Generation: Test php7.4 for dumps generation - https://phabricator.wikimedia.org/T295580 (10ArielGlenn) Checked adds/changes dujmps today, they look good. Same for CirrusSearch dumps and the shorturls dump. I'll check out the remaining "other" (not SQL/XMl) dumps tomorrow. [13:24:29] moritzm: yeah it is. Worse thing that can happen is some alert for higher latencies for the kube-apiserver service. Which is internal and doesn't matter, just keep it in mind if we do see an alert [13:26:43] ack, ok! I'm going to revert them to "plain" once the whole cluster is updated [14:33:33] 10serviceops, 10Patch-For-Review: Upgrade kafka-main nodes to buster - https://phabricator.wikimedia.org/T296641 (10elukey) The kafka-main[12]00[45] nodes running buster don't have, afaics, any specific bits related to buster, so there seems to be no need to do anything else than reimage the nodes. [16:30:37] Hi; I'm probably Doing It Wrong, but trying to push (via ssh) to gitlab is failing because I don't know the hostkey; I've tried " wmf-update-known-hosts-production ." from a checkout of operations/dns (which I think is how I'm meant to make sure I have correct hostkeys), but to no avail. How am I meant to make this work, please? [16:31:02] [I have a hostkey for gitlab1001 which is what's running the service, but AIUI I should be using gitlab not gitlab1001 as my remote] [16:36:22] Emperor: the wmf-update-known-hosts-production takes care only of CNAMEs in the DNS repo and gitlab doesn't have a CNAME but a dedicated IP that is different from the host's one [16:37:55] Emperor: the old canonical way was to add the fingerprints to https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints [16:38:24] also for those hosts that are used by volunteers that might not have our same setup or need the whole setup for all hosts [16:38:32] as youc an see there there is a page for gerrit: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gerrit.wikimedia.org:29418 [16:39:32] so I guess the short term fix could be to add one for gitlab too, but I'll leave it to those setting up the service to decide what's best. We can also tweak wmf-update-known-hosts-production to DTRT here too [16:40:05] volans: that'd be nice (it seems to get every other host's key for me that I've needed thus far :) ) [16:44:45] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) Deploy of puppet role `gitlab::runner` to `gitlab-runner1001` was successful. The runner showed up in GitLab Runner... [17:31:34] 10serviceops, 10Patch-For-Review: Upgrade kafka-main nodes to buster - https://phabricator.wikimedia.org/T296641 (10Ottomata) +1 [17:35:43] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.1.0 - https://phabricator.wikimedia.org/T296867 (10dancy) [17:54:52] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) [23:04:44] 10serviceops, 10Phabricator, 10Release-Engineering-Team: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Hawkeye7) Okay, I have over to using https.