[07:31:31] <elukey>	 good morning folks
[07:31:48] <elukey>	 I filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/743351 on Friday to move kafka main codfw to the fixed kafka uid/gid scheme
[07:32:17] <elukey>	 the procedure is very simple, but it requires stopping/starting the kafka daemons 
[07:32:30] <elukey>	 (already executed in kafka test)
[07:32:51] <elukey>	 If you like the idea I can execute it, and then we can think of kafka-main2003's reimage
[07:33:05] <elukey>	 (after the fixed uid/gid it should quick and easy)
[08:53:31] <_joe_>	 gave you a +1
[08:54:53] <elukey>	 ack thanks :) Ok to proceed then? I have some time now
[08:55:18] <_joe_>	 sure
[08:55:36] <_joe_>	 codfw right? the main thing is checking that no purged instance gets stuck
[08:56:24] <elukey>	 yes yes codfw, one broker at the time (leaving space for its recovery etc..)
[08:58:17] <_joe_>	 👍
[09:55:23] <wikibugs>	 10serviceops, 10Data-Engineering, 10Data-Engineering-Kanban, 10observability: Move kafka-jumbo to a fixed uid/gid - https://phabricator.wikimedia.org/T296990 (10BTullis) 05Open→03Stalled Postponing this work until the New Year, based on the feedback from @Jgreen here: T296064#7538028
[09:55:27] <wikibugs>	 10serviceops, 10Data-Engineering, 10observability, 10Patch-For-Review: Move kafka clusters to fixed uid/gid - https://phabricator.wikimedia.org/T296982 (10BTullis)
[10:03:05] <elukey>	 kafka-main codfw done!
[10:03:15] <elukey>	 we can now, in theory, reimage kafka-main2003 anytime
[10:04:57] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/742969 was already merged to avoid formatting /srv
[10:05:47] <_joe_>	 elukey: well done!
[10:12:39] <wikibugs>	 10serviceops, 10Data-Engineering, 10observability, 10Patch-For-Review: Move kafka clusters to fixed uid/gid - https://phabricator.wikimedia.org/T296982 (10elukey)
[10:14:31] <wikibugs>	 10serviceops, 10Patch-For-Review: Upgrade kafka-main nodes to buster - https://phabricator.wikimedia.org/T296641 (10elukey) The kafka-main codfw cluster is running with fixed gid/uid now, next steps:  ` sudo cookbook sre.hosts.reimage --os buster -t T296641 kafka-main2003 sudo cookbook sre.hosts.reimage --os b...
[15:55:07] <wikibugs>	 10serviceops, 10Scap, 10Patch-For-Review, 10Release-Engineering-Team (Doing): Deploy Scap version 4.0.2 - https://phabricator.wikimedia.org/T291095 (10dancy) >>! In T291095#7550012, @gerritbot wrote: > Change 744032 had a related patch set uploaded (by Simone Cuomo; author: Simone Cuomo): > %%%[mediawiki/e...
[21:28:04] <wikibugs>	 10serviceops, 10Infrastructure-Foundations, 10Mail, 10SRE, 10Znuny: OTRS/mail: investigate why "T=remote_smtp_signed: all hosts for 'ticket.wikimedia.org' have been failing for a long time" - https://phabricator.wikimedia.org/T297160 (10Dzahn)
[22:15:17] <wikibugs>	 10serviceops, 10Infrastructure-Foundations, 10Mail, 10SRE, 10Znuny: OTRS/mail: investigate why "T=remote_smtp_signed: all hosts for 'ticket.wikimedia.org' have been failing for a long time" - https://phabricator.wikimedia.org/T297160 (10Legoktm) My understanding per T225623#5253119 is that `@ticket.wikim...