[08:55:14] 10serviceops, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-RhinosF1: cross-validate-accounts: Malformed membership for ops user ..., has additional group(s): {'deployment-ci-admins'} - https://phabricator.wikimedia.org/T298815 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff... [08:55:31] 10serviceops, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-RhinosF1: cross-validate-accounts: Malformed membership for ops user ..., has additional group(s): {'deployment-ci-admins'} - https://phabricator.wikimedia.org/T298815 (10Joe) Sorry I didn't realize I needed to add the group to the cros... [09:28:12] 10serviceops, 10Patch-For-Review: Upgrade kafka-main nodes to buster - https://phabricator.wikimedia.org/T296641 (10elukey) [09:44:09] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) GitLab Runner in `eqiad` and `codfw` export Prometheus metrics now. I created a [GitLab CI overview dashboard](https... [09:56:39] hello folks [09:56:51] I filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/752613 to move kafka main eqiad to the fixed uid/gid scheme [09:57:00] (it will need stop/start kafka on every node) [09:57:44] I also filed the change to upgrade nic/bios/etc.. for kafka-main100[1-3], after that we'll be able to reimage [09:58:03] (basically the same that we did in codfw) [10:34:15] <_joe_> elukey: gave you a +1 already [10:38:11] super thanks :) [10:47:09] very nice log message in journalctl when starting kafka (before the acceptor threads are up) [10:47:12] [2022-01-10 10:42:35,708] ERROR Error while accepting connection (kafka.network.Acceptor) [10:47:15] java.lang.ArithmeticException: / by zero [10:48:48] anyway, kafka1005 up and running [10:52:14] 1004 done as well, will wait a bit [11:01:17] 1003 done [11:14:54] 1002 done [11:35:54] aaand 1001 done [11:36:03] main-eqiad with fixed uid/gid :) [11:40:30] <_joe_> \o/ [11:50:55] 10serviceops, 10Patch-For-Review: Upgrade kafka-main nodes to buster - https://phabricator.wikimedia.org/T296641 (10elukey) Next steps: - Upgrade BIOS+NIC on kafka-main100[1-3] - T298867 - Reimage the nodes to Buster [14:27:21] 10serviceops, 10Data-Engineering, 10observability, 10Patch-For-Review: Move kafka clusters to fixed uid/gid - https://phabricator.wikimedia.org/T296982 (10elukey) [19:12:47] 10serviceops, 10Infrastructure-Foundations, 10User-RhinosF1: cross-validate-accounts: Malformed membership for ops user ..., has additional group(s): {'deployment-ci-admins'} - https://phabricator.wikimedia.org/T298815 (10Dzahn) Yep, same here, now aware of the list. Thanks for the patch @RhinosF1 [20:00:18] 10serviceops, 10Release-Engineering-Team (Seen): contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10Dzahn) [20:04:52] 10serviceops, 10Release-Engineering-Team (Seen): contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10Dzahn) This would also fix T283582 and T298861 and putting further effort into that kind of thing [20:53:05] 10serviceops, 10SRE, 10Thumbor, 10User-jijiki: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10Ahecht) Stretch was supposed to be phased out by June 2021 per https://wikitech.wikimedia.org/wiki/Operating_system_upgrade_policy, and will be EOL in less than 6 months (June 30, 2022) p... [21:18:08] 10serviceops, 10SRE, 10Thumbor, 10User-jijiki: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10AntiCompositeNumber) >>! In T216815#7610660, @Ahecht wrote: > Is any work being done on this? At the moment, no. Thumbor currently has no maintainer, see T294484.