[04:08:45] 10serviceops, 10Phabricator, 10Release-Engineering-Team: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10hashar) [04:14:11] 10serviceops, 10Phabricator, 10Release-Engineering-Team: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10hashar) We have settled on migrating out of pws/gpg/git to store our credentials in favor of 1password.com . The migration itself is not that complicate... [04:14:37] 10serviceops, 10Phabricator, 10Release-Engineering-Team (Next): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10hashar) [09:57:11] hnowlan: re changeprop - this may be a red herring but https://github.com/wikimedia/change-propagation/blob/master/.pipeline/blubber.yaml#L29 seems suspiciously high [10:07:10] hello folks [10:07:32] one thing that I noticed on ml-serve2006 (bullseye + overlay) is that it is missing the hugetlb cgroup [10:07:37] that is present on buster [10:08:28] but not on stretch [10:08:48] (so cgroup mounts wise, we have the same on stretch and bullseye atm afaics) [10:08:55] nothing horrible in the kubelet logs related to it [10:12:10] elukey: o/ - where you actuall looking for differences or did that pop up somewhere? [10:13:19] jayme: o/ I was looking for differences, paranoid mode [10:13:28] ack :) [10:26:17] created https://gerrit.wikimedia.org/r/c/operations/puppet/+/762410 [10:26:20] with all the context etc.. [10:26:45] in theory the above should work fine with a reimage since we reboot after the first puppet run [10:43:20] cool 👍 [10:46:54] 10serviceops, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10elukey) Reporting my findings in here to keep archives happy. With `systemd.unified_cgroup_hierarchy=0 ` on Bullseye, I see the s... [11:01:38] mszabo: good question! I'm not sure what that was set to remedy, I'll try to look into that - afair it was something specific to changeprop underperforming in k8s. In general we spawn a consumer per topic on each node so the assignment issue will happen with any threadpool size. [11:03:35] yeah, I tried to follow `git appreciate` but the relevant change ID seems to have somehow disappeared from gerrit [11:03:52] http://docs.libuv.org/en/v1.x/threadpool.html notes that this threadpool is mostly used for dns and fs ops [11:09:51] 128 just strikes me as a (N * host cpu count) kind of value assignment [11:15:52] looks like that value has been used forever for one reason or another, long before k8s was a factor https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/73db34ac69c/modules/profile/manifests/cpjobqueue.pp#50 [11:57:23] nice find! [11:58:02] I guess the question is, whether that threadpool really gets saturated in k8s [12:50:29] 10serviceops, 10Add-Link, 10Growth-Team, 10Patch-For-Review: Many repeated config file changed / config file reloaded messages - https://phabricator.wikimedia.org/T300629 (10kostajh) >>! In T300629#7667467, @JMeybohm wrote: > This is actually common across all services that use prometheus-statsd exporter.... [15:50:52] 10serviceops, 10Phabricator, 10Release-Engineering-Team: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10Dzahn) 05duplicate→03Open [15:51:53] 10serviceops, 10Phabricator, 10Release-Engineering-Team: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10Dzahn) This ticket wasn't about migrating pws to another solution. It was about moving the repo out of phabricator or, alternatively, to stop using ssh... [17:33:00] 10serviceops, 10SRE, 10Wikimedia-Etherpad, 10vm-requests, 10Patch-For-Review: create bullseye VM for Etherpad upgrade (and upgrade it to 1.8.16) - https://phabricator.wikimedia.org/T300568 (10Dzahn) @Volans Yes, it has been fixed by making etherpad listen on "::" with https://gerrit.wikimedia.org/r/c/o... [17:58:19] 10serviceops, 10SRE, 10Wikimedia-Etherpad, 10vm-requests, 10Patch-For-Review: create bullseye VM for Etherpad upgrade (and upgrade it to 1.8.16) - https://phabricator.wikimedia.org/T300568 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `etherpad1002.eqiad.w... [17:59:59] 10serviceops, 10SRE, 10Wikimedia-Etherpad: vm request for etherpad1002 - https://phabricator.wikimedia.org/T243475 (10Dzahn) decom'ed today as part of T300568 [19:17:45] 10serviceops, 10Phabricator, 10Release-Engineering-Team: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10Aklapper) [21:38:37] Heads up that any Blubber containers using Blubber's `python` configuration may have issues caused by setuptools==60.9.0 being installed. Upstream issue is https://github.com/pypa/setuptools/issues/3102 and https://phabricator.wikimedia.org/T301690 documents us working around things for the very short term in Toolhub.