[08:07:31] <_joe_> jelto: great find with the global hooks [08:14:08] 10serviceops, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10elukey) @JMeybohm from what I got you've been voluntold to review my work before reimaging a wikikube staging node, when you have... [08:16:54] _joe_: thanks! do you want me to amend it to all other helmfiles? [08:17:17] <_joe_> jelto: yeah, I would assume we can do it at our pace [08:17:41] <_joe_> btw, I'm starting to hate the fact we need to c/p so much stuff across helmfiles [08:17:56] <_joe_> I'm kinda-tempted to make an autogeneration feature for them [08:20:13] ok I'll amend it for the other helmfiles as well. Having some kind of meta template helfile and autogenerate (or maybe include?) these files sounds like a good idea [08:21:48] <_joe_> yeah my idea was to have a simpler yaml file with just [08:21:51] <_joe_> -file hierarchies [08:21:54] <_joe_> - releases [08:22:01] <_joe_> if someone needs to override them [08:22:08] <_joe_> and the rest, autogenerate from a template [08:22:32] <_joe_> how much do you hate ruby? [08:22:34] <_joe_> :P [08:24:20] jinja2 templates! [08:24:52] sounds like a good idea. I have a aversion against ruby, let's say it this way :D [08:29:54] <_joe_> legoktm: that's more or less the only thing I despise almost as much as go text/template [08:30:11] <_joe_> erb is much better than jinja2, by being worse [08:30:42] heh [08:59:26] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Provide a convenient way to connect to services in kubernetes staging clusters - https://phabricator.wikimedia.org/T300740 (10Joe) I would generally agree that having all services with name *.staging.$dc.wmnet resolve to the same rr-record... [09:43:02] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Provide a convenient way to connect to services in kubernetes staging clusters - https://phabricator.wikimedia.org/T300740 (10JMeybohm) >>! In T300740#7713949, @Joe wrote: > I would generally agree that having all services with name *.stagi... [11:12:48] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: setup/install kubernetes10[18-21] - https://phabricator.wikimedia.org/T293728 (10JMeybohm) @akosiaris we could postpone this a bit and image the nodes with bullseye + overlayfs directly (T300744) to not loose capacity when something goes sideways. AIUI we will b... [13:08:19] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) [13:08:29] 10serviceops, 10SRE, 10Wikimedia-Etherpad: Prometheus etherpad scrape failure - https://phabricator.wikimedia.org/T301872 (10fgiunchedi) [13:11:31] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) Migration of new test instance to `wmcloud.org` zone was successful. SSO login using wmcloud idp also works. I would consider the test instance under https... [13:26:18] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:(Need By: TBD) rack/setup/install kubernetes10[18-21] - https://phabricator.wikimedia.org/T290202 (10akosiaris) [13:26:20] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: setup/install kubernetes10[18-21] - https://phabricator.wikimedia.org/T293728 (10akosiaris) 05Open→03Stalled >>! In T293728#7714290, @JMeybohm wrote: > @akosiaris we could postpone this a bit and image the nodes with bullseye + overlayfs directly (T300744) t... [13:34:44] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: setup/install kubernetes10[18-21] - https://phabricator.wikimedia.org/T293728 (10elukey) @akosiaris in theory we can have bullseye+overlay nodes simply adding this per-host hiera config: ` # See https://phabricator.wikimedia.org/T300744 profile::base::overlayfs... [13:36:49] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: setup/install kubernetes10[18-21] - https://phabricator.wikimedia.org/T293728 (10akosiaris) >>! In T293728#7714672, @elukey wrote: > @akosiaris in theory we can have bullseye+overlay nodes simply adding this per-host hiera config: > > ` > # See https://phabrica... [13:49:21] 10serviceops, 10SRE, 10Wikimedia-Etherpad: Prometheus etherpad scrape failure - https://phabricator.wikimedia.org/T301872 (10akosiaris) Hm, this shows up in the logs ` Feb 16 13:40:29 etherpad1003 prometheus-etherpad-exporter[805482]: UnboundLocalError: local variable 'metric_name' referenced before assignm... [13:49:56] hello! I need to delete an image from our Docker registry and I have no idea how to do that. Any guidance? The image is docker-registry.wikimedia.org/releng/quibble-buster:1.4.1 and is incorrect :D [13:50:36] or at least I could not find the docker cli command line to delete one [13:51:00] hashar: please check https://wikitech.wikimedia.org/wiki/Docker-registry#Deleting_images [13:51:09] we have doc!!! thank you ;) [13:51:14] yw [13:51:33] though I don't have access to deneb :-\ [13:51:38] tl;dr: you won't find a docker cli command, there is none [13:52:16] but I have the command on my host great [13:52:44] should work as long as you have write permissions to the registry [13:53:21] ping me if it does not. I can delete the image for you then [13:53:35] 10serviceops, 10SRE, 10Wikimedia-Etherpad, 10Patch-For-Review: Prometheus etherpad scrape failure - https://phabricator.wikimedia.org/T301872 (10akosiaris) There seem to be quite a few new metrics around. ` { "httpStartTime": 1644710413335, "memoryUsage": 328769536, "memoryUsageHeap": 170679824,... [13:54:06] jayme: that got rid of it thank you [14:33:38] 10serviceops, 10SRE, 10Wikimedia-Etherpad, 10Patch-For-Review: Prometheus etherpad scrape failure - https://phabricator.wikimedia.org/T301872 (10akosiaris) I 've hotpatched this in production to stop the bleeding for now but the proper way to solve this is of course to add support for the new metrics and m... [15:08:25] I just deployed the SAL logging changes for all helmfiles and they look fine. Do you think it's worth mentioning that logging output has changed when doing a deploy? In my opinion the logging is quite similar-ish for end users and I would not write something on ops@ [15:19:00] 10serviceops, 10SRE, 10Wikimedia-Etherpad, 10Patch-For-Review: Prometheus etherpad scrape failure - https://phabricator.wikimedia.org/T301872 (10akosiaris) p:05Triage→03Medium [15:19:18] jelto: I think it's fine given that nobody complained when we introduces the last change (wich lead to more lines) :) [15:32:56] jayme: ack! [16:37:14] 10serviceops: test pushing to phabricator repos over https - https://phabricator.wikimedia.org/T301889 (10Dzahn) [16:38:13] 10serviceops: test pushing to phabricator repos over https - https://phabricator.wikimedia.org/T301889 (10Dzahn) [16:38:21] 10serviceops, 10Phabricator, 10Release-Engineering-Team (Next): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) [16:38:25] 10serviceops, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10JMeybohm) >>! In T300744#7713782, @elukey wrote: > @JMeybohm from what I got you've been voluntold to review my work before reimag... [16:38:53] 10serviceops, 10Phabricator, 10Release-Engineering-Team (Next): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) [16:40:42] tickets in Phabricator that actually affect me but are writtein in Spanish..that's new to me..but also cool in a way https://phabricator.wikimedia.org/T296108 [16:51:55] <_joe_> jelto: I agree [16:52:32] <_joe_> mutante: meh, we chose english as a lingua franca of science and engineering for a reason [16:52:55] <_joe_> namely, that the native language of the american empire is english [16:53:13] <_joe_> I would be ok with a change, but I think we need to keep using a single lingua franca [16:53:36] <_joe_> if we want to switch, I vote for latin [16:55:47] heh, ok, fair enough.quickly going back to English then:) They see it as their personal ticket but it will help me if they do it [16:56:35] 10serviceops, 10Data-Engineering, 10SRE, 10observability: Upgrade Kafka to 2.x - https://phabricator.wikimedia.org/T300102 (10jbond) p:05Triage→03Medium [16:57:14] <_joe_> mutante: yeah i figured! [16:57:31] <_joe_> but we have native spanish speakers around if you want to be friendly [16:58:17] I will use Google translate and kind of like learning a bit of Spanish along the way. I already tried to read Spanish news for this reason, so works for me [17:04:46] if you're going to go for old languages, you know I'm gonna see your latin and raise you ancient greek :-P [17:37:55] <_joe_> apergos: "ancient greek" is a very vague term [17:38:03] <_joe_> do you refer to the Attic dialect? [17:38:17] choose any! [17:38:18] <_joe_> I can see that athenian bias [17:38:40] we try. [17:49:11] 10serviceops, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Yak Shaving 🐃🪒): contint1001 and contint2001 need a newer version of Docker installed - https://phabricator.wikimedia.org/T300682 (10dduvall) @Muehlenhoff for some reason, it seems the docker-ce package did not make it into th... [17:50:13] 10serviceops, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Yak Shaving 🐃🪒): contint1001 and contint2001 need a newer version of Docker installed - https://phabricator.wikimedia.org/T300682 (10dduvall) And many thanks to @dzahn for troubleshooting following the puppet apply of https://... [21:57:46] 10serviceops, 10SRE, 10Wikimedia-Etherpad, 10Patch-For-Review: Prometheus etherpad scrape failure - https://phabricator.wikimedia.org/T301872 (10Dzahn) ` [apt1001:~] $ sudo -E reprepro ls prometheus-etherpad-exporter prometheus-etherpad-exporter | 0.3 | buster-wikimedia | amd64, i386, source prometheus-e... [22:34:46] 10serviceops, 10SRE, 10Wikimedia-Etherpad: Prometheus etherpad scrape failure - https://phabricator.wikimedia.org/T301872 (10Dzahn) 05Open→03Resolved @akosiaris :) reviewed / merged patches, built package on deneb, uploaded package on apt1001, imported with reprepro, exported indices, installed package... [22:34:48] 10serviceops, 10SRE, 10Wikimedia-Etherpad, 10vm-requests, 10Patch-For-Review: create bullseye VM for Etherpad upgrade (and upgrade it to 1.8.16) - https://phabricator.wikimedia.org/T300568 (10Dzahn)