[01:04:20] <wikibugs>	 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10tstarling) p:05Unbreak!→03High So the wtp* servers were indeed out of memory, as reported at T296098. Ther...
[07:06:44] <wikibugs>	 10serviceops, 10MW-on-K8s, 10SRE: Make all httpbb tests pass on the mwdebug deployment. - https://phabricator.wikimedia.org/T285298 (10Joe)
[08:15:16] <wikibugs>	 10serviceops, 10SRE, 10WMF-Legal, 10Patch-For-Review: Move old transparency report pages to historical URLs and setup redirect - https://phabricator.wikimedia.org/T230638 (10Prtksxna) Thanks @Dzahn! I don't have access to see that ticket, but I'll keep this in mind.
[13:31:54] <wikibugs>	 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto)
[14:30:01] <wikibugs>	 10serviceops, 10Maps, 10Product-Infrastructure-Team-Backlog, 10Patch-For-Review, 10User-jijiki: Maps 2.0 roll-out plan - https://phabricator.wikimedia.org/T280767 (10MSantos)
[14:44:31] <wikibugs>	 10serviceops, 10MW-on-K8s: On the kube-experimental mwdebug cluster, MediaWiki sees all edits as coming from localhost - https://phabricator.wikimedia.org/T297613 (10Joe)
[14:44:40] <wikibugs>	 10serviceops, 10MW-on-K8s: On the kube-experimental mwdebug cluster, MediaWiki sees all edits as coming from localhost - https://phabricator.wikimedia.org/T297613 (10Joe) p:05Triage→03High
[14:48:21] <wikibugs>	 10serviceops, 10GitLab (Infrastructure), 10Security: GitLab Runner Critical Security Release: 14.5.2, 14.4.2, and 14.3.4 - https://phabricator.wikimedia.org/T297581 (10sbassett)
[15:15:25] <elukey>	 hello folks
[15:15:36] <elukey>	 I am experimenting a bit with the istio TLS mesh, and I filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/746880/
[15:15:57] <elukey>	 it is more to discuss what to do rather than a definitive statement of what is the best road to follow
[15:16:13] <elukey>	 the alternative is to use the istio cni plugin, but it seems way more complicated
[15:16:23] <elukey>	 (I could be wrong though)
[15:17:47] <elukey>	 https://istio.io/latest/docs/setup/additional-setup/cni/ is the docs from upstream
[15:28:58] <jayme>	 uh...this reminds me of how calico wants pivileged containers to place CNI binaries in the host filesystem
[15:29:01] <_joe_>	 elukey: we're a bit wary of istio mesh
[15:30:09] <elukey>	 I am very unfamiliar with the calico cni plugin, but it looks similar
[15:30:28] <elukey>	 meanwhile the istio-init container messes up with the pod's iptables rules
[15:30:39] <elukey>	 (hence it needs the extra capabilities)
[15:38:09] <_joe_>	 yeah it does godforsaken stuff so that the developer can write "en.wikipedia.org" in their manifests and be redirected to the correct service
[15:42:50] <jayme>	 elukey: those plugins will be chained. The calico CNI will call isio CNI in it's process.
[15:48:31] <elukey>	 _joe_ exactly yes, and since I am also experimenting with the egress gateway, I thought that it would have been nice to have that capability (without explicitly set proxy + Host headers etc..)
[15:48:45] <wikibugs>	 10serviceops, 10MW-on-K8s: On the kube-experimental mwdebug cluster, MediaWiki sees all edits as coming from localhost - https://phabricator.wikimedia.org/T297613 (10Joe) Narrowing down the problem: I see the actual client IP in the apache httpd logs for my requests. So it seems that the problem is somewhere b...
[15:48:49] <elukey>	 jayme: and we ship the calico binary via deb package right?
[15:49:08] <jayme>	 elukey: correct. That plus the cni config
[15:50:17] <elukey>	 ah yes I see, https://istio.io/latest/docs/setup/additional-setup/cni/#hosted-kubernetes-settings
[15:50:37] <elukey>	 so istioctl's manifest wants a place on disk where to find it
[16:02:28] <elukey>	 I see also that istioctl / IstioOperator may install the binary if needed, so I guess there will be a lot of fun making it work :D
[16:03:02] <elukey>	 anyway, my original motivation to progress this was to have a more flexible egress/proxy config, it sounds more work than needed :
[16:29:51] <wikibugs>	 10serviceops, 10Data-Engineering, 10observability, 10Patch-For-Review: Move kafka clusters to fixed uid/gid - https://phabricator.wikimedia.org/T296982 (10odimitrijevic) p:05Triage→03Medium
[16:42:27] <wikibugs>	 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10brennen) p:05High→03Unbreak! > In any case, it's not happening now, so it's not a UBN.  We're currently on...
[16:44:34] <wikibugs>	 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10Ladsgroup) This can be similar to {T297236} I suggest we deploy the fix and roll with wmf.12 to see if it stil...
[16:46:47] <wikibugs>	 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10Ladsgroup) This looks good: https://logstash.wikimedia.org/goto/d85f220054b5e2145a56b5fe99c4e653 possibly some...
[16:54:54] <wikibugs>	 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10dancy) >>! In T297517#7566865, @Ladsgroup wrote: > This can be similar to {T297236} I suggest we deploy the fi...
[18:08:46] <wikibugs>	 10serviceops, 10MW-on-K8s: On the kube-experimental mwdebug cluster, MediaWiki sees all edits as coming from localhost - https://phabricator.wikimedia.org/T297613 (10Legoktm) MediaWiki looks at `$_SERVER['REMOTE_ADDR']`, the relevant code is https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/...
[18:09:08] <wikibugs>	 10serviceops, 10GitLab (Infrastructure): Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) > I created a new instance called "runner-bullseye" with the idea to put the gitlab_runner puppet class on it and see how it goes and do so on bullseye. But I did not get to act...
[18:32:54] <wikibugs>	 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10Ladsgroup) The underlying memory leak has not been fixed but so far it looks good with the changes being backp...
[21:45:41] <wikibugs>	 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10wiki_willy) Hi @Joe - just following up on this.   Can we get any specific racking criteria for you on this install task?   Thanks, Willy
[23:16:56] <wikibugs>	 10serviceops, 10GitLab: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Dzahn)
[23:22:09] <wikibugs>	 10serviceops, 10GitLab (Infrastructure): Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) >>! In T297411#7567146, @Jelto wrote: > Thanks for thinking about moving the runners to bullseye. I'm not sure if this task has a lot of overlap with the migration of the Runner...