[09:14:32] hello folks! [09:14:42] I've upgraded docker-report on build2001, testing it now [09:15:36] it should now support a minimal debian version (to avoid weird filters in the config), and it reports the docker images that are not "supported" (like if there are Jessie/Stretch ones still surviving etc..) [09:15:42] credits/blame to Janis :D [09:42:30] <_joe_> it reports as in it logs? [09:48:10] yeah [10:04:16] 06serviceops, 10WMDE-TechWish-Maintenance, 07Epic, 10Maps (Kartotherian): Move Kartotherian to Kubernetes - https://phabricator.wikimedia.org/T216826#9984531 (10awight) Yes, +1 this is the most problematic dependency. We'll need to use the upstream node-mapnik and libmapnik-3.1, see {T327396} . We have m... [10:09:43] 06serviceops, 10WMDE-TechWish-Maintenance, 07Epic, 10Maps (Kartotherian): Move Kartotherian to Kubernetes - https://phabricator.wikimedia.org/T216826#9984582 (10elukey) @awight thanks for the reply! I agree with the proposal, but Debian Buster is already end of life support, so the containerization would r... [10:09:52] _joe_ yes ideally in the bright future we'll have breakdowns in debmonitor [10:10:07] like docker-image:X deployed Y times in these clusters: .... [10:10:23] <_joe_> debmonitor is not the right place for that [10:10:33] <_joe_> but there's a start of work in that direction that was never completed [10:10:42] <_joe_> (search "imagecatalog") [10:11:24] I am not sure what is the state of image catalog, it could be used as well. But to me having a single place where to check packages and os versions wouldn't be bad [10:11:40] anyway, as long as we have a place with the info, I'd be happy [10:31:00] akosiaris: 👋 I sent this patch to prepare for the complete deprecation of mobile-sections endpoint in a month https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1054512 [10:31:22] From what I saw in turnilo we dont get any requests from mw-offliner (our latest officially supported consumer) [10:32:01] and this patch would make things easier in the process of disabling parsoid pregeneration (less load to MW, less complicated changeprop config) [10:34:48] any chance we can get this merged now and not wait for the scheduled block of incoming traffic in Aug ? [10:38:23] hm, i think _akosiaris is out for the next few days [10:40:06] Either way, if there is anyone who can help with this patch it would be appreciated :) [11:04:29] nemo-yiannis: let me check with the others, I will ping you slightly later [11:53:55] thanks effie [13:50:56] 06serviceops, 10WMDE-TechWish-Maintenance, 07Epic, 10Maps (Kartotherian): Move Kartotherian to Kubernetes - https://phabricator.wikimedia.org/T216826#9985458 (10awight) The last time we did an analysis of maps usage was June 2023, see [[ https://docs.google.com/spreadsheets/d/1q71Swzxpf2U4shhSJl8fry-CHg1Ra... [14:31:37] claime: sorry to pick on you, per-chance can you assist draining the k8s/mw nodes before network maintenance in rack F2 today? [14:31:50] kubernetes1062, mw1494 & mw1495 [14:31:55] https://phabricator.wikimedia.org/T365997 [14:31:56] sorry meeting, be with you in a bit [14:32:48] yeah, on it [14:32:56] no probs at all [14:33:02] ping me when you're done [14:34:29] 06serviceops, 10WMDE-TechWish-Maintenance, 07Epic, 10Maps (Kartotherian), 13Patch-For-Review: Move Kartotherian to Kubernetes - https://phabricator.wikimedia.org/T216826#9985903 (10awight) I've just attached a slight tweak which successfully builds on nodejs18 and uses upstream mapnik: https://gerrit.wik... [14:42:02] topranks: done [14:42:45] claime: thanking you!! [14:47:34] jayme: rzl: another quick look at https://gerrit.wikimedia.org/r/1054394 would be appreciated <3 [15:10:03] 06serviceops, 06MediaWiki-Engineering, 10MediaWiki-libs-BagOStuff, 06MediaWiki-Platform-Team, and 2 others: Cache mw-mcrouter service ClusterIP in apcu cache - https://phabricator.wikimedia.org/T363186#9986079 (10jijiki) 05Open→03Invalid [15:12:43] jayme: hmm I still don't see a diff for the chart itself [15:12:54] damn [15:13:09] https://integration.wikimedia.org/ci/job/helm-lint/19245/console latest run [15:17:04] what I really don't understand is I made the lint fail in the past by having the wrong apiVersion [15:19:22] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Allow to address Kubernetes API servers from NetworkPolicy - https://phabricator.wikimedia.org/T287491#9986132 (10jijiki) [15:19:34] I think it tries to render without a fixture first [15:20:29] that probably failed in your first attempt. But I did not even properly recall the fixture dir thing, so it might be broken for some time... gonna check [15:21:02] hmm [15:23:50] eheh, the readme sais .fixturesctl.yml :D [15:24:01] .fixturectl.yml [15:24:28] but the code sais .yaml, so all good there I suppose [15:26:12] it also doesn't change anything if I make the name match the others (so ci/wmfnetworkpolicy-values.yaml), nor if I also include a much fuller definition (with mode: daemonset and all that, following from networkpolicy-override-values.yaml) [15:26:17] cdanis: oopsie [15:26:22] 👀 [15:26:29] .fixtures.yaml is useless [15:26:38] claime: all done in rack F2 you can repool those any time [15:26:39] it needs to be in .fixturesctl.yaml [15:26:47] topranks: thanks! [15:26:48] (the fixtures_dir) [15:27:56] not sure if that was different in the past, but I don't see anything reading the .fixture.yaml [15:28:14] testing now [15:34:50] jayme: okay ptal [15:39:34] cdanis: uhm...now the fixtures_dir statement is missing completely. It needs to be in .fixturectl.yaml (no idea what the .fixtures.yaml was supposed to do) [15:40:01] jayme: see parent patch :) [15:40:38] oh, sorry. My bad [15:41:25] looks good [15:41:28] thanks!! [15:54:11] 06serviceops, 10MW-on-K8s, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Create ValidatingAdmissionPolicies to replace mediawiki PSP - https://phabricator.wikimedia.org/T368251#9986399 (10JMeybohm) The policy parser tool has moved to https://gitlab.wikimedia.org/repos/sre/kyverno-policy-parser [16:09:58] 06serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: mcrouter daemonset on mw-on-k8s - https://phabricator.wikimedia.org/T346690#9986482 (10jijiki) All mediawiki pods have been successfully switched to using mw-mcrouter, while we removed the mcrouter and prometheus-mcrouter-exp... [16:15:21] 06serviceops, 10MediaWiki-Platform-Team (Radar): Enable extstore to a subset of memcached servers (experiment) - https://phabricator.wikimedia.org/T352885#9986523 (10jijiki) @Krinkle Extstore is fully enabled on eqiad [16:41:06] 06serviceops, 10AQS2.0, 06Data Products: Investigate effort in routing device analytics via istio - https://phabricator.wikimedia.org/T358324#9986724 (10VirginiaPoundstone) [17:04:38] akosiaris: claime: I guess we should merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1053791 to remove scandium from dsh groups? just so deployers don't get errors [17:04:51] if scandium is effectively decom'ed that is [17:07:53] mutante: hold up :D ot [17:07:58] it's not decom'd yet [17:08:17] ah, ok! I saw some activity on the decom task but wasn't sure [17:08:38] I think at some point deployers talked about seeing it durin scap [20:14:34] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#9987802 (10Scott_French) Current status: * appservers-rw and api-rw are depooled everywhere, and resolve to failoid as of 17:45 UTC * api-ro is serving only...