[01:25:48] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2057.codfw.wmnet with OS bullseye [01:35:52] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2058.codfw.wmnet with OS bullseye [01:36:45] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10Papaul) [01:56:23] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2059.codfw.wmnet with OS bullseye [02:07:57] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2057.codfw.wmnet with OS bullseye completed: - kubernetes2057 (**PASS**)... [02:08:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10Papaul) [02:09:22] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2060.codfw.wmnet with OS bullseye [02:20:00] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2058.codfw.wmnet with OS bullseye completed: - kubernetes2058 (**PASS**)... [02:44:07] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2059.codfw.wmnet with OS bullseye completed: - kubernetes2059 (**PASS**)... [02:50:00] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2060.codfw.wmnet with OS bullseye completed: - kubernetes2060 (**PASS**)... [03:07:00] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10Papaul) [03:07:59] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10Papaul) 05Open→03Resolved @Clement_Goubert @Joe all your's [08:11:34] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10brouberol) @pfischer Once you agree on the config, I can create and configure the topic for you. As this is to be a compacted... [08:16:16] <_joe_> brouberol: that topic will go on kafka-main, so please coordinate with us :) [08:18:50] sure, no problem [08:19:07] I can let you perform the creation if you want. I was @ on th [08:19:14] 10serviceops: Migrate etcd::tlsproxy Nginx certs to PKI - https://phabricator.wikimedia.org/T352245 (10MoritzMuehlenhoff) [08:19:32] *the ticket, but if you'd prefer to manage it, that's no problem 👍 [08:54:22] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [09:11:14] <_joe_> brouberol: so, no I'm very happy to let you do it, although it should be our duty :D [09:11:39] <_joe_> I was just asking to wait until we've taken another look at the task, nothing should be surprising as we've talked extensively about this producer [09:13:56] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10Clement_Goubert) Thanks! [09:18:02] 10serviceops, 10SRE: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10Clement_Goubert) [09:18:51] 10serviceops, 10SRE: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10Clement_Goubert) p:05Triage→03Medium [10:50:47] 10serviceops, 10SRE, 10Patch-For-Review: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1001 for host kubernetes2057.codfw.wmnet with OS bullseye [10:59:57] 10serviceops, 10SRE, 10Patch-For-Review: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1001 for host kubernetes2058.codfw.wmnet with OS bullseye [11:00:17] 10serviceops, 10SRE, 10Patch-For-Review: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1001 for host kubernetes2059.codfw.wmnet with OS bullseye [11:00:36] 10serviceops, 10SRE, 10Patch-For-Review: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1001 for host kubernetes2060.codfw.wmnet with OS bullseye [11:11:00] _joe_ re: kafka topics - afaics the request seems good, nothing weird that I see space-wise on the nodes [11:11:53] <_joe_> elukey: yeah the number looked ok-ish to me [11:13:26] _joe_ should we +1 it and give the green light? [11:13:38] <_joe_> elukey: sure :) [11:13:51] okok :) [11:15:20] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10elukey) @pfischer option A) is fine, if there is a way to add the new traffic incrementally (to double check space used by th... [11:34:05] 10serviceops, 10SRE, 10Patch-For-Review: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1001 for host kubernetes2057.codfw.wmnet with OS bullseye completed: - kubernetes2057 (**PASS**) - Down... [11:38:33] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Move MediaWiki jobs to mw-on-k8s - https://phabricator.wikimedia.org/T349796 (10Clement_Goubert) [11:43:17] 10serviceops, 10SRE, 10Patch-For-Review: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1001 for host kubernetes2058.codfw.wmnet with OS bullseye completed: - kubernetes2058 (**PASS**) - Down... [11:45:24] 10serviceops, 10SRE, 10Patch-For-Review: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1001 for host kubernetes2060.codfw.wmnet with OS bullseye completed: - kubernetes2060 (**PASS**) - Down... [12:02:14] 10serviceops, 10SRE, 10Patch-For-Review: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1001 for host kubernetes2059.codfw.wmnet with OS bullseye completed: - kubernetes2059 (**WARN**) - Down... [12:04:53] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10pfischer) @elukey, sure. We would start onboarding smaller wikis first (test, it, fr) before moving on to the bigger ones. @... [12:30:13] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349873 (10Clement_Goubert) [12:31:01] 10serviceops, 10SRE: setup/install kubernetes20[57-60] - https://phabricator.wikimedia.org/T352369 (10Clement_Goubert) 05Open→03Resolved Hosts are in production, resolving. [14:08:50] 10serviceops: Migrate etcd::tlsproxy Nginx certs to PKI - https://phabricator.wikimedia.org/T352245 (10MoritzMuehlenhoff) Turns out John already made a patch for this back in 2022: https://gerrit.wikimedia.org/r/c/operations/puppet/+/790657 [14:09:10] 10serviceops: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245 (10MoritzMuehlenhoff) [14:12:33] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:rack/setup/install 3 sessionstore hosts (codfw) - https://phabricator.wikimedia.org/T349876 (10Papaul) a:03Jhancock.wm [15:08:06] 10serviceops, 10WMF-JobQueue, 10Wikimedia-production-error: Make changeprop-jobqueue error handling/httpbb tests better behaved: Uncaught Error: Class 'MWExceptionHandler' not found in /srv/mediawiki/rpc/RunSingleJob.php:42 - https://phabricator.wikimedia.org/T352265 (10hashar) [15:13:51] 10serviceops, 10WMF-JobQueue, 10Wikimedia-production-error: Make changeprop-jobqueue error handling/httpbb tests better behaved: Uncaught Error: Class 'MWExceptionHandler' not found in /srv/mediawiki/rpc/RunSingleJob.php:42 - https://phabricator.wikimedia.org/T352265 (10hashar) Looks like it comes from https... [17:01:31] 10serviceops, 10Traffic: Java fails to install on WMF Debian container - https://phabricator.wikimedia.org/T352350 (10BCornwall) [17:19:53] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10EBernhardson) For deleting the topic, if we need to pause all writers and consumers that can relatively easily be done. test... [17:42:09] I'm running into some issues with docker-registry.wikimedia.org/{bookworm,bullseye}:latest - Installing [17:42:12] java packages is erroring out due to lack of a /usr/share/man directory. Searching seems to suggest this is a common issue on slim varian [17:42:14] ts of debian. Are these images based on slim? (I can't find the registry dockerfile/containerfile) [17:44:06] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work), 10Patch-For-Review: Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10brouberol) The partition count can be changed on the fly (only increased, never decreased), that's no i... [17:46:09] brett: we build the images ourselves, so yes but no. https://phabricator.wikimedia.org/T352350 ;-) [17:47:08] oh, that's actually a task of yours - sorry :o ... I think I meant to link https://phabricator.wikimedia.org/T289694 [17:47:24] IIRC we just mkdir in the java base images [18:00:29] jayme: Thanks for the reply. Should the base image mkdir it then instead of expecting downstream images to do that? [18:40:24] <_joe_> brett: we have java 11 images [18:40:58] <_joe_> brett: see https://phabricator.wikimedia.org/T352350#9369726 [18:41:11] <_joe_> so base your image on those [18:42:18] <_joe_> unless you *definitely* need a newer distro [18:42:50] <_joe_> but yes, our images are "slim" [18:43:43] <_joe_> brett: as to unblock you in this latter case - just create /usr/share/man [18:44:22] <_joe_> something like RUN mkdir /usr/share/man && apt... && rm -rf /usr/share/man [19:02:55] _joe_: java is just a passing dep, not the main program [19:03:18] It's made more awkward in that this is being used for kokkuri/blubber so it's not just a Dockerfile that's being written [19:03:38] <_joe_> brett: that I assumed was the case [19:03:59] _joe_: My question was whether it was worth adding that RUN into the image itself since any downstream will run into this problem [19:04:40] <_joe_> brett: well you can build a base java11-bookworm using docker-pkg [19:04:55] <_joe_> with that hack while the package is fixed [19:06:12] <_joe_> you can create an additional directory here https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/refs/heads/master/images/java/ [19:08:10] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:rack/setup/install 4 parsoid hosts - https://phabricator.wikimedia.org/T349874 (10Jclark-ctr) a:03VRiley-WMF [20:08:15] that jdk debian package issue is bug #1 https://salsa.debian.org/openjdk-team/openjdk/-/merge_requests/1 :D