[08:32:45] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10pfischer) @brouberol, if you have time, could you configure `codfw.mediawiki.cirrussearch.page_rerender.v1` and `eqiad.mediaw... [08:46:02] _joe_is that OK if I proceed https://phabricator.wikimedia.org/T351503#9377865 ? cc pfischer [08:48:52] (that'd be on kafka-main) [08:59:35] brouberol: o/ - I am a bit confused, why do we need a specific cleanup policy? [08:59:46] I thought the changes were only related to partitions [09:00:23] pfischer would know best here, I'm merely being asked to apply the changes [09:02:46] ack, going to ask in the task :) [09:04:00] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10elukey) >>! In T351503#9377865, @pfischer wrote: > @brouberol, if you have time, could you configure `codfw.mediawiki.cirruss... [09:35:59] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Bump istio Docker images to Bookworm - https://phabricator.wikimedia.org/T351933 (10elukey) [09:50:13] <_joe_> uhm brouberol I can't give you an opinion right now (also sorry - somehow I wasn't pinged by your mention...) [09:50:34] <_joe_> as in - I don't have enough expertise/context to say if it's a potential problem or not [09:50:42] <_joe_> and if it's needed or not :) [09:51:03] <_joe_> I would trust your judgement (and elukey's) there though [09:51:15] no worries, we're waiting for an answer from peter and we'll figure it out, thanks [09:51:30] <_joe_> (also, wasn't 1 week our default retention on kafka-main?) [09:51:43] <_joe_> maybe I have memories from 2016 [09:52:35] <_joe_> ah yes, so they just want to use compact [09:54:00] <_joe_> basically they want kafka to do deduplication for them [09:54:47] <_joe_> it's not wrong per se, depending on how their pipeline is done; but I tend to prefer clients to do the computation work when possible instead of the datastores [09:55:51] <_joe_> so they can keep an index of seen keys in their client, and discard events [09:56:19] <_joe_> OTOH, this would likely reduce network traffic if they have to re-load data [10:11:14] I asked to Peter in the task since we'd need to maintain a separate config for those specific topics, and atm we don't have a way to do it in a git-ops fashion [10:32:53] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10pfischer) @elukey, for page re-render, we're definitely interested only in the latest event, since we only care for the fact... [10:45:32] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10elukey) >>! In T351503#9378213, @pfischer wrote: > @elukey, for page re-render, we're definitely interested only in the lates... [11:05:55] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10pfischer) > We currently don't have any git-ops-like way to apply specific settings to topics Okay, I get your concern. Is t... [11:09:49] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10elukey) >>! In T351503#9378361, @pfischer wrote: >> We currently don't have any git-ops-like way to apply specific settings t... [11:15:43] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1001 for host mw2422.codfw.wmnet with OS bullseye [11:17:18] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1001 for host mw1462.eqiad.wmnet with OS bullseye [11:25:28] 10serviceops, 10Dumps-Generation, 10MW-on-K8s, 10Release-Engineering-Team: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650 (10Joe) [11:25:52] 10serviceops, 10Dumps-Generation, 10MW-on-K8s, 10Release-Engineering-Team: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650 (10Joe) p:05Triage→03Medium [11:27:12] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10MW-1.41-notes (1.41.0-wmf.13; 2023-06-13), 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10KTT-Commons) Another update by a layman user: not only the pa... [11:51:14] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1001 for host mw1462.eqiad.wmnet with OS bullseye completed: - mw1462 (**PASS**) - Downtimed on I... [11:54:50] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1001 for host mw2422.codfw.wmnet with OS bullseye completed: - mw2422 (**PASS**) - Downtimed on I... [12:15:39] <_joe_> elukey: gitops is overrated :) but I get your point [12:18:59] _joe_ somewhere to store settings to be able to re-add them if needed :D [12:21:36] <_joe_> elukey: yep as I said I get your point [12:22:48] Hey! Can somebody with access clarify what node version restbase is running ? I am trying to reduce the test targets of our current CI and I would like to keep it specific to what we run in prod. [12:52:14] 10serviceops, 10Dumps-Generation, 10Infrastructure-Foundations, 10SRE-tools, and 2 others: Some Service Operations clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271142 (10akosiaris) @Volans All of these (which can be grouped in 2 just 2 categores, **mw** and **mc**, have be... [12:54:18] nemo-yiannis: 12.22.12~dfsg-1~deb11u4 [12:54:33] thanks! [12:56:21] 10serviceops, 10Dumps-Generation, 10Infrastructure-Foundations, 10SRE-tools, and 2 others: Some Service Operations clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271142 (10Volans) @akosiaris sure, and having a cluster deemed as *not* IPv6 ready is totally ok. The problem arise... [13:25:51] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10MW-1.41-notes (1.41.0-wmf.13; 2023-06-13), 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10hnowlan) We recently re-raised concurrency on the thumbnailre... [13:43:30] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10pfischer) > Not right now, but if needed we'll surely be able to create something. It's not blocking us right now, so we cou... [13:47:35] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10pfischer) [14:19:42] 10serviceops, 10Dumps-Generation, 10Infrastructure-Foundations, 10SRE-tools, and 2 others: Some Service Operations clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271142 (10akosiaris) >>! In T271142#9378778, @Volans wrote: > @akosiaris sure, and having a cluster deemed as *not*... [14:41:41] 10serviceops, 10Machine-Learning-Team: Bump istio and Cert Manager Docker images to Bullseye - https://phabricator.wikimedia.org/T351933 (10elukey) [16:07:23] 10serviceops, 10API Platform, 10CirrusSearch, 10MediaWiki-Configuration, 10Discovery-Search (Current work): Provide a method for internal services to run api requests for private wikis - https://phabricator.wikimedia.org/T345185 (10FJoseph-WMF) [19:04:35] 10serviceops, 10API Platform, 10CirrusSearch, 10MediaWiki-Configuration, 10Discovery-Search (Current work): Provide a method for internal services to run api requests for private wikis - https://phabricator.wikimedia.org/T345185 (10aaron) So, without T246371 being done yet, it looks like current apache c... [19:05:18] 10serviceops, 10API Platform, 10CirrusSearch, 10MediaWiki-Configuration, 10Discovery-Search (Current work): Provide a method for internal services to run api requests for private wikis - https://phabricator.wikimedia.org/T345185 (10aaron) a:05aaron→03None [19:52:29] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10EBernhardson) Current plan for gradual deploy is to start with a selection of wikis that add up to ~25% of the total rate. If...