[06:27:12] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10hashar) [08:20:17] FYI I see that there are 100+ warnings in icinga for `PHP opcache health: opcache cache-hit ratio is below 99.99% on php 7.4.` both for eqiad and codfw hosts. [08:24:46] volans: hmm that's concerning [08:24:54] They seem to have all started around the same time [08:25:46] Oldest is 2 hours ago but everything else is 10 minutes or less [08:38:28] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 11), 10Patch-For-Review: New Service Request: flink-kubernetes-operator - https://phabricator.wikimedia.org/T333464 (10JMeybohm) >>! In T333464#8759346, @Ottomata wrote: >> Could you please share resource requirements for the operator... [08:40:30] akosiaris: Yeah, that's what I was thinking; I'll drop you an email about it. Any other people around who might be interested? effie? :) [09:14:14] 10serviceops, 10Wikimedia Enterprise, 10Wikimedia Enterprise Engineering: Handling of connection reset by peer errors in DLQ - https://phabricator.wikimedia.org/T332394 (10Protsack.stephan) [09:19:51] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10Clement_Goubert) [09:19:54] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking), 10Patch-For-Review: Increase memory_limit for jobrunners to $wmgMemoryLimitParsoid - https://phabricator.wikimedia.org/T333528 (10Clement_Goubert) 05In progress→03Resolved [09:29:36] 10serviceops, 10Add-Link, 10Growth-Team, 10MW-on-K8s: Migrate linkrecommendations to mw-api-int - https://phabricator.wikimedia.org/T334060 (10Clement_Goubert) [09:30:58] 10serviceops, 10Add-Link, 10Growth-Team, 10MW-on-K8s: Migrate linkrecommendations to mw-api-int - https://phabricator.wikimedia.org/T334060 (10Clement_Goubert) @kostajh @Tgr If you could provide me with a way to test that linkrecommendations works correctly after changing api backend, it'd much appreciated :) [09:31:51] 10serviceops, 10SRE: Move Kafka main to the new intermediate PKI CA - https://phabricator.wikimedia.org/T319372 (10elukey) Last steps: * clean up certs in puppet private * verify if any change is needed in deployment-prep [09:32:29] 10serviceops, 10MW-on-K8s, 10Product-Infrastructure-Team-Backlog-Deprecated, 10Push-Notification-Service, 10Patch-For-Review: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10Clement_Goubert) @Tgr @MSantos If you could provide me with a way to test that push-notific... [09:34:35] 10serviceops, 10MW-on-K8s, 10Recommendation-API, 10Patch-For-Review: Migrate recommendation-api to mw-api-int - https://phabricator.wikimedia.org/T334062 (10Clement_Goubert) @leila @ori If you could provide me with a way to test that recommendation-api works correctly after changing api backend, it'd be m... [10:39:34] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10akosiaris) Let me add 1 data point I just figured out. It doesn't look prudent to remove hosts from the parsoid cluster right now. Judging by [codfw parsoid clu... [10:59:49] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10daniel) >>! In T329366#8749828, @Clement_Goubert wrote: > Do we have a way to enable the jobs per wiki, even with pre-generation being globally enabled? This co... [11:00:17] I have a review for some Envoy bits for the rest gateway if anyone has time - mostly just setting headers identical to the way restbase did before (and optionally setting a CSP if not set) https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/890887 [11:01:45] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10daniel) >>! In T329366#8754733, @Kappakayala wrote: > Hi @daniel , as the Svc Ops team is figuring out what needs to be done, I would like to understand the pri... [12:57:00] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10Ladsgroup) Noting that I think (not sure, Daniel can confirm?) this is not going to be enabled for commons and wikidata which is the biggest firehose of edits s... [13:05:42] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate cxserver to mw-api-int - https://phabricator.wikimedia.org/T334204 (10Clement_Goubert) [13:07:36] 10serviceops, 10CX-cxserver, 10MW-on-K8s: Migrate cxserver to mw-api-int - https://phabricator.wikimedia.org/T334204 (10Clement_Goubert) p:05Triage→03Medium @KartikMistry @santhosh @Nikerabbit If you could provide me with a way to test that cxserver works correctly after changing api backend, it'd be muc... [13:08:51] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [13:10:00] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [14:13:21] 10serviceops, 10Machine-Learning-Team, 10SRE: Import and deploy istio 1.15.7 - https://phabricator.wikimedia.org/T334068 (10elukey) ` root@apt2001:/srv/wikimedia# reprepro lsbycomponent istio-cni istio-cni | 1.9.5-1 | bullseye-wikimedia | component/istio195 | amd64 istio-cni | 1.15.7-1 | bullseye-wikimedia... [14:17:01] 10serviceops, 10Machine-Learning-Team, 10SRE, 10Patch-For-Review: Import and deploy istio 1.15.7 - https://phabricator.wikimedia.org/T334068 (10elukey) [14:19:34] 10serviceops, 10MW-on-K8s, 10Push-Notification-Service, 10Patch-For-Review: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10ssastry) [14:20:19] 10serviceops, 10ChangeProp, 10Page Content Service: Mobile-html not purged after action=purge - https://phabricator.wikimedia.org/T333887 (10ssastry) [14:20:33] hnowlan: I'll add myself. Might not happen before tuesday, though [14:21:14] jayme: ack, thanks [14:39:07] Hey, just a heads up I am going to deploy some changes to restbase [15:29:20] 10serviceops, 10Machine-Learning-Team, 10SRE: Import and deploy istio 1.15.7 - https://phabricator.wikimedia.org/T334068 (10elukey) [15:36:56] 10serviceops, 10CX-cxserver, 10MW-on-K8s, 10Patch-For-Review: Migrate cxserver to mw-api-int - https://phabricator.wikimedia.org/T334204 (10KartikMistry) >>! In T334204#8762361, @Clement_Goubert wrote: > @KartikMistry @santhosh @Nikerabbit If you could provide me with a way to test that cxserver works corr... [15:58:38] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10jijiki) @Clement_Goubert, @daniel, and I had a short meeting today, we agreed to have the task description better worded as to provide more context, while the a...