[05:29:56] 10serviceops, 10Sustainability (Incident Followup): Repool jobrunners and videoscalers - https://phabricator.wikimedia.org/T336554 (10Joe) a:03Joe [05:42:50] 10serviceops, 10Sustainability (Incident Followup): Repool jobrunners and videoscalers - https://phabricator.wikimedia.org/T336554 (10Joe) The number of servers pooled in jobrunning was just 7, and some servers (mw1458,mw1461,nw1466,mw1495) were depooled from jobrunning but receiving no traffic; This is yet an... [06:39:20] 10serviceops, 10SRE, 10Traffic, 10Platform Team Initiatives (API Gateway): Handle edge cache invalidation for the api gateway - https://phabricator.wikimedia.org/T324200 (10Joe) p:05Triage→03High [07:04:37] 10serviceops, 10SRE, 10Traffic, 10Platform Team Initiatives (API Gateway): Handle edge cache invalidation for the api gateway - https://phabricator.wikimedia.org/T324200 (10Joe) [07:10:13] 10serviceops, 10SRE, 10Traffic, 10Platform Team Initiatives (API Gateway): Handle edge cache invalidation for the api gateway - https://phabricator.wikimedia.org/T324200 (10Joe) My idea for implementing this is as follows: - Create a benthos container - Add a release containing a `Deployment` with N replic... [09:17:09] 10serviceops, 10Patch-For-Review, 10Sustainability (Incident Followup): Repool jobrunners and videoscalers - https://phabricator.wikimedia.org/T336554 (10Joe) 05Open→03Resolved The videoscaling has slowed down, so the current situation is that we have: - All servers but 2 in the jobrunner pool - All ser... [12:42:22] 10serviceops, 10SRE, 10Traffic, 10Platform Team Initiatives (API Gateway): Handle edge cache invalidation for the api gateway - https://phabricator.wikimedia.org/T324200 (10fgiunchedi) >>! In T324200#8846715, @Joe wrote: > My idea for implementing this is as follows: > - Create a benthos container > - Add... [15:20:37] 10serviceops, 10SRE, 10Datacenter-Switchover: Find a way to verify mediawiki-config IPs ahead of datacenter switchovers - https://phabricator.wikimedia.org/T163354 (10Dzahn) [15:50:32] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10daniel) From IRC: > <_joe_> duesen, effie before we enable more jobs, I want us to take a hard look at the jobrunners cpus > <_joe_> it seems we're at 75% util... [15:54:59] 10serviceops, 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 13), 10Patch-For-Review, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10JArguello-WMF) [15:55:20] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 13), 10Patch-For-Review: New Service Request: flink-kubernetes-operator - https://phabricator.wikimedia.org/T333464 (10JArguello-WMF) [15:59:11] 10serviceops, 10Abstract Wikipedia team (Phase θ – Throttling), 10Patch-For-Review: Kubernetes Wikifunctions security and control measures - https://phabricator.wikimedia.org/T326785 (10CodeReviewBot) apine updated https://gitlab.wikimedia.org/repos/abstract-wiki/aw-ci-chart/-/merge_requests/11 Block egress... [16:40:43] 10serviceops, 10SRE: Better handling of memcached service - https://phabricator.wikimedia.org/T255132 (10Dzahn) [16:52:33] 10serviceops, 10Infrastructure-Foundations, 10SRE-tools: Some httpbb checks are flapping - https://phabricator.wikimedia.org/T336590 (10Volans) p:05Triage→03Medium [16:52:52] 10serviceops: Some httpbb checks are flapping - https://phabricator.wikimedia.org/T336590 (10Volans) [17:00:47] 10serviceops, 10SRE: Log the real X-Client-IP in apache mediawiki logs - https://phabricator.wikimedia.org/T246348 (10Dzahn) [17:04:16] ^ I am just tagging tickets because clinic duty asks for that.. tagging subteams on tickets that are currently "SRE" but not more specific. Please just correct me if I am wrong about any. [17:04:36] that's why you will see some old things pop up...triaging [17:04:51] also cool if stuff can be declined nowadays or something [17:23:38] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10Ladsgroup) It's mostly cirrus search though: https://performance.wikimedia.org/arclamp/svgs/daily/2023-05-11.excimer.RunSingleJob.svgz [18:02:55] 10serviceops, 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 13), 10Patch-For-Review, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10Eevans) [19:42:37] 10serviceops, 10DBA, 10Data-Platform-SRE, 10Infrastructure-Foundations, and 9 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10colewhite) [19:59:05] wikibugs: [23:49:38] 10serviceops, 10Arc-Lamp, 10Performance-Team, 10WikimediaDebug, 10Patch-For-Review: Add per-request flamegraph option to WikimediaDebug - https://phabricator.wikimedia.org/T291015 (10Krinkle)