[00:15:39] 10serviceops, 10SRE: Request to block ActionApi client (based on a specific user agent header) - https://phabricator.wikimedia.org/T243858 (10Dzahn) [01:46:48] 10serviceops, 10SRE: Nutcracker stats monitoring should only listen on localhost - https://phabricator.wikimedia.org/T111934 (10Dzahn) [01:48:32] 10serviceops, 10SRE, 10observability: Nutcracker stats monitoring should only listen on localhost - https://phabricator.wikimedia.org/T111934 (10Dzahn) [09:24:58] 10serviceops, 10Wikidata, 10Wikidata-Query-Service, 10Wikidata.org, and 3 others: [SW] Query service maxlag calculation should exclude datacenters that don't receive traffic and where the updater is turned off - https://phabricator.wikimedia.org/T331405 (10ItamarWMDE) Created a subtask to track our work on... [09:25:37] 10serviceops, 10Wikidata, 10Wikidata-Query-Service, 10Wikidata.org, and 3 others: Query service maxlag calculation should exclude datacenters that don't receive traffic and where the updater is turned off - https://phabricator.wikimedia.org/T331405 (10ItamarWMDE) [09:32:36] 10serviceops, 10MW-on-K8s, 10Performance-Team (Radar), 10Wikimedia-production-error: ResourceLoader icon rasterization fails via MediaWiki-on-Kubernetes - https://phabricator.wikimedia.org/T336025 (10Joe) Indeed. The code for resourceloader wants to use `rsvg-convert` and do it without going through shellb... [09:39:10] 10serviceops, 10MW-on-K8s, 10Performance-Team (Radar), 10Wikimedia-production-error: ResourceLoader icon rasterization fails via MediaWiki-on-Kubernetes - https://phabricator.wikimedia.org/T336025 (10Joe) For reference, the code in question is in `Resourceloader\Image::rasterize`: https://github.com/wikim... [09:40:17] 10serviceops, 10MW-on-K8s, 10Performance-Team (Radar), 10Wikimedia-production-error: ResourceLoader icon rasterization fails via MediaWiki-on-Kubernetes - https://phabricator.wikimedia.org/T336025 (10Joe) So, in order to get this working, we would need to install `rsvg-convert` in the base mediawiki image... [09:43:11] 10serviceops, 10wikitech.wikimedia.org: Create template on Wikitech for documenting production services - https://phabricator.wikimedia.org/T336354 (10kostajh) [10:19:08] 10serviceops, 10wikitech.wikimedia.org, 10Documentation: Create template on Wikitech for documenting production services - https://phabricator.wikimedia.org/T336354 (10Aklapper) [10:20:06] 10serviceops, 10Documentation: Create template on Wikitech for documenting production services - https://phabricator.wikimedia.org/T336354 (10taavi) [10:46:54] anyone have cycles free for some thumbor/haproxy changes? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/916506 [10:54:44] hnowlan: looking, with the usual disclaimer that I don't actually have all the context [11:07:09] kamila_: thanks! [11:08:03] hnowlan: np, happy to provide my marginally-better-than-a-coinflip services anytime :D [11:12:01] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10jijiki) [12:48:23] 10serviceops, 10Service-deployment-requests: New Service Request 'iPoid' - https://phabricator.wikimedia.org/T325147 (10jijiki) [13:44:51] 10serviceops, 10MW-on-K8s, 10Performance-Team (Radar), 10Wikimedia-production-error: ResourceLoader icon rasterization fails via MediaWiki-on-Kubernetes - https://phabricator.wikimedia.org/T336025 (10jijiki) a:03jijiki [14:03:45] 10serviceops, 10MW-on-K8s, 10Performance-Team (Radar), 10Wikimedia-production-error: ResourceLoader icon rasterization fails via MediaWiki-on-Kubernetes - https://phabricator.wikimedia.org/T336025 (10Krinkle) @Joe Thanks! If it's any concellation, these are not user-generated SVGs. They're only ~1KB SVGs... [14:05:58] 10serviceops, 10MW-on-K8s, 10Performance-Team (Radar), 10Wikimedia-production-error: ResourceLoader icon rasterization fails via MediaWiki-on-Kubernetes - https://phabricator.wikimedia.org/T336025 (10Joe) >>! In T336025#8841663, @Krinkle wrote: > @Joe Thanks! > > If it's any concellation, these are not us... [14:18:26] We're planning on updating admin_ng for production wikikube. Does anyone have time to review https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/917935 and/or suggestions on deploying? [14:27:14] <_joe_> inflatador: not right now, how urgent is it? [14:51:08] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools, 10serviceops-collab: Create a cookbook to automate gerrit's switchover - https://phabricator.wikimedia.org/T260666 (10LSobanski) [15:06:14] _joe_ I think we've stopped the bleeding for now...probably within the next wk or 2 is OK [16:41:57] 10serviceops, 10serviceops-collab, 10Release-Engineering-Team (They Live 🕶️🧟): Gitlab downtime blocking scap backport - https://phabricator.wikimedia.org/T336162 (10thcipriani) [20:03:06] 10serviceops, 10SRE, 10Scap, 10Release-Engineering-Team (Seen): Enable scap to roll back broken changes to MediaWiki - https://phabricator.wikimedia.org/T225207 (10Dzahn) [20:09:02] 10serviceops, 10SRE, 10observability: stop using $::site in description field of service.yaml - https://phabricator.wikimedia.org/T258697 (10Dzahn) [21:04:33] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase λ – Launch), 10HTTPS: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10BCornwall) 05In progress→03Resolved Resolving since it appears to...