[01:27:20] 10serviceops, 10Performance-Team, 10SRE, 10SRE-swift-storage, and 2 others: Progressive Multi-DC roll out - https://phabricator.wikimedia.org/T279664 (10tstarling) I tried a warmup request followed by another request for the same page view, the second having MW logging enabled with [[https://gerrit.wikimed... [02:09:18] 10serviceops, 10Performance-Team, 10SRE, 10SRE-swift-storage, and 2 others: Progressive Multi-DC roll out - https://phabricator.wikimedia.org/T279664 (10tstarling) I tested parse times with `ab -n10 -H'X-Forwarded-Proto: https' -X mw1441.eqiad.wmnet:80 'http://test2.wikipedia.org/w/api.php?action=parse&for... [02:10:23] 10serviceops, 10Performance-Team, 10SRE, 10SRE-swift-storage, and 2 others: Progressive Multi-DC roll out - https://phabricator.wikimedia.org/T279664 (10tstarling) [05:29:10] 10serviceops, 10Performance-Team, 10SRE: Clean up testwiki experiments - https://phabricator.wikimedia.org/T314750 (10tstarling) [06:35:47] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Properly support php7.4 across the observability stack - https://phabricator.wikimedia.org/T312634 (10Joe) 05Open→03Resolved [06:35:49] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Install php 7.4 in production - https://phabricator.wikimedia.org/T311386 (10Joe) [06:36:32] 10serviceops, 10SRE: eqiad (2) memcached host for wikifunctions service implementation tracking - https://phabricator.wikimedia.org/T313965 (10Joe) [06:42:32] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install new eqiad memcached hosts - https://phabricator.wikimedia.org/T313963 (10Joe) a:05Joe→03None [06:42:56] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install new eqiad memcached hosts - https://phabricator.wikimedia.org/T313963 (10Joe) @RobH all info should be filled in now. [06:43:40] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new codfw memcached hosts - https://phabricator.wikimedia.org/T313966 (10Joe) a:05Joe→03None [06:44:05] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new codfw memcached hosts - https://phabricator.wikimedia.org/T313966 (10Joe) a:03Papaul @RobH the task should be complete with all the info, reassigning to Papaul [06:44:29] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install new eqiad memcached hosts - https://phabricator.wikimedia.org/T313963 (10Joe) a:03Jclark-ctr [06:45:04] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install new eqiad memcached hosts - https://phabricator.wikimedia.org/T313963 (10Joe) [06:45:30] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new codfw memcached hosts - https://phabricator.wikimedia.org/T313966 (10Joe) [11:05:23] kubernetes2022 node is still cordoned and depooled from last weeks PDU swap. I'm going to uncordon and pool the node again as maintenance happened in this rack and the node is up again. [11:18:01] <_joe_> jelto: thanks [14:12:29] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase λ – Launch), 10HTTPS: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Jdforrester-WMF) a:05Jdforrester-WMF→03None [15:02:14] 10serviceops, 10Community-Tech, 10Data-Persistence (Consultation), 10MediaWiki-extensions-Phonos, 10SRE: SRE/Data Persistence consultation — use of FSFileBackend for caching audio files - https://phabricator.wikimedia.org/T314789 (10TheresNoTime) [15:11:16] 10serviceops, 10Community-Tech, 10Data-Persistence (Consultation), 10MediaWiki-extensions-Phonos, 10SRE: SRE/Data Persistence consultation — use of FSFileBackend for caching audio files - https://phabricator.wikimedia.org/T314789 (10MusikAnimal) It's worth mentioning that, like [[ https://www.mediawiki.o... [15:49:22] 10serviceops, 10SRE, 10ops-codfw: decommission mw2251-mw2255, mw2257-mw2258 - https://phabricator.wikimedia.org/T313730 (10Papaul) [15:50:12] 10serviceops, 10SRE, 10ops-codfw: decommission mw2251-mw2255, mw2257-mw2258 - https://phabricator.wikimedia.org/T313730 (10Papaul) 05Open→03Resolved complete [15:59:34] 10serviceops, 10SRE: Migrate node-based services in production to node12 - https://phabricator.wikimedia.org/T290750 (10Michael) [16:00:17] 10serviceops, 10SRE: Migrate node-based services in production to node14 - https://phabricator.wikimedia.org/T306995 (10Michael) [16:00:24] 10serviceops, 10SRE: Migrate node-based services in production to node14 - https://phabricator.wikimedia.org/T306995 (10Michael) [16:01:01] 10serviceops, 10Patch-For-Review: Migrate node-based services in production to node16 - https://phabricator.wikimedia.org/T308371 (10Michael) [16:33:53] 10serviceops, 10Community-Tech, 10Data-Persistence (Consultation), 10MediaWiki-extensions-Phonos, 10SRE: SRE/Data Persistence consultation — use of FSFileBackend for caching audio files - https://phabricator.wikimedia.org/T314789 (10Legoktm) I don't fully understand how FSFileBackend will work here, as t... [16:43:17] did a very basic haproxy container for use in thumbor pods https://gerrit.wikimedia.org/r/821275 [16:52:17] 10serviceops, 10DynamicPageList (Wikimedia), 10SRE, 10Patch-For-Review, and 7 others: Limit concurrency of DPL queries - https://phabricator.wikimedia.org/T263220 (10Krinkle) [17:13:35] hnowlan: just curious, what will haproxy do for a cluster of thumbor pods that a k8s Service object will not? [19:14:40] 10serviceops, 10Performance-Team, 10SRE, 10Patch-For-Review: Clean up testwiki experiments - https://phabricator.wikimedia.org/T314750 (10Krinkle) p:05Triage→03Low a:03tstarling [20:27:13] 10serviceops, 10serviceops-collab, 10GitLab (Infrastructure), 10Patch-For-Review: Document and test failover for GitLab and GitLab Replica - https://phabricator.wikimedia.org/T296713 (10Arnoldokoth) https://gerrit.wikimedia.org/r/c/operations/puppet/+/820563 @LSobanski This was the fix for the SSL issue. T... [23:14:35] 10serviceops, 10Observability-Logging: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy - https://phabricator.wikimedia.org/T313099 (10colewhite) Over half of the logs in the k8s partition come from `-proxy` and `-httpd` containers. istio-proxy appears to generate a similar num... [23:53:08] 10serviceops, 10Performance-Team, 10SRE, 10Patch-For-Review: Clean up testwiki experiments - https://phabricator.wikimedia.org/T314750 (10tstarling) 05Open→03Resolved [23:53:16] 10serviceops, 10Performance-Team, 10SRE, 10SRE-swift-storage, and 2 others: Progressive Multi-DC roll out - https://phabricator.wikimedia.org/T279664 (10tstarling)