[14:51:01] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11327600 (10Scott_French) [15:35:16] 06serviceops, 10MediaWiki-Core-AuthManager, 10MediaWiki-Documentation, 10MediaWiki-extensions-OATHAuth, and 3 others: Document best practices for user login if user is using 2FA - https://phabricator.wikimedia.org/T255370#11327895 (10JTweed-WMF) [16:17:02] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#11328214 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jasmine@cumin1002 pool for host wikikube-worker[2248-2267].codfw.wmnet completed: - wikikube-worker[2248-2267].codfw.w... [16:32:48] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#11328370 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jasmine@cumin1002 depool for host wikikube-worker[2003-2004,2007-2010,2019-2032].codfw.wmnet completed: - wikikube-wor... [16:44:09] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#11328499 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jasmine@cumin1002 pool for host wikikube-worker[2268-2287].codfw.wmnet completed: - wikikube-worker[2268-2287].codfw.w... [16:54:55] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker11XX - https://phabricator.wikimedia.org/T408749#11328564 (10Clement_Goubert) [16:58:25] 06serviceops, 06DC-Ops, 10ops-eqiad, 10procurement, 06SRE: Q2:rack/setup/install wikikube-worker refresh - https://phabricator.wikimedia.org/T408760#11328592 (10Clement_Goubert) [16:58:56] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install wikikube-worker1335-59 - https://phabricator.wikimedia.org/T408752#11328593 (10Clement_Goubert) [17:00:45] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#11328599 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jasmine@cumin1002 depool for host wikikube-worker[2040,2043,2045,2048,2052-2054,2063,2079-2084,2096-2101].codfw.wmnet... [17:01:44] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#11328604 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jasmine@cumin1002 pool for host wikikube-worker[2288-2299].codfw.wmnet completed: - wikikube-worker[2288-2299].codfw.w... [17:03:35] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: api-gateway chart: improve metrics mapping - https://phabricator.wikimedia.org/T408183#11328613 (10daniel) a:03daniel [17:03:49] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work: api rate limiting: Create a helper service for exposing per-user (per-bucket) metrics - https://phabricator.wikimedia.org/T407999#11328616 (10daniel) 05Open→03In progress [17:04:41] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work: Test api rate limiting on production cluster - https://phabricator.wikimedia.org/T406498#11328632 (10daniel) [17:04:45] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: Test api rate limiting on staging cluster - https://phabricator.wikimedia.org/T406490#11328633 (10daniel) [17:09:29] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#11328667 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jasmine@cumin1002 depool for host wikikube-worker[2230-2241].codfw.wmnet completed: - wikikube-worker[2230-2241].codfw... [17:09:33] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work: api-gateway chart: make emission of x-ratelimit headers configurable - https://phabricator.wikimedia.org/T408839 (10daniel) 03NEW [17:12:16] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#11328701 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jasmine@cumin1002 pool for host wikikube-worker[2300-2319].codfw.wmnet completed: - wikikube-worker[2300-2319].codfw.w... [17:12:56] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#11328704 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jasmine@cumin1002 pool for host wikikube-worker[2320-2330].codfw.wmnet completed: - wikikube-worker[2320-2330].codfw.w... [17:27:35] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#11328790 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jasmine@cumin1002 depool for host wikikube-worker[2116-2123,2216-2230].codfw.wmnet completed: - wikikube-worker[2116-2... [17:36:38] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: Test api rate limiting on staging cluster - https://phabricator.wikimedia.org/T406490#11328829 (10daniel) 05Open→03Resolved Completes successfully with one caveat. Test plan and protocol at https://docs.google.com/document/d/1... [17:37:49] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: Test api rate limiting on staging cluster - https://phabricator.wikimedia.org/T406490#11328855 (10daniel) [17:38:20] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: Test api rate limiting on staging cluster - https://phabricator.wikimedia.org/T406490#11328856 (10daniel) [17:41:33] 06serviceops, 06MediaWiki-Platform-Team, 13Patch-For-Review: Set up a rest-gateway deployment for rate limiting testing - https://phabricator.wikimedia.org/T402914#11328870 (10daniel) 05In progress→03Declined No longer needed, we successfully tested on the staging cluster, see T406490 [17:45:50] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11328889 (10Scott_French) As of 17:20 - 17:40 UTC we're enrolling 50% of eligible cookie-enabled external clients and serving 25% of mw-api-int / mw-jobrunner traffic on PHP 8.3. Sim... [17:48:23] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11328893 (10Scott_French) [20:46:07] 06serviceops, 10envoy, 06SRE, 13Patch-For-Review: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663#11329609 (10RLazarus) 05In progress→03Resolved