[10:07:36] 06serviceops, 06Traffic: X-Request-Id response header off by 5000 - https://phabricator.wikimedia.org/T407826#11293119 (10hnowlan) [10:25:24] 06serviceops, 06Traffic: X-Request-Id response header off by 5000 - https://phabricator.wikimedia.org/T407826#11293148 (10Clement_Goubert) This is envoy because of tracing: `lang=shell-session # Request apache directly in a mw-debug pod namespace root@wikikube-worker1064:/home/cgoubert# curl -k -v -s -I -H 'Ho... [10:47:36] 06serviceops, 06Traffic: X-Request-Id response header off by 5000 - https://phabricator.wikimedia.org/T407826#11293214 (10Clement_Goubert) We need to set ` request_id_extension: typed_config: "@type": type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig pack_trace_reason: fals... [10:51:50] 06serviceops, 06Traffic: X-Request-Id response header off by 5000 - https://phabricator.wikimedia.org/T407826#11293222 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High a:03Clement_Goubert [12:07:45] 06serviceops, 13Patch-For-Review, 07sre-alert-triage: Alert in need of triage: ProbeDown (instance proxoid:4260) - https://phabricator.wikimedia.org/T407615#11293385 (10jijiki) 05Open→03Resolved a:03jijiki [12:27:27] 06serviceops, 06Traffic, 13Patch-For-Review: X-Request-Id response header off by 5000 - https://phabricator.wikimedia.org/T407826#11293440 (10Clement_Goubert) ` cgoubert@deploy2002:/srv/deployment-charts/charts/mediawiki$ curl -s -I --connect-to en.wikipedia.org:443:mwdebug.svc.eqiad.wmnet:4444 -H'x-request-... [12:40:29] 06serviceops, 06Traffic: X-Request-Id response header off by 5000 - https://phabricator.wikimedia.org/T407826#11293479 (10CDanis) >>! In T407826#11293440, @Clement_Goubert wrote: > Fixed in debug, now to redeploy every service using the mesh... The Mediawiki deployments alone are probably 95%+ of the actual d... [13:20:44] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: Test api rate limiting on staging cluster - https://phabricator.wikimedia.org/T406490#11293609 (10daniel) [13:21:36] 06serviceops, 06MediaWiki-Platform-Team, 07OKR-Work, 13Patch-For-Review: Test api rate limiting on staging cluster - https://phabricator.wikimedia.org/T406490#11293613 (10daniel) a:05daniel→03Clement_Goubert [14:15:12] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.3 production workload migration - https://phabricator.wikimedia.org/T405955#11293944 (10Scott_French) [14:58:46] 06serviceops, 10RESTBase-API, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863 (10Ottomata) 03NEW [14:59:41] 06serviceops, 10RESTBase-API, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11294119 (10Ottomata) I much prefer option C, but I'm not sure difficult that might be, or if is setting to strong of a... [15:01:13] 06serviceops, 10RESTBase-API, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11294146 (10Ottomata) p:05Triage→03High [15:01:35] 06serviceops, 06MW-Interfaces-Team, 10RESTBase-API, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11294148 (10Ottomata) [15:04:01] 06serviceops, 06MW-Interfaces-Team, 10RESTBase-API, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11294152 (10Ottomata) [15:05:08] Hi! ^ has a bit of urgency for ongoing OKR work. Would appreciate some attention from serviceops ASAP. Thank you! [15:05:08] https://phabricator.wikimedia.org/T407863 [15:06:53] 06serviceops, 06MW-Interfaces-Team, 10RESTBase-API, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11294176 (10Ottomata) a:03Ottomata [15:14:58] 06serviceops, 06MW-Interfaces-Team, 10RESTBase-API, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11294203 (10Ottomata) In order to expedite this work, I am considering implementing versioning... [16:36:27] 06serviceops, 06Infrastructure-Foundations, 10Mail, 06SRE: Sendmail network error (deployment) - https://phabricator.wikimedia.org/T407723#11294776 (10Dzahn) [18:41:31] 06serviceops, 06MW-Interfaces-Team, 10RESTBase-API, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11295513 (10Ottomata) I tried ^, and I can make it work, but I cannot get AQS to auto-generate... [18:55:12] 06serviceops, 06MW-Interfaces-Team, 10RESTBase-API, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: AQS Wikimedia REST API - new API version - https://phabricator.wikimedia.org/T407863#11295578 (10Ottomata) [19:10:50] 06serviceops, 10MediaWiki-Core-AuthManager, 10MediaWiki-Documentation, 10MediaWiki-extensions-OATHAuth, and 3 others: Document best practices for user login if user is using 2FA - https://phabricator.wikimedia.org/T255370#11295648 (10TBurmeister) a:03TBurmeister [19:11:20] 06serviceops, 10MediaWiki-Core-AuthManager, 10MediaWiki-Documentation, 10MediaWiki-extensions-OATHAuth, and 3 others: Document best practices for user login if user is using 2FA - https://phabricator.wikimedia.org/T255370#11295653 (10TBurmeister) [20:36:36] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11296077 (10Scott_French) I'd like to get this moving again before we get too far into Q2. All that remains is to switch etcd over to cfssl-based PKI (i.e., for etcd <> etcd and etcd... [23:00:21] 06serviceops, 06Traffic: X-Request-Id response header off by 5000 - https://phabricator.wikimedia.org/T407826#11296564 (10RLazarus) This is deployed to all services. [23:27:42] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 10RESTBase Sunsetting, and 2 others: Reroute RESTBase /page/lint/ endpoints to MediaWiki REST endpoints - https://phabricator.wikimedia.org/T384216#11296641 (10Krinkle)