[03:52:07] _joe_: i take it back again, it does work! I set the wrong timeout setting in service_proxies. needed to set keepalive. Looking good! [03:52:07] https://grafana.wikimedia.org/goto/I_f6pq4Ik?orgId=1 [08:29:20] 10serviceops, 10iPoid-Service, 10Trust and Safety Product Sprint (Sprint Bodhrán): [M] Write CronJob configuration - https://phabricator.wikimedia.org/T346861 (10jijiki) [08:34:14] 10serviceops, 10iPoid-Service, 10Trust and Safety Product Sprint (Sprint Bodhrán): [M] Write CronJob configuration - https://phabricator.wikimedia.org/T346861 (10jijiki) 05In progress→03Resolved Fixed, you will need to run helmfile apply for this change to take effect. [08:41:53] 10serviceops, 10SRE: Rebuild PHP 7.4 packages for Bullseye - https://phabricator.wikimedia.org/T350767 (10MoritzMuehlenhoff) [09:31:10] o/ would someone know what component could set "User-Agent": "MediaWiki/1.42.0-wmf.4" in http requests made to MW? [09:32:45] investigating search requests that have this UA and wondering from where they could come from, I suspect that it's not the actual client setting this UA but rather something in-between [09:34:02] the search requests themselves have nothing in particular, they can be api.php or Special:Search [09:35:14] dcausse: o/ could it be coming from a jobrunner? [09:35:22] (in response to a specific event) [09:36:19] in Lift Wing we see UAs like those when we get requests triggered by the Ores extension for example [09:36:21] elukey: the client_ip seems to be external so no? [09:36:58] dcausse: ah ok this detail was not clear :D [09:40:56] yes I can confirm I see only interal ips.. dcausse do you have an example that we can check? [09:42:02] elukey: select geocoded_data, params, http, source `database` from event.mediawiki_cirrussearch_request where params['action'] is null and lower(http.request_headers['user-agent']) like '%mediawiki%' and datacenter = 'eqiad' and year='2023' and month = 11 and day = 9 and hour = 18 LIMIT 100; [09:42:25] will try to find correlated requests from the weblogs [09:43:27] dcausse: can you share a paste with wmf-nda or similar with one/two results? [09:43:39] sure [10:03:13] 10serviceops, 10SRE: Rebuild PHP 7.4 packages for Bullseye - https://phabricator.wikimedia.org/T350767 (10MoritzMuehlenhoff) [10:08:33] 10serviceops, 10iPoid-Service, 10Patch-For-Review, 10Service-deployment-requests, 10Trust and Safety Product Sprint: New Service Request 'iPoid' - https://phabricator.wikimedia.org/T325147 (10kostajh) [10:09:24] 10serviceops, 10iPoid-Service, 10Patch-For-Review, 10Trust and Safety Product Sprint (Sprint Bodhrán): [M] Implement proxy configuration for kubernetes deployment - https://phabricator.wikimedia.org/T349171 (10kostajh) 05Open→03Resolved [10:26:29] 10serviceops, 10SRE: Rebuild PHP 7.4 packages for Bullseye - https://phabricator.wikimedia.org/T350767 (10MoritzMuehlenhoff) [10:27:50] 10serviceops, 10iPoid-Service, 10Kubernetes: Create helm chart for iPoid - https://phabricator.wikimedia.org/T336163 (10CodeReviewBot) stran merged https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/175 config: Use correct environment variables for MySQL password [11:22:17] 10serviceops, 10SRE: Rebuild PHP 7.4 packages for Bullseye - https://phabricator.wikimedia.org/T350767 (10MoritzMuehlenhoff) [12:31:36] 10serviceops, 10SRE: Rebuild PHP 7.4 packages for Bullseye - https://phabricator.wikimedia.org/T350767 (10MoritzMuehlenhoff) [13:27:51] 10serviceops, 10SRE: Rebuild PHP 7.4 packages for Bullseye - https://phabricator.wikimedia.org/T350767 (10MoritzMuehlenhoff) [13:49:10] 10serviceops, 10API Platform (RESTbase Deprecation Roadmap), 10Patch-For-Review: Migrate node-based services in production to node16 - https://phabricator.wikimedia.org/T308371 (10lbowmaker) [13:49:17] 10serviceops, 10SRE, 10API Platform (RESTbase Deprecation Roadmap), 10Patch-For-Review: Migrate node-based services in production to node14 - https://phabricator.wikimedia.org/T306995 (10lbowmaker) [13:49:25] 10serviceops, 10CX-cxserver, 10Citoid, 10Content-Transform-Team-WIP, and 9 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118 (10lbowmaker) [13:49:31] 10serviceops, 10ChangeProp, 10EventStreams, 10Image-Suggestion-API, and 5 others: Migrate node-based services in production to node12 - https://phabricator.wikimedia.org/T290750 (10lbowmaker) [13:49:39] 10serviceops, 10Data-Engineering, 10Data Engineering and Event Platform Team (Sprint 4), 10Event-Platform: [Event Platform] Gracefully handle pod termination in eventgate Helm chart - https://phabricator.wikimedia.org/T349823 (10lbowmaker) 05Open→03Resolved [13:53:14] 10serviceops, 10Data-Engineering, 10Data Engineering and Event Platform Team (Sprint 4), 10Event-Platform: [Event Platform] eventgate-wikimedia occasionally fails to produce events due schema fetch errors - https://phabricator.wikimedia.org/T350713 (10Ottomata) Okay, it turns out I set the wrong value for... [14:13:37] 10serviceops, 10Data-Engineering (Sprint 5), 10Event-Platform: [Event Platform] eventgate-wikimedia occasionally fails to produce events due schema fetch errors - https://phabricator.wikimedia.org/T350713 (10lbowmaker) [14:29:59] 10serviceops, 10Data-Engineering, 10Data-Platform-SRE, 10SRE, 10Event-Platform: DRY kafka broker declaration in helmfiles - https://phabricator.wikimedia.org/T253058 (10lbowmaker) [14:30:13] 10serviceops, 10Data-Engineering, 10Discovery-Search (Current work), 10Event-Platform, 10Patch-For-Review: Improve the flink-app chart to provide more useful defaults - https://phabricator.wikimedia.org/T346315 (10lbowmaker) [14:31:53] 10serviceops, 10Data-Engineering, 10Data-Platform-SRE, 10SRE, and 3 others: Upgrade Kafka to 2.x or 3.x - https://phabricator.wikimedia.org/T300102 (10lbowmaker) [14:32:16] 10serviceops, 10Data-Engineering, 10SRE-OnFire, 10Event-Platform: Incident: 2022-12-09 api appserver worker starvation - https://phabricator.wikimedia.org/T324994 (10lbowmaker) [14:32:24] 10serviceops, 10Data-Engineering, 10Data-Platform-SRE, 10SRE-OnFire, and 3 others: Uneven CPU throttling of eventgate-analytics under load - https://phabricator.wikimedia.org/T325068 (10lbowmaker)