[01:47:28] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list April 2025 - https://phabricator.wikimedia.org/T392868 (10GFontenelle_WMF) 03NEW [03:32:09] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Commons-Impact-Metrics, 13Patch-For-Review: [CIM] Skewed ranking with the top Editors monthly API - https://phabricator.wikimedia.org/T370470#10775110 (10EChukwukere-WMF) Test status: //**QA PASS**// Checked the ranking Fix in production and this has bee... [07:16:02] 10Data-Engineering (Q4 2025 April 1st - June 30th): Some search entries in wmf.webrequest have their query appended to their uri_path - https://phabricator.wikimedia.org/T383135#10775330 (10Antoine_Quhen) 05Open→03Declined I don't think we should alter the current value of the column uri_query. The creat... [08:12:26] 10Data-Engineering (Q4 2025 April 1st - June 30th): Some search entries in wmf.webrequest have their query appended to their uri_path - https://phabricator.wikimedia.org/T383135#10775454 (10AndrewTavis_WMDE) I don't have any major preference on this being done, so if it's unlikely to be prioritized then it c... [08:25:21] 06Data-Engineering, 10Data-Platform-SRE (2025-04-12 - 2025-05-02), 13Patch-For-Review, 10Sustainability (Incident Followup): airflow: Consider restricting the rights for airflow deployers to destroy postgresql clusters - https://phabricator.wikimedia.org/T391348#10775484 (10brouberol) 05In progress→... [09:03:48] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop afl_patrolled_by from abuse_filter_log in production - https://phabricator.wikimedia.org/T391056#10775581 (10Marostegui) >>! In T391056#10772395, @Ladsgroup wrote: > We could now do masters (except s1/s4/s8) Let's add... [09:51:29] (03PS1) 10Lucas Werkmeister (WMDE): Don’t send targetDate to Prometheus in minutely metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139800 (https://phabricator.wikimedia.org/T389344) [09:56:11] (03PS1) 10Lucas Werkmeister (WMDE): Rename site label to site_id in Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139801 (https://phabricator.wikimedia.org/T389344) [10:00:23] (03PS1) 10Lucas Werkmeister (WMDE): Remove current date targetDate from Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139802 (https://phabricator.wikimedia.org/T389344) [10:47:24] (03CR) 10Filippo Giunchedi: [C:03+1] Don’t send targetDate to Prometheus in minutely metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139800 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:47:36] (03CR) 10Filippo Giunchedi: [C:03+1] Rename site label to site_id in Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139801 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:48:04] (03CR) 10Filippo Giunchedi: [C:03+1] Remove current date targetDate from Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139802 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:54:04] (03CR) 10Hasan Akgün (WMDE): [C:03+2] Don’t send targetDate to Prometheus in minutely metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139800 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:54:05] (03CR) 10Hasan Akgün (WMDE): [C:03+2] Rename site label to site_id in Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139801 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:54:07] (03CR) 10Hasan Akgün (WMDE): [C:03+2] Remove current date targetDate from Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139802 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:54:40] (03Merged) 10jenkins-bot: Don’t send targetDate to Prometheus in minutely metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139800 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:54:42] (03Merged) 10jenkins-bot: Rename site label to site_id in Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139801 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:54:50] (03Merged) 10jenkins-bot: Remove current date targetDate from Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139802 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:56:11] (03PS1) 10Lucas Werkmeister (WMDE): Don’t send targetDate to Prometheus in minutely metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139813 (https://phabricator.wikimedia.org/T389344) [10:56:18] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] Don’t send targetDate to Prometheus in minutely metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139813 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:56:26] (03PS1) 10Lucas Werkmeister (WMDE): Rename site label to site_id in Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139814 (https://phabricator.wikimedia.org/T389344) [10:56:32] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] Rename site label to site_id in Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139814 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:56:40] (03PS1) 10Lucas Werkmeister (WMDE): Remove current date targetDate from Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139816 (https://phabricator.wikimedia.org/T389344) [10:56:47] (03Merged) 10jenkins-bot: Don’t send targetDate to Prometheus in minutely metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139813 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:56:48] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] Remove current date targetDate from Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139816 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:56:56] (03Merged) 10jenkins-bot: Rename site label to site_id in Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139814 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [10:57:15] (03Merged) 10jenkins-bot: Remove current date targetDate from Prometheus metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139816 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [13:28:01] btullis, brouberol: headsup, I'll merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139850 tomorrow. this is our first KDC on Bookworm (I've done smoke tests which were fine), but if there's anything else popping up related to auth tomorrow, let me know [13:28:42] (03PS1) 10Lucas Werkmeister (WMDE): Send entity usage aspect with modifier combined to Prometheus [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139854 (https://phabricator.wikimedia.org/T389344) [13:29:00] (03CR) 10CI reject: [V:04-1] Send entity usage aspect with modifier combined to Prometheus [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139854 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [13:29:16] moritzm: Great, thanks. We might need to make sure that this is also available to the Kerberos clients in K8S as well. [13:29:19] moritzm: b=great thanks! [13:29:31] (03PS2) 10Lucas Werkmeister (WMDE): Send entity usage aspect with modifier combined to Prometheus [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139854 (https://phabricator.wikimedia.org/T389344) [13:30:01] btullis: indeed, that's what I'll be looking at next [13:30:16] * btullis Oh, it should be fine, as long as they are in the same role. https://github.com/wikimedia/operations-puppet/blob/production/modules/profile/manifests/kubernetes/deployment_server/global_config.pp#L213-L229 [13:30:48] oh, perfect. yes, they are using the same role on both distros [13:31:42] Great. We will be on the lookout for anything weird with Kerberos tomorrow then. [13:36:50] (03CR) 10Filippo Giunchedi: [C:03+1] Send entity usage aspect with modifier combined to Prometheus [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139854 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [13:47:05] 06Data-Engineering, 06Discovery-Search, 06Infrastructure-Foundations, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10776547 (10elukey) >>! In T390860#10765552, @bking wrote: > As far as our future plans, we haven't... [13:49:31] Can I find the browser rollup data that we publish to analytics.wikimedia.org through Hive/Spark as well? It seems that it would be in "browser_general" but that table appears to be empty. [14:03:58] 06Data-Engineering, 06Traffic, 10DPE HAProxy Migration: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10776594 (10Fabfur) @JAllemandou the change has been deployed in production, now all haproxykafka instances on cache hosts are sending the `termination_state` field... [14:04:15] 06Data-Engineering, 06Traffic, 10DPE HAProxy Migration: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10776600 (10Fabfur) [14:07:22] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Create Airflow pipeline to produce wmf_content.mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T391283#10776618 (10xcollazo) Grafana shows [[ https://grafana.wikimedia.org/d/000000585/hadoop?orgId=1&from=1745926238645... [14:42:32] 06Data-Engineering, 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list April 2025 - https://phabricator.wikimedia.org/T392868#10776786 (10mforns) Reveiwed and merged! Since the migration of Airflow to Kubernetes, the merge triggers the deployment automatical... [15:44:14] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] "Gonna go ahead and self-merge this so it’s deployed in time for the next daily run." [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139854 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [15:44:21] (03PS1) 10Lucas Werkmeister (WMDE): Send entity usage aspect with modifier combined to Prometheus [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139890 (https://phabricator.wikimedia.org/T389344) [15:45:24] (03Merged) 10jenkins-bot: Send entity usage aspect with modifier combined to Prometheus [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139854 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [15:45:44] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] Send entity usage aspect with modifier combined to Prometheus [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139890 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [15:47:24] (03Merged) 10jenkins-bot: Send entity usage aspect with modifier combined to Prometheus [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1139890 (https://phabricator.wikimedia.org/T389344) (owner: 10Lucas Werkmeister (WMDE)) [15:48:24] (03PS2) 10Lucas Werkmeister (WMDE): Remove Facebook, IRC and Twitter social metrics [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/734898 (https://phabricator.wikimedia.org/T294014) [15:49:01] (03CR) 10Lucas Werkmeister (WMDE): "Rebased. They’re not sending any metrics to Prometheus either (according to Thanos)." [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/734898 (https://phabricator.wikimedia.org/T294014) (owner: 10Lucas Werkmeister (WMDE)) [15:54:55] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for madalina - https://phabricator.wikimedia.org/T392893#10777075 (10tappof) [16:00:42] 06Data-Engineering, 10Community-Tech (Sea Lion Squad), 07PostgreSQL, 07SQLite: SQLite/PostgreSQL native upsert - https://phabricator.wikimedia.org/T392786#10777092 (10Ladsgroup) Maybe I'm wrong but if I remember correctly these upsert have been written when the native way didn't exist or was not in support... [16:46:34] 06Data-Engineering, 06Discovery-Search, 06Infrastructure-Foundations, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10777498 (10bking) >>! In T390860#10776547, @elukey wrote: >>>! In T390860#10765552, @bking wrote: >... [17:36:05] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for madalina - https://phabricator.wikimedia.org/T392893#10777736 (10Ahoelzl) Approved. [17:36:34] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for madalina - https://phabricator.wikimedia.org/T392893#10777738 (10Ahoelzl) [17:36:42] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list April 2025 - https://phabricator.wikimedia.org/T392868#10777740 (10Ahoelzl) [17:37:56] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Post-Migration Cleanup - https://phabricator.wikimedia.org/T392698#10777746 (10Ahoelzl) [17:38:03] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Update Refine Documentation on Wikitech - https://phabricator.wikimedia.org/T392697#10777748 (10Ahoelzl) [17:38:13] 10Data-Engineering (Q4 2025 April 1st - June 30th): Refine to Hive with Airflow – Switch Over day scripts - https://phabricator.wikimedia.org/T392696#10777750 (10Ahoelzl) [17:38:56] 10Analytics-Canonical-Data, 06Movement-Insights, 06Security-Team, 06WMF-Legal, 07SecTeam-Processed: Confirm stewardship of the Country and Territory Protection List - https://phabricator.wikimedia.org/T381944#10777768 (10nshahquinn-wmf) @sbassett @MHedlund-WMF do you know what came out of @acooper's meet... [17:39:32] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team, 06serviceops: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10777776 (10Scott_French) [17:45:15] 06Data-Engineering, 06Data-Engineering-Radar, 06Discovery-Search, 06Infrastructure-Foundations, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10777796 (10Ahoelzl) [17:47:30] 07Analytics-Data-Problem, 06Data-Engineering: Strange statistics for language variants for languages other than zh and sr - https://phabricator.wikimedia.org/T392624#10777815 (10Ahoelzl) @OSefu-WMF can you help assess the effort, impact and prioritize? [17:47:41] 07Analytics-Data-Problem, 06Data-Engineering, 06Data-Engineering-Radar: Strange statistics for language variants for languages other than zh and sr - https://phabricator.wikimedia.org/T392624#10777818 (10Ahoelzl) [17:49:06] 06Data-Engineering, 06cloud-services-team, 10Cloud-VPS, 06Data-Platform-SRE, and 2 others: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10777833 (10Ahoelzl) [17:50:11] 06Data-Engineering, 06Data-Platform-SRE, 06Product-Analytics: Allow curl commands from Airflow BashOperator - https://phabricator.wikimedia.org/T392288#10777836 (10Ahoelzl) [17:51:29] 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: [MinIO] Investigate load-balancing approaches to eliminate SPOF - https://phabricator.wikimedia.org/T392249#10777839 (10Ahoelzl) [17:52:39] 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: [Trino] additional worker nodes for eqiad - https://phabricator.wikimedia.org/T392131#10777841 (10Ahoelzl) [17:52:49] 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: [MinIO] Improve cluster to minimum system configuration for production - https://phabricator.wikimedia.org/T392112#10777848 (10Ahoelzl) [17:53:02] 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: Trino implementation - https://phabricator.wikimedia.org/T392093#10777849 (10Ahoelzl) [17:53:10] 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: EPIC: MinIO implementation - https://phabricator.wikimedia.org/T392090#10777850 (10Ahoelzl) [17:54:17] 06Data-Engineering-Radar, 06Data-Platform-SRE, 06Traffic, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Replicate current low-message alerting from VarnishKafka - https://phabricator.wikimedia.org/T391810#10777852 (10Ahoelzl) [17:55:24] 06Data-Engineering, 10Event-Platform: Bug: event validation error: mediawiki.page-restrictions-change - https://phabricator.wikimedia.org/T390012#10777865 (10Ahoelzl) @gmodena can you take a look and assess? [17:56:58] 06Data-Engineering, 10MediaWiki-Page-rename, 10Event-Platform, 07Wikimedia-production-error: InvalidArgumentException: No page moved from 'File:Hospital sign.svg' to 'File:MUTCD D9-2.svg' with ID 1208082 could be found - https://phabricator.wikimedia.org/T387695#10777871 (10Ahoelzl) @gmodena can you take a... [17:59:59] 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE: Jupyter spawner's list of Conda environments only updates when an environment is spawned - https://phabricator.wikimedia.org/T391894#10777890 (10Ahoelzl) [18:02:46] 06Data-Engineering: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10777896 (10Ahoelzl) @mforns can you take a look and assesss? [18:06:57] 10Data-Engineering (Q4 2025 April 1st - June 30th): Duplicate revisions and excess reverts in 2025-03 MediaWiki History snapshot - https://phabricator.wikimedia.org/T391708#10777901 (10nshahquinn-wmf) [18:07:14] 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th): Duplicate revisions and excess reverts in 2025-03 MediaWiki History snapshot - https://phabricator.wikimedia.org/T391708#10777902 (10nshahquinn-wmf) [18:38:13] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Create Airflow pipeline to produce wmf_content.mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T391283#10778013 (10xcollazo) For some reason the MR did not link, but we made the changes from T391283#10773426 permanent... [18:40:13] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Document wmf_content.mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T392948 (10xcollazo) 03NEW