[07:01:25] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10542626 (10daniel) >>! In T379935#10542045, @Ottomata wrote: > I have a feeling that int... [07:15:00] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10542632 (10daniel) >>! In T379935#10541716, @Ottomata wrote: > The EventBus based JobQue... [08:08:33] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Discovery-Search, 10Dumps 2.0, 10Data-Platform-SRE (2025.02.10 - 2025.02.28), 13Patch-For-Review: Add relevant kafka clusters to defined airflow connections in puppet - https://phabricator.wikimedia.org/T379676#10542669 (10brouberol) I'm not too fa... [08:11:34] 06Data-Engineering, 06Data-Engineering-Radar, 10Research-engineering, 10Data-Platform-SRE (2025.02.10 - 2025.02.28): Requesting Ceph S3 credentials for research - https://phabricator.wikimedia.org/T385608#10542677 (10Gehel) Just note that how we use Ceph is still a work in progress. Feel free to experiment... [08:30:26] 14Analytics-Radar, 10CirrusSearch, 10MediaWiki-Core-JobQueue, 10Event-Platform: Expose a metric that reflect EventBus queue pressure - https://phabricator.wikimedia.org/T190416#10542764 (10Gehel) [08:31:35] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10CirrusSearch: Load cirrussearch data into druid - https://phabricator.wikimedia.org/T156037#10542792 (10Gehel) [08:33:33] 14Analytics, 10ChangeProp, 10Elasticsearch, 10Event-Platform: Port CirrusSearch update JobQueue jobs to EventBus - https://phabricator.wikimedia.org/T150283#10542856 (10Gehel) [08:36:36] 14Data-Engineering (Q2 2024 October 1st - December 31th): Create and distribute a flink base image with flink 1.20.0 - https://phabricator.wikimedia.org/T377134#10542951 (10Gehel) [08:37:29] 06Data-Engineering, 06Java-Scala-Standardization, 10Metrics Platform, 06Release-Engineering-Team: Adapt gitlab pipelines for the new wmf-jvm-parent-pom - https://phabricator.wikimedia.org/T358841#10542978 (10Gehel) [08:38:05] 06Data-Engineering, 07Epic, 10Event-Platform: [Epic] Set up multi DC Kafka stretch cluster - https://phabricator.wikimedia.org/T340492#10542989 (10Gehel) [08:40:34] 06Data-Engineering, 14Data-Engineering-Kanban, 10Data Pipelines, 06Product-Analytics, and 3 others: Write an Airflow job converting commons structured data dump to Hive - https://phabricator.wikimedia.org/T299059#10543043 (10Gehel) [10:12:47] 06Data-Engineering, 06Data-Platform-SRE: Grow kafka partition number for topics `webrequest_frontend_text` and `webrequest_frontend_upload` - https://phabricator.wikimedia.org/T386173 (10JAllemandou) 03NEW [10:15:44] 06Data-Engineering, 06Data-Platform-SRE: Grow number of Gobblin mappers ingesting `webrequest_frontend` data - https://phabricator.wikimedia.org/T386174 (10JAllemandou) 03NEW [10:16:05] 06Data-Engineering, 06Data-Platform-SRE: Grow kafka partition number for topics `webrequest_frontend_text` and `webrequest_frontend_upload` - https://phabricator.wikimedia.org/T386173#10543554 (10JAllemandou) [10:31:45] 06Data-Engineering: Add `is_redirect_to_pageview` field to `wmf_staging.webrequest` table - https://phabricator.wikimedia.org/T386176 (10JAllemandou) 03NEW [10:34:32] 06Data-Engineering: Switch webrequest dataset to feed from HAProxy instead of VarnishKafka - https://phabricator.wikimedia.org/T386177 (10JAllemandou) 03NEW [10:35:00] 06Data-Engineering: Add `is_redirect_to_pageview` field to `wmf_staging.webrequest` table - https://phabricator.wikimedia.org/T386176#10543660 (10JAllemandou) [10:35:02] 06Data-Engineering: Switch webrequest dataset to feed from HAProxy instead of VarnishKafka - https://phabricator.wikimedia.org/T386177#10543661 (10JAllemandou) [10:35:25] 06Data-Engineering, 06Data-Platform-SRE: Grow number of Gobblin mappers ingesting `webrequest_frontend` data - https://phabricator.wikimedia.org/T386174#10543664 (10JAllemandou) [10:35:26] 06Data-Engineering: Switch webrequest dataset to feed from HAProxy instead of VarnishKafka - https://phabricator.wikimedia.org/T386177#10543665 (10JAllemandou) [10:37:43] 10Data-Engineering (Q3 2024 January 1st - March 31th): Haproxy kafka and varnishkafka produce compatible datasets - https://phabricator.wikimedia.org/T382571#10543667 (10JAllemandou) I've gone through more data validation (`webrequest_text` only, 2025-02-06T06:00) and found some things to discuss with @Fabfur.... [10:40:10] 06Data-Engineering, 07Epic, 13Patch-For-Review: [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition - https://phabricator.wikimedia.org/T354694#10543676 (10JAllemandou) [10:40:26] 06Data-Engineering, 07Epic, 13Patch-For-Review: [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition - https://phabricator.wikimedia.org/T354694#10543679 (10JAllemandou) [10:41:20] 06Data-Engineering: Switch webrequest dataset to feed from HAProxy instead of VarnishKafka - https://phabricator.wikimedia.org/T386177#10543680 (10JAllemandou) [10:58:27] 06Data-Engineering, 06Data-Platform-SRE: Grow number of Gobblin mappers ingesting `webrequest_frontend` data - https://phabricator.wikimedia.org/T386174#10543720 (10JAllemandou) Using the `hdfs_usage` hive table, I have checked that as of 2025-02-03 the `wmf_raw.webrequest` table had 250426 files over 1496 hou... [11:04:35] 06Data-Engineering, 06Data-Platform-SRE: Grow number of Gobblin mappers ingesting `webrequest_frontend` data - https://phabricator.wikimedia.org/T386174#10543743 (10JAllemandou) [11:20:03] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 10Dumps-Generation, and 3 others: WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/25 - Validate Dumps 1.0 compatibility with PHP 8.1 - https://phabricator.wikimedia.org/T382484#10543793 (10BTullis) 05Open→03Resolved I think that we can call... [11:43:44] 06Data-Engineering, 06Commons, 06Data-Persistence, 10MediaWiki-File-management, and 4 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#10543914 (10Ladsgroup) Yes! [12:27:14] !log draining dse-k8s-worker1001 ready for reimage to bookworm and containerd for T377875 [12:27:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:27:18] T377875: Migrate dse-k8s cluster from docker to containerd - https://phabricator.wikimedia.org/T377875 [12:40:00] !log reimaging dse-k8s-worker1001 [12:40:02] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:02:20] 06Data-Engineering, 06Data-Platform-SRE: Grow kafka partition number for topics `webrequest_frontend_text` and `webrequest_frontend_upload` - https://phabricator.wikimedia.org/T386173#10544167 (10brouberol) ` brouberol@kafka-jumbo1014:~$ kafka topics --topic webrequest_frontend_text --alter --partitions 256 ka... [13:02:58] 06Data-Engineering, 06Data-Platform-SRE: Grow kafka partition number for topics `webrequest_frontend_text` and `webrequest_frontend_upload` - https://phabricator.wikimedia.org/T386173#10544171 (10brouberol) ` brouberol@kafka-jumbo1014:~$ kafka topics --topic webrequest_frontend_upload --alter --partitions 256... [13:09:36] !log deploying conda-analytics 0.0.38 to the cluster for T383284 and T380477 [13:09:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:09:41] T380477: Conda-Analytics clone fails when .conda/pkgs/cache is populated - https://phabricator.wikimedia.org/T380477 [13:18:27] 06Data-Engineering, 06Data-Platform-SRE: Grow kafka partition number for topics `webrequest_frontend_text` and `webrequest_frontend_upload` - https://phabricator.wikimedia.org/T386173#10544213 (10brouberol) The following [graph](https://thanos.wikimedia.org/graph?g0.expr=avg%28kafka_log_Size%7Bcluster%3D%22kaf... [13:44:10] 06Data-Engineering, 06Data-Platform-SRE: Grow kafka partition number for topics `webrequest_frontend_text` and `webrequest_frontend_upload` - https://phabricator.wikimedia.org/T386173#10544360 (10JAllemandou) Gobblin has adapted nicely as expected: https://grafana-rw.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&v... [14:24:56] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10544559 (10Ottomata) > It's not broadcasting events. It's sending messages to known reci... [14:27:13] 10Data-Engineering (Q3 2024 January 1st - March 31th): Update eventstreams Grafana Dashboards to use histogram for router metrics - https://phabricator.wikimedia.org/T386204 (10tchin) 03NEW [14:31:00] 06Data-Engineering, 06Java-Scala-Standardization: Resolve conflict between GitLab CI automated package deployment token variable names - https://phabricator.wikimedia.org/T386056#10544618 (10Ottomata) In parent pom.xml, scmUrl is `lang=xml scm:git:https://project_${gitlab.projectId}_bot:${env.CI_RELEA... [14:42:56] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10544712 (10Ottomata) > I think it would be a good start to examine places where we use t... [14:49:28] 06Data-Engineering, 06Data-Platform-SRE, 10Data-Platform (Data Platform Ops Week Working Group), 14Mediawiki Content: DAGs failing due to failure to acquire lock on wmf_data_ops.data_quality_metrics table - https://phabricator.wikimedia.org/T386114#10544769 (10Ahoelzl) [14:58:56] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Dumps 2.0, 13Patch-For-Review: Modify code to dump all slots - https://phabricator.wikimedia.org/T384945#10544813 (10xcollazo) Got it, thanks for the example @daniel. I incorporated [[ https://www.mediawiki.org/w/index.php?title=Manual%3ASlots_table&di... [15:06:55] 06Data-Engineering, 10Dumps 2.0: Modify wmf_content.mediawiki_content_history_v1 to include slot origin - https://phabricator.wikimedia.org/T386211 (10xcollazo) 03NEW [15:07:45] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Dumps 2.0, 13Patch-For-Review: Modify code to dump all slots - https://phabricator.wikimedia.org/T384945#10544860 (10xcollazo) >>! In T384945#10544813, @xcollazo wrote: > ... > This is definitely a gap we didn't think about, and will require a backward... [15:13:19] !log draining dse-k8s-worker1002 ready for reimage to bookworm and containerd for T377875 [15:13:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:13:22] T377875: Migrate dse-k8s cluster from docker to containerd - https://phabricator.wikimedia.org/T377875 [15:17:11] 10Data-Engineering (Q3 2024 January 1st - March 31th): HDFS capacity needs HTML dumps - https://phabricator.wikimedia.org/T384099#10544895 (10xcollazo) 05Open→03Resolved [15:18:08] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Experimentation Lab, 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Dashboard and alerting of data quality metrics for wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T357684#10544898 (10xcollazo) 05Open→03Resolve... [15:18:14] !log reimaging dse-k8s-worker1002 [15:18:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:22:11] 06Data-Engineering, 10Dumps 2.0, 07Epic, 14Mediawiki Content: Dumps 2.0 Phase II: Production intermediate table milestone - https://phabricator.wikimedia.org/T358877#10544931 (10xcollazo) 05Open→03Resolved [15:24:23] 06Data-Engineering, 06Data-Platform-SRE, 10Data-Platform (Data Platform Ops Week Working Group), 14Mediawiki Content: DAG failing due to failure to acquire lock on wmf_data_ops.data_quality_metrics table - https://phabricator.wikimedia.org/T386114#10544946 (10xcollazo) 05Open→03In progress p:05Triage... [15:29:23] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 10Dumps-Generation, and 3 others: WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/25 - Validate Dumps 1.0 compatibility with PHP 8.1 - https://phabricator.wikimedia.org/T382484#10544963 (10xcollazo) 🎉 🎉 🎉 [15:29:57] Hello. I want to have a request to https://en.wikipedia.org/w/api.php?action=parse&page=Assassin%27s_Creed_(video_game)&prop=text to get only the game engine. How can I have the url please? [15:36:59] 06Data-Engineering, 06Data-Platform-SRE, 10Data-Platform (Data Platform Ops Week Working Group), 14Mediawiki Content: DAG failing due to failure to acquire lock on wmf_data_ops.data_quality_metrics table - https://phabricator.wikimedia.org/T386114#10545090 (10tchin) There's another dag that fails but I tur... [15:53:12] 06Data-Engineering: Requesting Kerberos Password Reset - https://phabricator.wikimedia.org/T386225 (10SCherukuwada) 03NEW [15:53:12] 06Data-Engineering, 06Java-Scala-Standardization: Resolve conflict between GitLab CI automated package deployment token variable names - https://phabricator.wikimedia.org/T386056#10545152 (10amastilovic) > What if instead of using env.CI_RELEASE_TOKEN, we make another property for the gitlab project password?... [15:53:50] 06Data-Engineering: Requesting Kerberos Password Reset - https://phabricator.wikimedia.org/T386225#10545153 (10SCherukuwada) [15:54:11] 06Data-Engineering, 06Java-Scala-Standardization: Resolve conflict between GitLab CI automated package deployment token variable names - https://phabricator.wikimedia.org/T386056#10545154 (10amastilovic) The other approach would be to change the trigger_release CI template to use CI_RELEASE_TOKEN instead, corr... [16:17:24] 06Data-Engineering, 06Data-Platform-SRE: Grow kafka partition number for topics `webrequest_frontend_text` and `webrequest_frontend_upload` - https://phabricator.wikimedia.org/T386173#10545262 (10JAllemandou) Expected number of Gobblin-generated-file growth: ` hdfs dfs -count /wmf/data/raw/webrequest_frontend/... [16:17:45] 06Data-Engineering, 06Data-Platform-SRE: Grow number of Gobblin mappers ingesting `webrequest_frontend` data - https://phabricator.wikimedia.org/T386174#10545265 (10JAllemandou) Cross posting from here: https://phabricator.wikimedia.org/T386173#10545262 [16:18:03] 06Data-Engineering, 06Data-Platform-SRE: Grow number of Gobblin mappers ingesting `webrequest_frontend` data - https://phabricator.wikimedia.org/T386174#10545270 (10JAllemandou) [16:18:54] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10545272 (10daniel) >>! In T379935#10544712, @Ottomata wrote: > Oh, I totally misundersto... [16:23:49] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10545297 (10daniel) >>! In T379935#10544559, @Ottomata wrote: > Do you mean to say that y... [17:26:35] 06Data-Engineering, 06Java-Scala-Standardization: Resolve conflict between GitLab CI automated package deployment token variable names - https://phabricator.wikimedia.org/T386056#10545645 (10Ottomata) > we encountered an issue where the order in which maven properties are initialized, used, and populated was c... [17:42:35] 06Data-Engineering, 06Java-Scala-Standardization: Resolve conflict between GitLab CI automated package deployment token variable names - https://phabricator.wikimedia.org/T386056#10545723 (10amastilovic) > I would be surprised if the other properties that parent pom's scmUrl uses (e.g. gitlab.projectId) work,... [17:59:41] 06Data-Engineering, 06Java-Scala-Standardization: Resolve conflict between GitLab CI automated package deployment token variable names - https://phabricator.wikimedia.org/T386056#10545978 (10Ottomata) > Yes overriding the whole of scmUrl works, but the use case I referred to was when you override a Maven prope... [19:07:02] 06Data-Engineering, 10Dumps-Generation: wikidatawiki dump never started for 20250101 - https://phabricator.wikimedia.org/T383568#10546320 (10xcollazo) 05Open→03Declined We skipped this particular dump due to lack of time to run it. Elsewhere, @BTullis has been working on re-enabling all XML dumps, and... [19:11:24] 06Data-Engineering, 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 2 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255 (10Michael) 03NEW [19:17:30] 06Data-Engineering, 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 2 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10546360 (10xcollazo) CC @Ahoelzl [19:28:18] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814#10546398 (10Ottomata) @tchin has discovered that nodejs20-slim does not contain ssl package needed for tls to Kafka. eventgate probab... [19:42:33] 06Data-Engineering, 06Data-Platform-SRE, 10Data-Platform (Data Platform Ops Week Working Group), 14Mediawiki Content: DAG failing due to failure to acquire lock on wmf_data_ops.data_quality_metrics table - https://phabricator.wikimedia.org/T386114#10546444 (10xcollazo) Notes so far: Let's DESCRIBE the tab... [19:55:04] 06Data-Engineering, 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 2 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10546501 (10KStoller-WMF) I just wanted to note that the downstream dependency actua... [19:57:29] 06Data-Engineering, 06Data-Platform-SRE, 10Data-Platform (Data Platform Ops Week Working Group), 14Mediawiki Content: DAG failing due to failure to acquire lock on wmf_data_ops.data_quality_metrics table - https://phabricator.wikimedia.org/T386114#10546515 (10xcollazo) Reproing the issue manually: ` analy... [20:13:01] 06Data-Engineering, 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 2 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10546604 (10Ottomata) FWIW (in case we haven't reached [[ https://en.wikipedia.org/w... [20:33:43] 06Data-Engineering, 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 2 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10546677 (10fkaelin) This also affects the knowledge gaps/content gaps, article qual... [20:34:12] 06Data-Engineering, 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 3 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10546678 (10fkaelin) [21:14:55] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592#10546763 (10Marostegui) [21:15:11] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592#10546764 (10Marostegui) 05Open→03Resolved All done [21:26:29] 06Data-Engineering, 06Data-Platform-SRE, 10Data-Platform (Data Platform Ops Week Working Group), 14Mediawiki Content: DAG failing due to failure to acquire lock on wmf_data_ops.data_quality_metrics table - https://phabricator.wikimedia.org/T386114#10546816 (10xcollazo) @bking was able to list existing lock... [21:31:23] 06Data-Engineering, 06Data-Platform-SRE, 10Data-Platform (Data Platform Ops Week Working Group), 14Mediawiki Content: DAG failing due to failure to acquire lock on wmf_data_ops.data_quality_metrics table - https://phabricator.wikimedia.org/T386114#10546836 (10xcollazo) >>! In T386114#10545090, @tchin wrot... [21:57:47] 06Data-Engineering, 10ActiveAbstract, 10Dumps-Generation, 13Patch-For-Review: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10546922 (10xcollazo) +1 to move ahead and stop this dump. [23:01:56] 06Data-Engineering: Migrate analytics Airflow DAGs to k8s Airflow deployment - https://phabricator.wikimedia.org/T386282 (10amastilovic) 03NEW