[00:02:45] PROBLEM - Webrequests Varnishkafka log producer on cp4042 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:02:45] PROBLEM - statsv Varnishkafka log producer on cp4038 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:02:47] PROBLEM - Webrequests Varnishkafka log producer on cp4043 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:02:47] PROBLEM - Webrequests Varnishkafka log producer on cp4037 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:02:49] PROBLEM - statsv Varnishkafka log producer on cp4039 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:09:10] FIRING: VarnishkafkaNoMessages: [WARNING] - varnishkafka on cp4037 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp4037%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [00:24:10] RESOLVED: VarnishkafkaNoMessages: [WARNING] - varnishkafka on cp4037 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp4037%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:10:42] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team, and 2 others: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10655908 (10BTullis) I am rolling out PHP version 8.1 to all snapshot hosts except snapshot1016. They should... [08:16:17] 06Data-Engineering, 10DPE-Mediawiki-Content: Go over tasks in #dumps-generation and figure what makes sense to fix in Dumps 2.0 (DPE-Mediawiki-Content) - https://phabricator.wikimedia.org/T379410#10655998 (10Aklapper) [10:09:28] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team, and 2 others: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10656294 (10BTullis) 7 of the 8 snapshot servers are now running PHP 8.1 for dumps. ` btullis@cumin1002:~$ su... [11:13:55] !log restarting ceph-mds@cephosd1004.service for T389470 [11:13:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:13:59] T389470: CephOSDClusterInWarning - https://phabricator.wikimedia.org/T389470 [12:33:05] 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, 10MW-on-K8s, and 3 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484 (10Clement_Goubert) 03NEW [12:34:06] 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, 10MW-on-K8s, and 3 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10656829 (10Clement_Goubert) p:05Triage→03High [12:34:46] 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, 10MW-on-K8s, and 3 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10656831 (10Clement_Goubert) [12:43:33] 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, 10MW-on-K8s, and 3 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10656844 (10Clement_Goubert) [12:43:58] 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, 10MW-on-K8s, and 3 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10656845 (10Clement_Goubert) [13:31:35] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 6 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10657077 (10Ahoelzl) Gzipped download files lande... [13:39:27] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 6 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10657101 (10xcollazo) Ok good news are starting t... [14:03:07] (03CR) 10Xcollazo: Add columns to data_quality_alerts to support inserting ResultKey (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1127967 (https://phabricator.wikimedia.org/T384962) (owner: 10TChin) [14:03:38] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 10Image-Suggestions, 10Section-Level-Image-Suggestions, 10Structured-Data-Backlog (Current Work): [SPIKE] Check the Wikimedia content history dataset - https://phabricator.wikimedia.org/T385787#10657188 (10Cparle) We can get... [14:28:14] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 10Image-Suggestions, 10Section-Level-Image-Suggestions, 10Structured-Data-Backlog (Current Work): [SPIKE] Check the Wikimedia content history dataset - https://phabricator.wikimedia.org/T385787#10657250 (10Cparle) [14:52:01] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 10Image-Suggestions, 10Section-Level-Image-Suggestions, 10Structured-Data-Backlog (Current Work): [SPIKE] Check the Wikimedia content history dataset - https://phabricator.wikimedia.org/T385787#10657308 (10xcollazo) Re T38578... [14:55:52] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Support downstream users in adopting mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T387021#10657318 (10xcollazo) [15:16:00] 10Data-Engineering (Q3 2025 January 1st - March 31th): Use the Spark-Iceberg built in CDC mechanism to PoC a replacement for wmf.wikimedia_wikitext_current - https://phabricator.wikimedia.org/T366544#10657404 (10xcollazo) [15:16:11] 10Data-Engineering (Q3 2025 January 1st - March 31th): Use the Spark-Iceberg built in CDC mechanism to PoC a replacement for wmf.wikimedia_wikitext_current - https://phabricator.wikimedia.org/T366544#10657405 (10xcollazo) 05Open→03In progress p:05Triage→03High [15:16:42] 10Data-Engineering (Q3 2025 January 1st - March 31th): Use the Spark-Iceberg built in CDC mechanism to PoC a replacement for wmf.wikimedia_wikitext_current - https://phabricator.wikimedia.org/T366544#10657409 (10xcollazo) a:03xcollazo [15:43:06] 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, 10MW-on-K8s, and 4 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10657560 (10Clement_Goubert) Build succeeded, the image is currently being tested by @brouberol If it's conclusive, we can resolve this, and wo... [15:53:06] 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, 10MW-on-K8s, and 4 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10657651 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert Image has what's needed for `dumps`, resolving. [16:14:02] 06Data-Engineering, 10DPE-Mediawiki-Content: dumps_publish_wikitext_raw_to_xml DAG fails sporadically. Needs sensors to wait before running. - https://phabricator.wikimedia.org/T363941#10657737 (10xcollazo) 05Open→03Invalid We are abandoning this DAG as it needs a full rewrite. [17:39:03] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 10Image-Suggestions, 10Section-Level-Image-Suggestions, 10Structured-Data-Backlog (Current Work): [SPIKE] Check the Wikimedia content history dataset - https://phabricator.wikimedia.org/T385787#10658240 (10Cparle) [17:39:23] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 10Image-Suggestions, 10Section-Level-Image-Suggestions, 10Structured-Data-Backlog (Current Work): [SPIKE] Check the Wikimedia content history dataset - https://phabricator.wikimedia.org/T385787#10658245 (10Cparle) 05Open... [17:41:21] (03PS2) 10Snwachukwu: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T241741) (owner: 10Milimetric) [17:48:13] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Traffic, 10DPE HAProxy Migration, 13Patch-For-Review: Make webrequest_frontend being ingested using the in-data `dt` field - https://phabricator.wikimedia.org/T388397#10658363 (10JAllemandou) [17:48:29] (03PS1) 10Joal: Update Gobblin webrequest_frontend timestamp [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1129900 (https://phabricator.wikimedia.org/T388397) [17:52:42] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Traffic, 10DPE HAProxy Migration: Update HAProxyKafka kafka-timestamp type - https://phabricator.wikimedia.org/T389521 (10JAllemandou) 03NEW [18:28:27] 06Data-Engineering: Set up Alerting for Data Quality dags in Airflow. - https://phabricator.wikimedia.org/T377333#10658758 (10xcollazo) Oh, I though we had this mechanism already. So, today, there is no alerting whatsoever from `data_quality_alerts`? [18:28:46] 06Data-Engineering: Set up Alerting for Data Quality dags in Airflow. - https://phabricator.wikimedia.org/T377333#10658762 (10xcollazo) CC @tchin. [20:08:21] 06Data-Engineering, 10Data Pipelines: Provide an easy way for MediaWiki to fetch aggregate statistics from the data lake - https://phabricator.wikimedia.org/T341649#10659122 (10Ottomata) [20:09:04] 06Data-Engineering, 06Experimentation Lab: NEW/CHANGE FEATURE REQUEST: Documentation for v1 Enterprise endpoint deprecation - https://phabricator.wikimedia.org/T389542 (10Cpetrillo) 03NEW [20:10:54] 06Data-Engineering, 10Data Pipelines: Provide an easy way for MediaWiki to fetch aggregate statistics from the data lake - https://phabricator.wikimedia.org/T341649#10659141 (10Ottomata) Related: {T258511} [20:15:15] 06Data-Engineering, 06Experimentation Lab: NEW/CHANGE FEATURE REQUEST: Documentation for v1 Enterprise endpoint deprecation - https://phabricator.wikimedia.org/T389542#10659158 (10Ottomata) Hello! If you feel frisky, please submit a patch. The files you'll need to change are static html, at https://gerrit.w... [20:23:41] 06Data-Engineering, 10Data Pipelines: Provide an easy way for MediaWiki to fetch aggregate statistics from the data lake - https://phabricator.wikimedia.org/T341649#10659177 (10kostajh) [20:26:24] 06Data-Engineering, 10Data Pipelines: Provide an easy way for MediaWiki to fetch aggregate statistics from the data lake - https://phabricator.wikimedia.org/T341649#10659194 (10Ottomata) BTW I gave a [[ https://docs.google.com/presentation/d/1cum7oFGN_pdx3za3hHgYYRkqOE76RicO1veYgFLZVrk/edit#slide=id.g2fc2bee90... [20:51:04] (03CR) 10Aqu: [C:03+1] "Not sure to understand. Could you point me to the code where clearing `source.kafka.extract.timestampTypes=CreateTime` would change the pa" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1129900 (https://phabricator.wikimedia.org/T388397) (owner: 10Joal) [21:21:49] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10Charts (Sprint 18), 07Schema-change-in-production: Deploy patch-gjlw_namespace_text.sql on x1.commonswiki for JsonConfig - https://phabricator.wikimedia.org/T385917#10659426 (10bvibber) [22:05:52] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 6 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10659774 (10xcollazo) Partition `snapshot=2025-03... [22:12:23] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 6 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10659803 (10xcollazo) Partition `snapshot=2025-03... [22:26:23] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 6 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10659833 (10xcollazo) 05In progress→03Reso...