[01:01:22] 06Data-Engineering, 10Dumps-Generation, 06Privacy Engineering, 13Patch-Needs-Improvement: Create partial SQL dump of watchlist table - https://phabricator.wikimedia.org/T51133#11769543 (10Pppery) [01:35:36] 06Data-Engineering, 10Dumps-Generation, 13Patch-Needs-Improvement: incomplete conversion of flow revisions after disabling flow, breaks stubs dumps - https://phabricator.wikimedia.org/T228921#11769705 (10Pppery) [01:56:41] 06Data-Engineering, 10Dumps-Generation, 13Patch-Needs-Improvement: stubs dumps run much slower after move from bufferied queries to batches - https://phabricator.wikimedia.org/T207628#11769807 (10Pppery) [02:44:17] 06Data-Engineering, 10Dumps-Generation, 10MediaWiki-General, 10MediaWiki-Maintenance-system, and 4 others: BatchRowIterator slow query on commonswiki - https://phabricator.wikimedia.org/T260232#11770045 (10Pppery) [07:58:12] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Investigate Gobblin failures - https://phabricator.wikimedia.org/T419436#11770751 (10JAllemandou) 2 failed instance this morning: * 2x connection problem to metawiki API. [08:22:33] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 13Patch-For-Review: Deploy turnilo to dse-k8s-eqiad - https://phabricator.wikimedia.org/T416113#11770890 (10brouberol) 05Open→03Resolved a:03brouberol [08:42:21] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Audit and fix observability (logging and metrics) for pyflink jobs - https://phabricator.wikimedia.org/T418996#11770966 (10JMonton-WMF) There is a possible improvement here, not related to logs or monitoring, but I found it while explo... [08:44:19] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Audit and fix observability (logging and metrics) for pyflink jobs - https://phabricator.wikimedia.org/T418996#11770985 (10JMonton-WMF) This warning I'm not sure if it appears because the application was restarting and the metric name... [08:47:30] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Audit and fix observability (logging and metrics) for pyflink jobs - https://phabricator.wikimedia.org/T418996#11771062 (10JMonton-WMF) Another warning to investigate: ` The operator name process_error_enrich_with_parent_html, convert_... [08:50:06] 06Data-Engineering, 06Data-Engineering-Radar, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11771122 (10gkyziridis) > I recall we verified it works on staging. Is there anything... [10:50:10] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10GlobalBlocking, and 2 others: Drop global_block_whitelist from closed wikis - https://phabricator.wikimedia.org/T420525#11771832 (10Marostegui) I've renamed the tables on s3 master (db1183) to: `T420525_global_block_whitelist` Then the only two wikis n... [10:51:53] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions, 13Patch-For-Review: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11771834 (10APizzata-WMF) ` spark.sql(""" ( select * from analytics_platform_eng.image... [11:29:07] Puppet is disabled on an-worker1148 since March 17, meaning it'll soon be evicted from puppetdb? is that intentional or some oversight? [11:36:58] moritzm: oversight. We're dealing with a hardware issue on the host that took a long time to fix https://phabricator.wikimedia.org/T411919 I'm guessing puppet was not re-enabled. Let me have a look [11:37:21] thanks [11:37:39] I re-enabled puppet and am running the agent atm [11:38:38] huh [11:38:39] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Number of datanode mountpoints (9) below threshold: 10, please check. (file: /srv/puppet_code/environments/production/modules/profile/manifests/hadoop/common.pp, line: 418, column: 9) on node [11:38:39] an-worker1148.eqiad.wmnet [11:39:55] It seems there's a broken disk [11:40:07] /dev/sdk is spewing errors in dmesg [11:41:25] yep [11:55:34] !log Test Kitchen edge-unique experiments (poll 50540) - adds: we-1-8-mobile-account-menu; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [11:55:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:31:15] !log Test Kitchen mw-user experiment (poll 50645) - adds: sample-experiment-test; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [12:31:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:38:59] !log Test Kitchen mw-user experiment (poll 50668) - adds: none; removes: none; fields: sample-experiment-test - xLab/MPIC/TK tips at https://w.wiki/FwuD [12:39:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:39:44] !log Test Kitchen mw-user experiment (poll 50670) - adds: none; removes: sample-experiment-test; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [12:39:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:49:17] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions, 13Patch-For-Review: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11772487 (10dcausse) >>! In T368987#11771834, @APizzata-WMF wrote: > All looks good to m... [13:00:12] moritzm: puppet is now running on the host [13:06:14] (he quit) [13:06:22] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Wikidata full .json.gz dumps not published since 20250625 - https://phabricator.wikimedia.org/T412428#11772578 (10Gehel) [13:20:55] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 10Event-Platform: Flink base image should not install into system python environment - https://phabricator.wikimedia.org/T418525#11772655 (10Gehel) [13:21:26] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 10Event-Platform, 07good first task: Flink base image should not install into system python environment - https://phabricator.wikimedia.org/T418525#11772657 (10Gehel) Thank you for tagging this task with #good_first_task for Wikimedia newco... [13:28:40] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions, 13Patch-For-Review: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11772704 (10APizzata-WMF) [[ https://airflow-search.wikimedia.org/dags/image_suggestions... [14:05:37] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Visualizing inconsistencies and reconciles via Superset - https://phabricator.wikimedia.org/T420787#11773019 (10Ahoelzl) @gmodena loop us in! [14:24:42] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Audit and fix observability (logging and metrics) for pyflink jobs - https://phabricator.wikimedia.org/T418996#11773142 (10JMonton-WMF) Possible issues with metrics being reported: ` Invalid type for Gauge org.apache.flink.runtime.che... [15:03:26] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Audit and fix observability (logging and metrics) for pyflink jobs - https://phabricator.wikimedia.org/T418996#11773404 (10Ottomata) > Config uses deprecated configuration key 'state.backend' instead of proper key 'state.backend.type'... [15:17:09] 06Data-Engineering: Backfill newly productionized edit types dataset - https://phabricator.wikimedia.org/T421919 (10CMyrick-WMF) 03NEW [15:19:14] 06Data-Engineering: Backfill newly productionized edit types dataset - https://phabricator.wikimedia.org/T421919#11773545 (10Ottomata) [[ https://wikimedia.slack.com/archives/C02291Z9YQY/p1774920403892289?thread_ts=1772047120.154549&cid=C02291Z9YQY | Slack thread ]] with backfill approach. [15:23:00] 06Data-Engineering: Backfill newly productionized edit types dataset - https://phabricator.wikimedia.org/T421919#11773578 (10Ottomata) It would be nice to backfill the final production table, rather than our current development tables. Perhaps for now, we should create a separate table to hold the backfill so... [15:29:37] 06Data-Engineering: Backfill newly productionized edit types dataset - https://phabricator.wikimedia.org/T421919#11773679 (10Ottomata) OTOH, perhaps all we need is something for a one off now? If so this, would be a quick one off and easy enough to do. [16:01:03] 06Data-Engineering, 06Data-Engineering-Radar, 10FR-Tech-Analytics, 13Patch-For-Review: Create FR Tech Airflow instance - https://phabricator.wikimedia.org/T417213#11773913 (10MoritzMuehlenhoff) The new instance is missing at https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Instances#Li... [17:02:53] !log Test Kitchen mw-user experiment (poll 51448) - adds: sample-experiment-test; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [17:02:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:09:21] !log Test Kitchen mw-user experiment (poll 51467) - adds: none; removes: none; fields: sample-experiment-test - xLab/MPIC/TK tips at https://w.wiki/FwuD [17:09:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:46:03] 06Data-Engineering, 06Growth-Team: Investigate empty Constructive edit rate of newer editors (mobile web) - https://phabricator.wikimedia.org/T421514#11774640 (10Sgs) Discussed with @MNeisler and a plausible explanation would be that we're sending `edit_saved` events for anon users, and these do not contain th... [18:06:18] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Observability-Metrics: [Data Quality] Sending Apache Spark metrics to PushGateway - https://phabricator.wikimedia.org/T297231#11774780 (10Ahoelzl) [18:12:53] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Observability-Metrics: [Data Quality] Sending Apache Spark metrics to PushGateway - https://phabricator.wikimedia.org/T297231#11774814 (10JAllemandou) I wish to revive this task, maybe not sending data to push-gateway at first, but at least storing me... [18:17:45] 06Data-Engineering, 10Event-Platform: eventutilities-python - make Flink Source and Sink parallelism configurable - https://phabricator.wikimedia.org/T421951 (10Ottomata) 03NEW [18:18:07] 06Data-Engineering, 10Event-Platform: eventutilities-python - make Flink Source and Sink parallelism configurable - https://phabricator.wikimedia.org/T421951#11774845 (10Ottomata) [18:18:13] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform, 13Patch-For-Review: Event stream with latest revision HTML & parent revision HTML diff - https://phabricator.wikimedia.org/T360794#11774846 (10Ottomata) [18:45:08] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0: Consider updating our heuristics for media type classification in AQS / wikistats - https://phabricator.wikimedia.org/T419882#11774929 (10Ahoelzl) @GGoncalves-WMF can you get back to Sandra on her proposal? [18:46:17] !log Test Kitchen mw-user experiment (poll 51754) - adds: none; removes: sample-experiment-test; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [18:46:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:54:52] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add support for variables to DbtSkeinOperator - https://phabricator.wikimedia.org/T421789#11774964 (10Mayakp.wiki) hey @amastilovic , confirming, this is for the backfill use case we discussed yesterday, right? Also you may want to change the Q3 tag to Q... [19:24:13] 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: eventutilities-python - make Flink Source and Sink parallelism configurable - https://phabricator.wikimedia.org/T421951#11775064 (10Ottomata) It works! ` helmfile apply -i -e dse-k8s-eqiad --set app.config_files.app\\.config\\.yaml.stream_manager.so... [19:27:22] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Debug edit type pipeline for production readiness - https://phabricator.wikimedia.org/T421026#11775078 (10AKhatun_WMF) Something to consider: The research spark job for html based edit-type has 60s timeout (previously 300s). And from... [20:29:35] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform: eventutilties-python - support synchronous Flink process function mode - https://phabricator.wikimedia.org/T421965 (10Ottomata) 03NEW [20:29:57] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research, 10Event-Platform: eventutilties-python - support synchronous Flink process function mode - https://phabricator.wikimedia.org/T421965#11775286 (10Ottomata) [20:38:07] 06Data-Engineering, 06MW-Interfaces-Team, 06Traffic, 07OKR-Work: Log Api-User-Agent header in Turnilo - https://phabricator.wikimedia.org/T373871#11775321 (10LucasWerkmeister) >>! In T373871#11208344, @HCoplin-WMF wrote: > Updating as low priority since we don't think anyone is actually using it right now... [22:08:27] !log Test Kitchen mw-user experiment (poll 52352) - adds: none; removes: none; fields: growthexperiments-revise-tone - xLab/MPIC/TK tips at https://w.wiki/FwuD [22:08:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:53:35] 10Analytics-Canonical-Data, 10Test Kitchen, 10Test Kitchen UI: Enable targeting auth wiki for experiments in Test Kitchen UI - https://phabricator.wikimedia.org/T421972 (10mpopov) 03NEW [23:08:16] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-DomainEvents, 05MW-1.45-release, and 3 others: Page-related DomainEvent classes with "@deprecated temporary alias, remove before 1.45 release" - https://phabricator.wikimedia.org/T417721#11775756 (10BPirkle) Aaron surveyed all extensions and found n... [23:44:20] 10Analytics-Canonical-Data, 10Test Kitchen, 10Test Kitchen UI: Enable targeting auth wiki for experiments in Test Kitchen UI - https://phabricator.wikimedia.org/T421972#11775835 (10mpopov)