[07:19:26] 10Data-Engineering (Q3 2024 January 1st - March 31th), 03Abstract Wikipedia Fix-It tasks, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 13Patch-For-Review: Update wikifunctions Grafana dashboard for service-utils - https://phabricator.wikimedia.org/T387360#10599384 (10DSantamaria) 05Open→03In progress [07:50:35] 10Data-Engineering (Q3 2025 January 1st - March 31th): Fix service-utils metrics routing naming discrepancy - https://phabricator.wikimedia.org/T387824 (10tchin) 03NEW [07:52:20] 10Data-Engineering (Q3 2025 January 1st - March 31th), 03Abstract Wikipedia Fix-It tasks, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 13Patch-For-Review: Update wikifunctions Grafana dashboard for service-utils - https://phabricator.wikimedia.org/T387360#10599508 (10tchin) @DSantamaria Sorry, the ticket to... [10:15:09] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839 (10JAllemandou) 03NEW [10:15:21] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#10600037 (10JAllemandou) p:05Triage→03High [10:58:29] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#10600147 (10ayounsi) Let's see what other people think, but I think it would be fine to : * Keep only 1 month... [11:00:05] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10Charts (Sprint 17), 07Schema-change-in-production: Deploy patch-gjlw_namespace_text.sql on x1.commonswiki for JsonConfig - https://phabricator.wikimedia.org/T385917#10600158 (10Marostegui) I think this has caused {T387843} [11:01:40] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10Charts (Sprint 17), 07Schema-change-in-production: Deploy patch-gjlw_namespace_text.sql on x1.commonswiki for JsonConfig - https://phabricator.wikimedia.org/T385917#10600164 (10hashar) I have rolled back the train cause `patch-gjlw_namespace_text.sql... [11:06:08] !log Deploying refinery using scap [11:06:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:12:36] !log Deploy refinery to HDFS [11:12:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:23:05] !log Deploying airflow-dags/analytics [11:23:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:47:13] !log backfilling webrequest_frontend [11:47:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:31:43] 10Data-Engineering (Q3 2025 January 1st - March 31th), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814#10600486 (10fgiunchedi) >>! In T383814#10596859, @Ottomata wrote: > Uh, oops. I deployed eventgate-logging-externa... [13:39:32] 06Data-Engineering, 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10600695 (10Joe) p:05Triage→03Unbreak! Changing to UBN! as per-country NELs are needed for quite a few things, including paging SREs. [13:53:10] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#10600761 (10cmooney) +1 I've no objection to any of these. 30 days for the full data is probably enough. In... [14:05:09] 06Data-Engineering, 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10600804 (10tchin) Could it have something to do with T382173? [14:22:36] 06Data-Engineering, 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10600881 (10Ottomata) Will look into this ASAP today. I suspect @tchin is correct. [14:23:04] 06Data-Engineering, 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10600882 (10Ottomata) We can also rollback asap if needed. [14:24:20] 06Data-Engineering, 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10600886 (10Joe) >>! In T387850#10600882, @Ottomata wrote: > We can also rollback asap if needed. Unless you think you have an obvious fix, yes please. [14:41:03] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Dumps-Generation, 06Growth-Team, 10GrowthExperiments, and 3 others: structured_data.commons_entity stuck at 2025-01-20 - https://phabricator.wikimedia.org/T387470#10600968 (10xcollazo) Agreed this is likely related to T386255, as all the `wikibase` r... [14:41:41] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 7 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10600970 (10xcollazo) >>! In T387470#10600968, @x... [15:03:08] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 7 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10601067 (10xcollazo) I want to try the following... [15:22:53] 06Data-Engineering, 06Data-Engineering-Icebox, 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown: Provide dumps using bittorrent - https://phabricator.wikimedia.org/T29653#10601174 (10valerio.bozzolan) So this seems not possible as long as the only way to access/mount dumps is through... [15:27:36] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#10601220 (10ayounsi) Mostly to be able to see long term trends, for example per destination AS. [15:50:25] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 7 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10601324 (10xcollazo) @bking ran the following: `... [16:04:57] 06Data-Engineering, 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10601404 (10Ottomata) I rolled back. Let's find and fix the bug before deploying again. [16:05:19] 10Data-Engineering (Q3 2025 January 1st - March 31th), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814#10601419 (10Ottomata) I rolled back. Let's find and fix the bug before deploying again. [16:05:46] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10601428 (10Ottomata) [16:11:25] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10601478 (10Ottomata) @fgiunchedi do you see the headers flowing in now? I just checked [[ https://thanos.wikimedia.org/graph?g0.ex... [16:18:16] 06Data-Engineering, 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Seen): Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10601555 (10Ottomata) Ya makes sense! And we do that now. But, users of packages have to know and configure... [16:28:56] (03PS18) 10Peter Fischer: Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) [16:29:16] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#10601646 (10Ottomata) FWIW, netflow is ingested into the Data Lake, so is queryable using SQL and/or [[ https... [16:29:17] (03PS19) 10Peter Fischer: Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) [16:29:17] (03PS4) 10Peter Fischer: Adapt table/column names [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1115440 (https://phabricator.wikimedia.org/T384385) [16:29:18] (03PS3) 10Peter Fischer: Partial dumps [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1115441 (https://phabricator.wikimedia.org/T384383) [16:39:22] (03CR) 10CI reject: [V:04-1] Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) (owner: 10Peter Fischer) [16:52:17] 06Data-Engineering, 06Data-Platform-SRE, 10DPE-Mediawiki-Content, 13Patch-For-Review: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#10601830 (10Gehel) [16:58:13] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#10601848 (10JAllemandou) >>! In T387839#10601646, @Ottomata wrote: > FWIW, netflow is ingested into the Data... [17:05:50] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10601870 (10Joe) >>! In T387850#10601478, @Ottomata wrote: > @fgiunchedi do you see the headers flowing in now? > > I just checked... [17:16:00] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10601932 (10fgiunchedi) >>! In T387850#10601478, @Ottomata wrote: > @fgiunchedi do you see the headers flowing in now? > > I just c... [17:16:21] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10601935 (10fgiunchedi) p:05Unbreak!→03Medium [17:20:44] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10601956 (10Ottomata) Ah, I read that screen shot metric name like 5 times and thought I had the correct one. Oh well thanks. Glad... [17:28:01] 06Data-Engineering, 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Seen): Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10601981 (10brennen) [17:39:01] 10Data-Engineering (Q3 2025 January 1st - March 31th), 07Essential-Work: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10602057 (10Ahoelzl) a:05JEbe-WMF→03mforns [18:01:51] 06Data-Engineering, 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Seen): Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10602151 (10thcipriani) Got it. Well. The "instance" scope seems fully bought-in to the github model of each... [18:32:23] 06Data-Engineering, 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Seen): Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10602206 (10Ottomata) > Who is using these packages, currently? Any team that uses service-runner, or hopef... [18:35:09] 06Data-Engineering, 06Data-Engineering-Radar, 10Page Content Service, 10RESTBase Sunsetting, and 3 others: Add page namespace information on resource change events - https://phabricator.wikimedia.org/T387435#10602213 (10Ahoelzl) [18:38:28] 06Data-Engineering, 10Data Pipelines, 06Movement-Insights: Add the global registration date to mediawiki_history - https://phabricator.wikimedia.org/T363775#10602226 (10Ottomata) [18:56:03] 06Data-Engineering: NEW/CHANGE FEATURE REQUEST:  - https://phabricator.wikimedia.org/T383959#10602389 (10VirginiaPoundstone) 05Open→03Invalid [18:57:06] 10Data-Engineering (Q3 2025 January 1st - March 31th): Investigate why the mw-content-history-reconcile-enrich Flink job failed. - https://phabricator.wikimedia.org/T387906#10602394 (10xcollazo) [18:58:34] 06Data-Engineering, 06Traffic: GeoDNS: Pipeline from event.development_network_probe to operations/dns.git - https://phabricator.wikimedia.org/T380626#10602407 (10Ottomata) [19:06:58] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Product-Analytics, 10Event-Platform: [BUG] new eventgate-wikimedia header enrich config loses client set headers - https://phabricator.wikimedia.org/T387908 (10Ottomata) 03NEW [19:08:34] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06SRE Observability, 10Event-Platform: NEL logs are missing geoip information - https://phabricator.wikimedia.org/T387850#10602468 (10Ottomata) 05Open→03Resolved a:03Ottomata Filed T387908 to track the underlying issue. I'll resolve this one... [19:13:59] 06Data-Engineering, 06Machine-Learning-Team, 06Research, 10Event-Platform: Expose revision revert risk scores in EventStreams - https://phabricator.wikimedia.org/T326179#10602482 (10Ottomata) FWIW, I believe that if this task had been done, investigatory work for tasks like {T374440} would be much easier. [19:31:40] (03PS20) 10Peter Fischer: Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) [19:31:59] (03PS21) 10Peter Fischer: Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) [19:31:59] (03PS5) 10Peter Fischer: Adapt table/column names [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1115440 (https://phabricator.wikimedia.org/T384385) [19:32:16] (03PS4) 10Peter Fischer: Partial dumps [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1115441 (https://phabricator.wikimedia.org/T384383) [19:33:12] !log Deploy latet DAGs for analytics Airflow instance. T387906. [19:33:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:33:16] T387906: Investigate why the mw-content-history-reconcile-enrich Flink job failed. - https://phabricator.wikimedia.org/T387906 [19:47:00] 10Data-Engineering (Q3 2025 January 1st - March 31th), 13Patch-For-Review: Investigate why the mw-content-history-reconcile-enrich Flink job failed. - https://phabricator.wikimedia.org/T387906#10602684 (10xcollazo) [19:50:30] 06Data-Engineering, 06Machine-Learning-Team, 06Research, 10Event-Platform: Expose revision revert risk scores in EventStreams - https://phabricator.wikimedia.org/T326179#10602692 (10diego) I'm confused, I think in T374440 they are working just with dumps, nothing like Eventstreams. >>! In T326179#106024... [20:19:49] 06Data-Engineering, 06Machine-Learning-Team, 06Research, 10Event-Platform: Expose revision revert risk scores in EventStreams - https://phabricator.wikimedia.org/T326179#10602807 (10Ottomata) If revert risk scores were in event streams (lower case, not necessarily stream.wikimedia.org EventStreams service)... [20:43:27] 10Data-Engineering (Q3 2025 January 1st - March 31th): Fix service-utils metrics routing naming discrepancy - https://phabricator.wikimedia.org/T387824#10602882 (10ecarg) TY @tchin! I like option 3 😄 and going with the easiest fix to start [20:44:35] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content: Figure root cause of silent failures when computing metrics for mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T387033#10602886 (10xcollazo) 05In progress→03Resolved [21:37:36] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 06Research: A dataset sensor should work indepent of airflow instance - https://phabricator.wikimedia.org/T386973#10603151 (10xcollazo) [21:38:43] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 06Research: A dataset sensor should work indepent of airflow instance - https://phabricator.wikimedia.org/T386973#10603153 (10xcollazo) @amastilovic I believe we are done here, yes? [21:44:11] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#10603177 (10Ahoelzl) [21:53:06] 06Data-Engineering, 10ActiveAbstract, 10Dumps-Generation, 13Patch-For-Review: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10603205 (10Ladsgroup) Two dump runs now don't include Yahoo! abstracts anymore. I haven't heard even a single complaint so far. Shall we ax the code... [22:08:50] 10Data-Engineering (Q3 2025 January 1st - March 31th): Provide Data Engineering Q4 draft - https://phabricator.wikimedia.org/T387385#10603264 (10Ahoelzl) Essential Work catalog draft https://docs.google.com/document/d/1-LgIPqYSmz83Ujl-Zu95hTE1Fta6RQUH89GRolYTev4/edit?tab=t.0 [22:10:00] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10603265 (10Ottomata) Hm, I think I'm following, but not sure. > I was thinking that, if... [22:35:53] 10Data-Engineering (Q3 2025 January 1st - March 31th): Fix service-utils metrics routing naming discrepancy - https://phabricator.wikimedia.org/T387824#10603338 (10tchin) Yeah I'm going option 3 as well. The main issue is that the middleware is not aware of the router or the path it's on, so I need to figure out... [22:36:00] 10Data-Engineering (Q3 2025 January 1st - March 31th): Fix service-utils metrics routing naming discrepancy - https://phabricator.wikimedia.org/T387824#10603339 (10tchin) 05Open→03In progress [22:51:36] 06Data-Engineering, 06Data-Persistence, 10MediaWiki-Page-derived-data, 07Schema-change: Add page_is_redirect/page_namespace/page_title index - https://phabricator.wikimedia.org/T387537#10603404 (10Ladsgroup) >>! In T387537#10595028, @tstarling wrote: >>>! In T387537#10590702, @Ladsgroup wrote: >> I suggest... [23:23:03] (03CR) 10Xcollazo: [C:03+2] Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) (owner: 10Peter Fischer) [23:36:02] (03Merged) 10jenkins-bot: Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) (owner: 10Peter Fischer)