[17:11:17] RESOLVED: [3x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:18:40] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review: Global Editor Metrics - Data Persistence Design Review - https://phabricator.wikimedia.org/T401260#11207023 (10Ottomata) Option C: Actually, storing e.g. top 30 pages with pageviews is not going... [17:19:44] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 10Data-Persistence-Design-Review: Global Editor Metrics - Data Persistence Design Review - https://phabricator.wikimedia.org/T401260#11207033 (10Ottomata) Total storage size is really quite hard to estimate. We are going to gener... [17:51:03] 06Data-Engineering, 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11207212 (10EBernhardson) >>! In T405360#11206505, @amastilovic wrote: > How big are the individual files we need to move for this? In my dataset we've targeted... [17:55:31] 06Data-Engineering, 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11207233 (10Ottomata) TIL rclone! Agree the puppet scheduled pulls of hdfs-rsync are not idea. If we can use this (or hdfs-rsync?) to rsync push from HDFS (sc... [18:00:01] 06Data-Engineering, 06Data-Platform-SRE: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#11207252 (10Ottomata) Imagine if you could airflow schedule a rsync push to [[ https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#S3_API | Toolfor... [18:02:30] 06Data-Engineering: Clean up artifacts.yaml - https://phabricator.wikimedia.org/T405379#11207285 (10Ottomata) > consider defining the artifacts as name-of-artifact-latest.jar and instead of doing individual bumps, we bump every existing job each time we release a new version? This might be quite onerous on ops... [18:31:12] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Core-Revision-backend, 10MediaWiki-DomainEvents, 10Event-Platform, and 2 others: MediaWiki\Revision\RevisionAccessException: Unable to load fresh row for rev_id: {rev_id} - https://phabricator.wikimedia.org/T400380#11207404 (10Ottomata)... [18:48:46] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Core-Revision-backend, 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, and 3 others: MediaWiki\Revision\RevisionAccessException: Unable to load fresh row for rev_id: {rev_id} - https://phabricator.wikimedia.org/T400380#11207468 (10Ottoma... [19:02:56] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Bug: event validation error: mediawiki.page-restrictions-change - https://phabricator.wikimedia.org/T390012#11207517 (10Ottomata) Okay, I think this might be a very long standing issue that no one really has noti... [19:08:21] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Core-Revision-backend, 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, and 3 others: MediaWiki\Revision\RevisionAccessException: Unable to load fresh row for rev_id: {rev_id} - https://phabricator.wikimedia.org/T400380#11207540 (10Ottoma... [19:08:35] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Bug: event validation error: mediawiki.page-restrictions-change - https://phabricator.wikimedia.org/T390012#11207543 (10Ottomata) 05Open→03In progress a:03Ottomata [19:12:07] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Investigate reasons for remaining inconsistencies - https://phabricator.wikimedia.org/T385112#11207560 (10xcollazo) >>! In T385112#11084123, @xcollazo wrote: > ... > I speculate the top category `missing_f... [19:34:19] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Investigate reasons for remaining inconsistencies - https://phabricator.wikimedia.org/T385112#11207670 (10xcollazo) On a debugging session today with @JAllemandou we found a bug in the code that can potent... [19:44:46] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10MediaWiki-Core-Revision-backend, 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, and 3 others: MediaWiki\Revision\RevisionAccessException: Unable to load fresh row for rev_id: {rev_id} - https://phabricator.wikimedia.org/T400380#11207694 (10daniel... [20:01:07] 06Data-Engineering: Clean up artifacts.yaml - https://phabricator.wikimedia.org/T405379#11207759 (10amastilovic) > This might be quite onerous on ops week duty and/or folks just trying to upgrade or deploy their job. We have that manual forced cache warmup for precisely this scenario by the way. [20:05:35] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06Data-Persistence (work done), and 4 others: Update DB schema to store whether contribution tracking is enabled for a given event - https://phabricator.wikimedia.org/T402816#11207779 (10vaughnwalters) ✅ The campaign_events table gets a new bo... [20:35:24] (03PS8) 10Ottomata: spark HiveExtensions now support column COMMENTs in DDL and merge helpers [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/987195 (https://phabricator.wikimedia.org/T307040) [20:36:43] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics, 13Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables - https://phabricator.wikimedia.org/T307040#11207867 (10Ottomata) Latest patch I think is on track to do what we need: - Let HiveExtensions just do t... [23:22:42] 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE: Jupyter spawner's list of Conda environments only updates when an environment is spawned - https://phabricator.wikimedia.org/T391894#11208338 (10nshahquinn-wmf) [23:27:11] 06Data-Engineering, 06MW-Interfaces-Team, 06Traffic, 07OKR-Work: Log Api-User-Agent header in Turnilo - https://phabricator.wikimedia.org/T373871#11208344 (10HCoplin-WMF) p:05Triage→03Low Updating as low priority since we don't think anyone is actually using it right now, and the work to add it to our...