[05:05:43] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9980111 (10Marostegui) [08:46:14] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#9980350 (10Clement_Goubert) 05Open→03In progress a:03Clement_Goubert [09:05:06] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#9980431 (10Clement_Goubert) 05In progress→03Resolved p:05Triage→03Medium @XiaoXiao-WMF You should have received an email with instructions on... [09:22:04] (03PS1) 10Btullis: Add ae.wikimedia project to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1054282 (https://phabricator.wikimedia.org/T362529) [09:30:36] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 06Privacy Engineering: Raw IPs of logged-out users disclosed in wiki-replicas - https://phabricator.wikimedia.org/T284948#9980515 (10fnegri) [09:34:05] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 06Privacy Engineering: Raw IPs of logged-out users disclosed in wiki-replicas - https://phabricator.wikimedia.org/T284948#9980518 (10fnegri) p:05Triage→03Low > I am fine with having this ticket stalled until IP masking (T283177) is effective,... [09:34:51] (03PS1) 10Btullis: Add aewikimedia to the sqoop list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1054285 (https://phabricator.wikimedia.org/T362529) [09:51:39] 06Data-Engineering, 10Data-Platform-SRE (2024.07.08 - 2024.07.28): Some wikibase tables not available in commonswiki_p - https://phabricator.wikimedia.org/T298452#9980566 (10fnegri) [09:59:35] 14Analytics-Radar, 06Data-Engineering-Icebox, 06cloud-services-team, 10Data-Services: Mitigate breaking changes from the new Wiki Replicas architecture - https://phabricator.wikimedia.org/T280152#9980570 (10fnegri) 05Open→03Resolved a:03fnegri Marking this as Resolved as all the main subtasks hav... [10:26:56] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9980707 (10Marostegui) [11:50:28] Hi xcollazo we have had some dumps related airflow alerts, probably related to the recent snapshot1017 work. would you mind helping me have a look? The SLA misses are in failed status for the times listed below [11:50:53] https://www.irccloud.com/pastebin/8pltQeXR/ [11:52:45] oops just seen xcollazo is OOO anyone else available to help cc aqu btullis [11:53:52] I can help, if you can give me half an hour. Just getting lunch. [11:54:14] sure, Thanks Ben :) [13:52:07] 06Data-Engineering, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 13Patch-For-Review: Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#9981346 (10bking) Per IRC conversation with @hashar last week, I think it would be prudent to invite #release-engineering-team into this c... [14:14:22] 10Data-Engineering (Q4 2024 April 1st - June 30th), 13Patch-For-Review: Update MW history data quality job to use Deequ Anomaly detection Capability - https://phabricator.wikimedia.org/T362803#9981401 (10lbowmaker) 05Open→03Declined Decided not to implement this way @Snwachukwu - please add more detai... [14:14:31] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 07Spike: [SPIKE] Can we express Event Platform configs in Datasets Config? - https://phabricator.wikimedia.org/T361017#9981404 (10lbowmaker) 05Open→03Resolved [14:14:54] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Data Quality] Migrate MWHistoryChecker to DeeQu checks - https://phabricator.wikimedia.org/T361016#9981407 (10lbowmaker) 05Open→03Resolved [14:15:18] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Datasets Config][Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator - https://phabricator.wikimedia.org/T361853#9981410 (10lbowmaker... [14:15:37] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Dataset Config Store] Deploy poc to dse-k8s - https://phabricator.wikimedia.org/T357434#9981414 (10lbowmaker) [14:15:48] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Dataset Config Store] Deploy poc to dse-k8s - https://phabricator.wikimedia.org/T357434#9981415 (10lbowmaker) 05Open→03Resolved [14:18:07] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board), 07Spike: [Status Store] [SPIKE] Investigate and document approach for Iceberg Sensors - https://phabricator.wikimedia.org/T360922#9981423 (10lbowmaker) [14:18:32] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Add instrumentation for actor signatures - https://phabricator.wikimedia.org/T362783#9981421 (10lbowmaker) [14:19:12] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Add host level instrumentation on webrequest - https://phabricator.wikimedia.org/T362785#9981419 (10lbowmaker) [14:24:20] 10Data-Engineering (Q1 2024 July 1st - September 30th): [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition - https://phabricator.wikimedia.org/T354694#9981441 (10lbowmaker) [14:24:59] 10Data-Engineering (Q1 2024 July 1st - September 30th): Airflow mapped tasks UI & metrics - https://phabricator.wikimedia.org/T357430#9981445 (10lbowmaker) [14:25:08] 10Data-Engineering (Q1 2024 July 1st - September 30th): [Refine Refactoring] [Spike] Define a concept and provide a PoC for dynamic DAG execution in Airflow - https://phabricator.wikimedia.org/T356362#9981443 (10lbowmaker) [14:25:20] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 06SRE Observability: [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query - https://phabricator.wikimedia.org/T347430#9981453 (10lbowmaker) [14:25:23] 10Data-Engineering (Q1 2024 July 1st - September 30th): Replace service runner with a simplified library to better support metrics and debugging: service-utils - https://phabricator.wikimedia.org/T360924#9981447 (10lbowmaker) [14:25:56] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): MediaWiki Reconciliation API - https://phabricator.wikimedia.org/T368782#9981460 (10lbowmaker) a:03gmodena [14:25:58] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data Products, 06Structured-Data-Backlog: [Maintenance] Set up deletion jobs for Structured Data's data pipelines - https://phabricator.wikimedia.org/T347561#9981455 (10lbowmaker) [14:26:19] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06serviceops-radar: Rewrite all Airflow sensors that use datacenter prepartitions to depend on both datacenters - https://phabricator.wikimedia.org/T338796#9981457 (10lbowmaker) [14:27:10] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 13Patch-For-Review: [Data Platform] Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641#9981451 (10lbowmaker) [14:31:30] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.11; 2024-06-25): Event validation errors for mediawiki.page_change.v1 due to missing performer field on revision suppressions - https://phabricator.wikimedia.org/T367923#9981473 (10lbowmaker) [14:32:00] 06Data-Engineering: Fix DPE alerts dashboard to work with Google Groups - https://phabricator.wikimedia.org/T365829#9981477 (10lbowmaker) [14:32:14] 06Data-Engineering: Update Airflow Developer Guide on WikiTech - https://phabricator.wikimedia.org/T365658#9981479 (10lbowmaker) [14:33:08] 07Analytics-Data-Problem, 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform, 06Movement-Insights: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions - https://phabricator.wikimedia.org/T369851#9981482 (10lbowmaker) [14:34:33] 06Data-Engineering: [Datasets Config] Define and implement SLOs, monitoring and logging - https://phabricator.wikimedia.org/T362291#9981493 (10lbowmaker) [14:34:42] 06Data-Engineering, 10Data Pipelines, 10Data-Catalog: Spike: Integrate Spark with DataHub - https://phabricator.wikimedia.org/T306896#9981495 (10lbowmaker) [14:35:19] 06Data-Engineering: [Data Quality] Migrate the anomaly detection job to DeeQu checks - https://phabricator.wikimedia.org/T361014#9981499 (10lbowmaker) [14:35:41] 06Data-Engineering, 10Metrics Platform Backlog, 10Event-Platform: Document instructions for deleting an event stream and its usages - https://phabricator.wikimedia.org/T360210#9981497 (10lbowmaker) [14:35:52] 06Data-Engineering: [Spike] List out SystemD timers migration targets - https://phabricator.wikimedia.org/T361507#9981501 (10lbowmaker) [14:35:53] 06Data-Engineering: [Spike] [Maintenance] Define late arrival event strategy and idem-potent backfilling concept. - https://phabricator.wikimedia.org/T361503#9981503 (10lbowmaker) [14:35:55] 06Data-Engineering, 10Event-Platform: Implement stream of HTML content on mw.page_change event - https://phabricator.wikimedia.org/T360794#9981505 (10lbowmaker) [14:37:15] 06Data-Engineering, 10Observability-Metrics, 13Patch-For-Review, 10Sustainability (Incident Followup): Site Issue: Delayed data in the `webrequest_sampled_live` kafka topic - https://phabricator.wikimedia.org/T369737#9981507 (10fgiunchedi) [14:38:17] 06Data-Engineering, 06Research, 06Structured-Data-Backlog: Make HTML Dumps available in hadoop - https://phabricator.wikimedia.org/T305688#9981511 (10lbowmaker) [14:38:23] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Maintenance] Migrate wmcs to Airflow - https://phabricator.wikimedia.org/T357938#9981516 (10lbowmaker) 05Open→03Resolved [14:38:24] 06Data-Engineering, 07Spike: [SPIKE] [Dataset Config Store] - Design how config store feeds DataHub - https://phabricator.wikimedia.org/T360896#9981509 (10lbowmaker) [14:38:42] 06Data-Engineering: Delete reportupdater jobs data/puppet-code - https://phabricator.wikimedia.org/T358210#9981513 (10lbowmaker) [14:38:47] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): [Data Quality] Update data_quality schemas to be compatible with Iceberg tables - https://phabricator.wikimedia.org/T356866#9981515 (10lbowmaker) [14:39:08] 06Data-Engineering: Migrate and re-deploy eventgate using new service runner - https://phabricator.wikimedia.org/T361768#9981519 (10lbowmaker) [14:39:10] 06Data-Engineering: Support metrics platform backend migration to new service runner - https://phabricator.wikimedia.org/T361770#9981521 (10lbowmaker) [14:39:26] 06Data-Engineering: [Dataset Config Store] Setup initial CI checks - https://phabricator.wikimedia.org/T357468#9981523 (10lbowmaker) [14:39:31] 06Data-Engineering: [Spike] Define technology roadmap around Airflow / k8s / ceph - https://phabricator.wikimedia.org/T361509#9981525 (10lbowmaker) [14:44:53] 06Data-Engineering-Icebox: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712#9981559 (10elukey) At this point I'd proceed with the following: * Announce to Wikitech that we want to get rid or #central * File a change to stop sending events (I guess... [18:25:07] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9982962 (10Marostegui) [19:54:25] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Release-Engineering-Team, 07Spike: [Developer Experience] [SPIKE] Investigate process to automate deployment of folders and artifacts to HDFS - https://phabricator.wikimedia.org/T360968#9983226 (10lbowmaker) [20:33:11] 06Data-Engineering, 10LDAP-Access-Requests, 10SRE-Access-Requests: LDAP access to the analytics-privatedata-users group for Quiddity - https://phabricator.wikimedia.org/T370091#9983336 (10MNeisler) [23:12:59] 06Data-Engineering, 06DC-Ops, 10ops-eqiad, 06SRE: Degraded RAID on dumpsdata1007 - https://phabricator.wikimedia.org/T369829#9983662 (10Jclark-ctr) You have successfully submitted request SR194058934.