[00:10:50] 14Analytics-Radar, 06Data-Engineering-Icebox, 06Fundraising-Backlog, 10fundraising-tech-ops: Bring Banner History data into Fundraising infrastructure - https://phabricator.wikimedia.org/T253050#9971837 (10AKanji-WMF) [03:56:22] 06Data-Engineering, 10SRE-Access-Requests: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#9972112 (10Pppery) [07:35:20] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9972333 (10Marostegui) [08:15:11] 06Data-Engineering, 06Data Products, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9972434 (10Marostegui) [09:20:03] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.14; 2024-07-16), 13Patch-For-Review: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#9972761 (10gmodena) [09:25:56] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.14; 2024-07-16), 13Patch-For-Review: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#9972784 (10gmodena) Instrumentation has been enabled in beta. You... [09:29:54] !log temporarily disabled gobblin ingestion to facilitate an-mariadb role swap. [09:29:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:57:51] 06Data-Engineering, 10Sustainability (Incident Followup): Site Issue: Delayed data in the `webrequest_sampled_live` Druid table - https://phabricator.wikimedia.org/T369737#9972847 (10fgiunchedi) I've taken a look at this as well for the benthos bits, and there was significant kafka lag for the benthos consumer... [10:09:39] !log swapping an-mariadb100[1-2] roles back. [10:09:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:14:35] !log failed back hive and presto services to an-coord1003 [13:14:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:17:46] !log draining dse-k8s-worker1007 ready for T365996 [13:17:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:17:49] T365996: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996 [13:18:03] !log setting cephosd cluster to noout mode for T365996 [13:18:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:19:40] 06Data-Engineering, 10Sustainability (Incident Followup): Site Issue: Delayed data in the `webrequest_sampled_live` Druid table - https://phabricator.wikimedia.org/T369737#9973615 (10fgiunchedi) A bit of context: benthos@webrequest_live normally runs on centrallog1002 and centrallog2002 consuming the webreques... [14:21:29] 06Data-Engineering, 10Sustainability (Incident Followup): Site Issue: Delayed data in the `webrequest_sampled_live` kafka topic - https://phabricator.wikimedia.org/T369737#9973625 (10fgiunchedi) [14:24:19] 06Data-Engineering, 10Observability-Metrics, 10Sustainability (Incident Followup): Site Issue: Delayed data in the `webrequest_sampled_live` kafka topic - https://phabricator.wikimedia.org/T369737#9973631 (10fgiunchedi) [14:25:07] 06Data-Engineering, 10Observability-Metrics, 10Sustainability (Incident Followup): Site Issue: Delayed data in the `webrequest_sampled_live` kafka topic - https://phabricator.wikimedia.org/T369737#9973638 (10fgiunchedi) a:05BTullis→03None [14:47:01] 06Data-Engineering, 06DC-Ops, 10ops-eqiad, 06SRE: Degraded RAID on dumpsdata1007 - https://phabricator.wikimedia.org/T369829#9973802 (10Marostegui) [14:59:42] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Dumps 2] Spike: Figure root causes of missing rows when doing reconciliation - https://phabricator.wikimedia.org/T368176#9973872 (10xcollazo) Very interesting. >>! In T368176#9971336, @Ottomata wrote: >> we seem to just be missing all page d... [15:05:20] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Dumps 2] Spike: Figure root causes of missing rows when doing reconciliation - https://phabricator.wikimedia.org/T368176#9973909 (10Ottomata) > page_content_change only cares about the latest revision, not the historical revisions. Right, bu... [15:13:47] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Dumps 2] Spike: Figure root causes of missing rows when doing reconciliation - https://phabricator.wikimedia.org/T368176#9973938 (10Milimetric) Ok, I think I got this query to make sense... the results: ` presto:milimetric> select reasons, c... [20:04:05] 06Data-Engineering, 10CirrusSearch, 03Discovery-Search (Current work), 13Patch-For-Review: [Search Update Pipeline] Source streams for private wikis - https://phabricator.wikimedia.org/T346046#9974999 (10Ottomata) > I wonder if we shouldn't consider just removing and refactoring that, and to make use of th... [20:05:20] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Refine Refactoring] Switch new Refine system outputs to production location and monitor - https://phabricator.wikimedia.org/T369845#9975020 (10Ottomata) [20:05:21] 06Data-Engineering, 10Data Pipelines: Refine jobs should be scheduled by Airflow - https://phabricator.wikimedia.org/T307505#9975021 (10Ottomata) [20:59:02] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Request for Kerb credentials for Ariel Glenn - https://phabricator.wikimedia.org/T368911#9975173 (10Dzahn) @ArielGlenn You are now in the additional group and I created a Kerberos principal. You should have received an email with instru... [20:59:32] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Request for Kerb credentials for Ariel Glenn - https://phabricator.wikimedia.org/T368911#9975174 (10Dzahn) a:03ArielGlenn [21:50:29] 07Analytics-Data-Problem, 06Data-Platform, 06Movement-Insights: NEW BUG REPORT Mediawiki_history contains duplicate rows for some revisions - https://phabricator.wikimedia.org/T369851#9975293 (10nshahquinn-wmf) This may help in diagnosing the problem: looking at the snapshot, the number of duplicates is not...