[02:56:33] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10050030 (10Liz) Well, it didn't finish in less than a day. Currently a 112 hour lag. [06:24:14] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#10050082 (10SLyngshede-WMF) Hi @XiaoXiao-WMF, when logging in, please try to run the command "kinit" that should prompt you for your password, and will give you a valid Kerber... [08:03:52] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10050182 (10Ladsgroup) >>! In T367856#10050030, @Liz wrote: > Well, it didn't finish in less than a day. Currently a 112 hour lag. Your comment was just 12... [09:24:41] !log restarting mariadb instances on dbstore100[7-9] to pick up binlog change for T370354 [09:24:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:24:49] T370354: [SPIKE] Experiment with approaches for a incremental updates of MediaWiki data in the Data Lake - https://phabricator.wikimedia.org/T370354 [09:36:41] !log rebooting dbstore1009 due to CPU soft lockup. [09:36:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:08:29] 06Data-Engineering, 06Product-Analytics, 07Epic, 13Patch-For-Review: [SPIKE] Experiment with approaches for a incremental updates of MediaWiki data in the Data Lake - https://phabricator.wikimedia.org/T370354#10050514 (10BTullis) I have enabled the binlog on all of these servers and restarted the services.... [12:27:40] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#10050899 (10XiaoXiao-WMF) HI @SLyngshede-WMF, I have done these steps a while ago, here is what I see - it seems that it is only valid for a few days, does it look right to yo... [12:32:51] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#10050928 (10SLyngshede-WMF) Yes, the tickets are short lived, so you need to run the kinit regularly. See: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Kerberos... [12:37:25] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10050972 (10Nintendofan885) web.db.svc.wikimedia.cloud has now caught up [13:14:43] 06Data-Engineering, 06Product-Analytics, 07Epic, 13Patch-For-Review: [SPIKE] Experiment with approaches for a incremental updates of MediaWiki data in the Data Lake - https://phabricator.wikimedia.org/T370354#10051128 (10Ottomata) I think we'd need `SELECT` as well, but I think that should be it! [13:56:15] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10051296 (10GTrang) But the enwiki replag is still 5 days. [14:18:45] 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): request for new matomo site: trace.wikimedia.org/ - https://phabricator.wikimedia.org/T371124#10051405 (10BTullis) p:05Triage→03Medium [14:35:09] 06Data-Engineering, 06Data-Platform-SRE, 06SRE: Streamline Data Platform access approvals for WMF staff - https://phabricator.wikimedia.org/T370424#10051517 (10BTullis) This sounds sensible to me, too. Given that we have approval for the proposal as it stands, how do we proceed? Do we just need to update th... [15:15:50] 06Data-Engineering, 06Data-Platform-SRE, 06SRE: Streamline Data Platform access approvals for WMF staff - https://phabricator.wikimedia.org/T370424#10051639 (10Ottomata) That and maybe some comments in puppet admin data.yaml to instruct SREs on the right thing to do? [15:21:22] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10051678 (10fnegri) As shown in https://replag.toolforge.org/ replag is 5 days on `analytics.db.svc.wikimedia.cloud` (clouddb1017), but it's back to zero on `... [15:45:07] 06Data-Engineering, 03Discovery-Search (Current work), 10MW-1.43-notes (1.43.0-wmf.17; 2024-08-06), 07Wikimedia-production-error: '.event.pageViewId' should be string, '.event.subTest' should be string, '.event.searchSessionId' should be string - https://phabricator.wikimedia.org/T286814#10051779 (10EBernha... [16:01:12] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#10051874 (10XiaoXiao-WMF) okay, so how can I renew this now? I have passed the 7/22 deadline [16:13:04] 10Data-Engineering (Q1 2024 July 1st - September 30th): Implement automatic sync of refinery HQL files to HDFS - https://phabricator.wikimedia.org/T365659#10051921 (10Ottomata) [16:21:55] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data Pipelines, 10Data-Catalog: Spike: Integrate Spark with DataHub - https://phabricator.wikimedia.org/T306896#10051950 (10Ottomata) [17:13:33] 06Data-Engineering, 06Data Products, 07Schema-change: Test task for herald rule - https://phabricator.wikimedia.org/T372084 (10odimitrijevic) 03NEW [17:13:46] 06Data-Engineering, 06Data Products, 07Schema-change: Test task for herald rule - https://phabricator.wikimedia.org/T372084#10052201 (10odimitrijevic) 05Open→03Invalid [17:19:46] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#10052216 (10BTullis) >>! In T369517#10051874, @XiaoXiao-WMF wrote: > okay, so how can I renew this now? I have passed the 7/22 deadline You just have to run `kinit` and then... [17:33:17] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#10052258 (10XiaoXiao-WMF) Okay perfect. Thanks, I will mark this as resolve! [17:34:17] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#10052259 (10XiaoXiao-WMF) 05Open→03Resolved a:03XiaoXiao-WMF [18:39:17] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10052491 (10Liz) The replag is at 0 but when I do Quarry queries, the information is all from August 3rd, nothing more recent than that and there should be a... [19:56:04] 06Data-Engineering, 10Data Pipelines: Add support for repository artifacts in Airflow - https://phabricator.wikimedia.org/T322690#10052719 (10Ottomata) We discussed the above in a [[ https://docs.google.com/document/d/1b-TvjlllQ7sGmk4ysWUi33j2Za93daZPVOaxRG8djY0/edit#heading=h.lyd1v7kpf6cb | Data Eng Sync meet... [19:58:03] 06Data-Engineering, 10Data Pipelines: Add support for repository artifacts in Airflow - https://phabricator.wikimedia.org/T322690#10052736 (10Ottomata) In today's [[ https://docs.google.com/document/d/1b-TvjlllQ7sGmk4ysWUi33j2Za93daZPVOaxRG8djY0/edit#heading=h.r0tqn2jg0kuz | Data Eng Sync meeting ]], we discus...