[02:40:49] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board), 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.15; 2024-07-23), 13Patch-For-Review: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#10047269 (10Ottomata) @gmoden... [06:59:40] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10047372 (10Fabfur) >>! In T370741#10041302, @Vgutierrez wrote: > we had some alerts ongoing during the weekend due to this task: > > `... [08:32:50] 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Requesting Kerberos access for ifrahkhanyaree - https://phabricator.wikimedia.org/T371894#10047522 (10BTullis) a:03BTullis I will work on this access request. [08:46:41] 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Requesting Kerberos access for ifrahkhanyaree - https://phabricator.wikimedia.org/T371894#10047558 (10BTullis) This is Ifrah's user page: https://www.mediawiki.org/wiki/User:Ifrahkhanyaree_WMDE This is the relevant LDAP account: https://ldap.t... [08:51:49] 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Requesting Kerberos access for ifrahkhanyaree - https://phabricator.wikimedia.org/T371894#10047578 (10BTullis) @Ifrahkhanyaree_WMDE I have now generated your kerberos principal. Please check your email for a message containing your initial pas... [09:24:28] 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16), 13Patch-For-Review: Requesting Kerberos access for ifrahkhanyaree - https://phabricator.wikimedia.org/T371894#10047615 (10Ifrahkhanyaree_WMDE) Thank you @BTullis! I'm having issues with ssh but that doesn't have anything to do with this ticke... [09:58:08] (03PS36) 10Aqu: Refactor Refine to be triggerd by Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [10:00:08] 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16), 13Patch-For-Review: Requesting Kerberos access for ifrahkhanyaree - https://phabricator.wikimedia.org/T371894#10047697 (10BTullis) 05Openβ†’03Resolved >>! In T371894#10047615, @Ifrahkhanyaree_WMDE wrote: > Thank you @BTullis! I'm having... [10:01:53] (03PS1) 10Sergio Gimeno: analytics/legacy/homepagemodule: remove deprecated newimpact discovery tour actions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1060409 (https://phabricator.wikimedia.org/T370120) [10:02:37] (03CR) 10CI reject: [V:04-1] analytics/legacy/homepagemodule: remove deprecated newimpact discovery tour actions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1060409 (https://phabricator.wikimedia.org/T370120) (owner: 10Sergio Gimeno) [10:06:39] (03CR) 10Sergio Gimeno: [C:04-1] "Hmm, how can we "clean up" properties in a compatible way?" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1060409 (https://phabricator.wikimedia.org/T370120) (owner: 10Sergio Gimeno) [10:59:13] 06Data-Engineering, 06collaboration-services, 10Data Pipelines, 10Data-Platform-SRE (2024.07.29 - 2024.08.16), and 2 others: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10047858 (10Stevemunene) We seem to be running into a similar challenge on protected tags as we had with the prot... [12:14:17] 06Data-Engineering, 06collaboration-services, 10Data Pipelines, 10Data-Platform-SRE (2024.07.29 - 2024.08.16), and 2 others: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10048036 (10Stevemunene) >>! In T365449#10047858, @Stevemunene wrote: > We seem to be running into a similar cha... [12:56:16] (03CR) 10Michael Große: [C:03+1] "Wording looks fine to me. Though I have no clue about the codebase and so can't say if this new copy is actually correct. Leaving that to " [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1059254 (https://phabricator.wikimedia.org/T335716) (owner: 10Sergio Gimeno) [13:14:25] !log disable puppet on an-test-client1002 to test new airflow version T365449 [13:23:16] (03CR) 10Aqu: Refactor Refine to be triggerd by Airflow (037 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) (owner: 10Aqu) [13:50:29] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10048338 (10Thryduulf) >>! In T367856#10043858, @Liz wrote: > I know, right now the replag is 61 hours. Replag is over 99 hours currently. Can we get an upd... [13:57:04] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10048344 (10Fabfur) [13:57:13] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10048350 (10Fabfur) 05Openβ†’03Resolved This has been completed and verified on hosts in ulsfo [14:18:37] (03CR) 10Ottomata: Refactor Refine to be triggerd by Airflow (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) (owner: 10Aqu) [14:21:58] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10048401 (10fnegri) > Replag is over 99 hours currently. Can we get an updated estimate of when the task will be complete The sanitarium host (db1154) comple... [14:28:04] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10048437 (10Rchard2scout) Looking at the graph you posted earlier (for anyone curious: https://grafana.wikimedia.org/d/000000303/mysql-replication-lag?orgId=1... [15:02:43] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10048558 (10fnegri) @Rchard2scout yes that's a possible explanation. The load on those servers doesn't look too high, though. [15:20:25] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10048617 (10Ladsgroup) I checked the wikireplica slave stauts. For clouddb1013 it's: > Slave_SQL_Running_State: copy to tmp table and for clouddb1017: >... [15:57:27] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Develop Airflow ExternalTaskSensor to orchestrate DAG dependencies - https://phabricator.wikimedia.org/T369900#10048730 (10amastilovic) [15:57:28] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Develop Airflow ExternalTaskSensor to orchestrate DAG dependencies - https://phabricator.wikimedia.org/T369900#10048729 (10amastilovic) [16:26:57] (03CR) 10Milimetric: [V:03+2 C:03+2] "Merging to deploy this change. It's backwards-compatible so I'll clear and re-run as needed. I've backed up the data as generated by the" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1049281 (https://phabricator.wikimedia.org/T342267) (owner: 10Milimetric) [16:30:41] !log deploying refinery to sync hql [16:30:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:53:31] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board), 07Spike: [Status Store] [SPIKE] Investigate and document approach for Iceberg Sensors - https://phabricator.wikimedia.org/T360922#10048991 (10amastilovic) [17:23:22] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Release-Engineering-Team, 07Spike: [Developer Experience] [SPIKE] Investigate process to automate deployment of folders and artifacts to HDFS - https://phabricator.wikimedia.org/T360968#10049078 (10amastilovic) [17:36:02] 10Data-Engineering (Q1 2024 July 1st - September 30th): Obtain SRE resources needed to test the HDFS synchronizer service - https://phabricator.wikimedia.org/T371994 (10amastilovic) 03NEW [17:36:41] 10Data-Engineering (Q1 2024 July 1st - September 30th): Obtain SRE resources needed to test the HDFS synchronizer service - https://phabricator.wikimedia.org/T371994#10049137 (10amastilovic) [17:36:44] 10Data-Engineering (Q1 2024 July 1st - September 30th): Implement automatic sync of refinery HQL files to HDFS - https://phabricator.wikimedia.org/T365659#10049136 (10amastilovic) [17:38:35] (03PS1) 10Clare Ming: Add bdrwiki to allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1060481 [17:39:12] (03CR) 10Mforns: [C:03+2] Add bdrwiki to allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1060481 (owner: 10Clare Ming) [17:39:16] (03CR) 10Mforns: [V:03+2 C:03+2] Add bdrwiki to allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1060481 (owner: 10Clare Ming) [18:15:20] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#10049179 (10XiaoXiao-WMF) Hi! I have followed the email instruction and I have done this step on May 23rd, and now I log into the stat machine I still... [18:15:26] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#10049184 (10XiaoXiao-WMF) 05Resolvedβ†’03Open [18:17:02] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting Kerberos access for xiaoxiao - https://phabricator.wikimedia.org/T369517#10049185 (10XiaoXiao-WMF) a:05Clement_Goubertβ†’03None [18:18:43] (03PS1) 10Milimetric: Hotfix: update to match iceberg version [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1060486 [18:21:09] (03CR) 10Milimetric: [V:03+2 C:03+2] "tested to verify this version works, tested for logic in previous change to the iceberg file, merging to deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1060486 (owner: 10Milimetric) [18:22:45] 10Data-Engineering (Q1 2024 July 1st - September 30th): Implement automatic sync of refinery HQL files to HDFS - https://phabricator.wikimedia.org/T365659#10049227 (10amastilovic) I talked to @BTullis about obtaining a functional test environment that would mimic the real world this service would be operating in... [18:48:32] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#10049261 (10Ottomata... [19:01:44] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#10049309 (10Ottomata... [19:12:41] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#10049330 (10Ottomata... [19:45:53] !log deploying airflow-dags/analytics for browser general daily dag [19:45:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:47:45] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): Problem deploying - missing airflow_client dependency - https://phabricator.wikimedia.org/T372014 (10Milimetric) 03NEW [21:09:19] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 17), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#10049723 (10Milimetric) Status update... [21:27:37] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 17), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#10049760 (10Milimetric) For reference... [21:35:48] (03PS37) 10Aqu: Refactor Refine to be triggerd by Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [21:41:06] (03CR) 10Aqu: Refactor Refine to be triggerd by Airflow (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) (owner: 10Aqu) [21:42:32] 10Analytics-Canonical-Data, 06Movement-Insights: Periodically update the canonical wiki dataset while Neil is on sabbatical - https://phabricator.wikimedia.org/T372018 (10nshahquinn-wmf) 03NEW p:05Triageβ†’03Medium [21:42:43] 10Analytics-Canonical-Data, 06Movement-Insights: Periodically update the canonical wiki dataset while Neil is on sabbatical - https://phabricator.wikimedia.org/T372018#10049796 (10nshahquinn-wmf) [21:49:24] 10Analytics-Canonical-Data, 06Movement-Insights: Periodically update the canonical wiki dataset while Neil is on sabbatical - https://phabricator.wikimedia.org/T372018#10049803 (10nshahquinn-wmf) Currently there is one new wiki waiting to be added: https://bdr.wikipedia.org/. The data has not been added to the...