[01:55:36] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Data Quality] Migrate MWHistoryChecker to DeeQu checks - https://phabricator.wikimedia.org/T361016#9916254 (10Snwachukwu) There is a fix to resolve the Null input Exceptions experienced after deploying the migration. We get null values in mediawiki history bec... [05:41:36] 06Data-Engineering, 10Dumps-Generation, 06SRE: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9916375 (10Joe) [05:42:06] 06Data-Engineering, 10Dumps-Generation, 06SRE: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9916373 (10Joe) [07:45:05] 06Data-Engineering, 10Dumps-Generation, 06SRE: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9916447 (10MatthewVernon) [if a proper fix is going to be time-consuming, at least only running dumps when there are staff around would b... [08:04:16] 06Data-Engineering, 10Dumps-Generation, 06SRE: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9916477 (10MatthewVernon) [09:05:00] 06Data-Engineering, 10Dumps-Generation, 06SRE: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9916740 (10Ladsgroup) >>! In T368098#9916447, @MatthewVernon wrote: > [if a proper fix is going to be time-consuming, at least only runni... [09:05:32] 06Data-Engineering, 10Dumps-Generation, 06SRE: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9916743 (10Joe) >>! In T368098#9916447, @MatthewVernon wrote: > [if a proper fix is going to be time-consuming, at least only running dum... [09:12:55] (03PS20) 10Aqu: Refactor Refine to be triggerd by Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [09:13:22] 06Data-Engineering, 10Dumps-Generation, 06SRE: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9916796 (10Joe) I think actually @Ladsgroup's proposal seems like the easier solution on the short term. @xcollazo do you see any complic... [09:26:14] (03PS1) 10KCVelaga: [WIP] [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/1049106 (https://phabricator.wikimedia.org/T362615) [09:28:56] (03PS2) 10KCVelaga: Add Automoderator metrics to WMF Product jobs [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/1049106 (https://phabricator.wikimedia.org/T362615) [11:34:35] (03PS1) 10KCVelaga: Add Automoderator metrics to WMF Product jobs [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/1049133 (https://phabricator.wikimedia.org/T362615) [11:35:58] (03Abandoned) 10KCVelaga: Add Automoderator metrics to WMF Product jobs [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/1049133 (https://phabricator.wikimedia.org/T362615) (owner: 10KCVelaga) [11:54:09] 06Data-Engineering, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 10 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#9917252 (10Jgiannelos) [15:38:51] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Investigate increase in CD termination state after upgrading eqsin/ulsfo to HAProxy 2.8.10 - https://phabricator.wikimedia.org/T367963#9918190 (10Vgutierrez) 05Open→03Resolved a:05Fabfur→03Vgutierrez cache_haproxy.mtail failed to accept -1 as an... [15:46:40] 06Data-Engineering, 10Dumps-Generation, 06SRE, 13Patch-For-Review: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9918255 (10colewhite) [15:54:14] 06Data-Engineering, 06Tech-Docs-Team, 07Epic, 05Goal: Redesign Data Platform docs on Wikitech - https://phabricator.wikimedia.org/T350911#9918281 (10TBurmeister) [16:01:40] 06Data-Engineering, 10Dumps-Generation, 06SRE, 13Patch-For-Review: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9918320 (10xcollazo) Ok, I want to revise my previous assessment. Nominal network usage with bursts up to ~90 MB/s... [16:03:31] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data Products (Data Products Sprint 15), 13Patch-For-Review: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9918323 (10xcollazo) [16:08:12] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services Migration🐤), 13Patch-For-Review: Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9918332 (10Snwachukwu) [16:45:55] 06Data-Engineering, 06Data Products: Past edits increase in wmf.edit_hourly with every new snapshot - https://phabricator.wikimedia.org/T355182#9918569 (10mpopov) 05Open→03Resolved a:03mpopov Sounds good, yeah! Thanks for closing the loop on this. [16:48:11] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 15), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9918574 (10mforns) So cool @Milimetri... [17:07:55] 06Data-Engineering-Icebox: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712#9918667 (10Krinkle) In light of T348388, I'm checking to make sure SULWatcher does not rely on creations happening on login.wikimedia.org. In reviewing its code, I find th... [17:44:23] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data Products (Data Products Sprint 15), 13Patch-For-Review: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9918849 (10xcollazo) Coming back to: >>! In T368098#9916796, @Joe wro... [17:54:32] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: Spike: Figure root cause of missing rows when doing reconciliation - https://phabricator.wikimedia.org/T368176#9918900 (10Ottomata) [18:06:14] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data Products (Data Products Sprint 15), 13Patch-For-Review: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9918925 (10xcollazo) Ok after observing the dumps for a bit now, here... [18:07:39] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#9918941 (10Ottomata) >> event service name (eventgate name) > We can prase this from $this->url It won't hurt to add the event servic... [18:14:03] (03PS1) 10Triciaburmeister: Update links to documentation on wikitech [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1049252 (https://phabricator.wikimedia.org/T357327) [18:19:41] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#9918977 (10gmodena) >>! In T363587#9918941, @Ottomata wrote: >>> event service name (eventgate name) >> We can prase this from $this->... [18:35:53] 06Data-Engineering-Icebox: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712#9919028 (10Ottomata) > SULWatcher does not rely on creations happening on login.wikimedia.org. In reviewing its code, I find that it is using stream.wikimedia.org and look... [19:11:49] 06Data-Engineering-Icebox: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712#9919146 (10Krinkle) >>! In T242712#9919028, @Ottomata wrote: >> SULWatcher does not rely on creations happening on login.wikimedia.org. In reviewing its code, I find that... [20:27:14] 10Quarry: refreshing a running query changes favicon from orange to blue - https://phabricator.wikimedia.org/T362101#9919300 (10SD0001) 05Open→03Resolved a:03SD0001 [20:27:40] 06Data-Engineering-Icebox: Deprecation (if possible) of the #central channel on irc.wikimedia.org - https://phabricator.wikimedia.org/T242712#9919302 (10Ottomata) Thanks! [20:41:05] (03CR) 10Ottomata: Refactor Refine to be triggerd by Airflow (0344 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) (owner: 10Aqu) [20:47:08] (03PS1) 10Triciaburmeister: Update links to docs and repo [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1049265 (https://phabricator.wikimedia.org/T357327) [20:55:53] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data Products (Data Products Sprint 15), 13Patch-For-Review: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9919385 (10ssingh) This happened today again, starting at 20:05 and re... [21:06:30] 06Data-Engineering, 10Data Pipelines: Refine jobs should be scheduled by Airflow - https://phabricator.wikimedia.org/T307505#9919435 (10Ottomata) @Antoine_Quhen can you remind me? How did we over come the 'gotcha' described in the task description here? > The main issue is that the source data for the Refine... [21:08:18] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data Products (Data Products Sprint 15), 13Patch-For-Review: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9919438 (10Dzahn) This took basically everything down. Wikis returned... [21:21:08] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9919468 (10Dzahn) [21:31:24] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9919486 (10Dzahn) Hello, does this have approval from one of the owners of analytics-privatedata? ` approval: - Olja Dimitrjevic -... [21:31:45] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9919473 (10Dzahn) - Confirmed approving party via orgchart in Betterworks. - Tagging Data-Engineering for group approval [21:35:53] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9919504 (10Dzahn) Hello @Dreamy_Jazz are you requesting membership with or without Kerberos? Here is more on the difference: https://wikitech.wiki... [21:36:23] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9919505 (10Dzahn) 05Open→03In progress [21:36:31] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9919506 (10Dzahn) p:05Triage→03High [21:39:04] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9919508 (10Dzahn) Hello @cwylo are you requesting membership with or without Kerberos? Here is more on the difference: https://wik... [21:40:33] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9919509 (10Dzahn) 05Open→03In progress p:05Triage→03High [21:40:34] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9919513 (10Dzahn) [21:42:43] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9919514 (10Dzahn) a:03leila Hello Leila, does this have your approval? It's just about the web-based logins, right? [21:45:26] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to private data-based dashboards for Jsn.sherman - https://phabricator.wikimedia.org/T367295#9919533 (10Dzahn) [22:48:51] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data Products (Data Products Sprint 15), 13Patch-For-Review: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9919665 (10Scott_French) I just spot-checked some of the `runPrimaryTr... [22:50:37] (03CR) 10Pppery: "See also https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1048026" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1048496 (https://phabricator.wikimedia.org/T368038) (owner: 10Xcollazo) [22:51:14] (03CR) 10Pppery: "Also note that gerritbot doesn't detect the bug number if there's a newline between it and the change-id" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1048496 (https://phabricator.wikimedia.org/T368038) (owner: 10Xcollazo) [23:04:09] 06Data-Engineering, 06Data-Platform-SRE: Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#9919694 (10amastilovic) >>! In T368033#9909148, @BTullis wrote: > My feeling is that, at our scale and given its proximity to the dse-k8s cluster, Ceph will be just fine for these DAG vo... [23:05:44] 06Data-Engineering, 06Data-Platform-SRE: Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#9919695 (10amastilovic) >>! In T368033#9912923, @BTullis wrote: > In some ways, the fundamental question is: do we want to move to a //continuous deployment// model, or do we want to ret...