[07:02:42] 10Analytics-Wikistats, 10Data-Engineering, 10NFDI: Is it possible to setup wikistats for a new wiki? - https://phabricator.wikimedia.org/T308253 (10Physikerwelt) [10:01:40] 10Analytics, 10Metrics-Platform: TypeError: navigator.sendBeacon is not a function - https://phabricator.wikimedia.org/T308311 (10AlexisJazz) [10:12:44] (03CR) 10Joal: "Some more comments, plus closing of the original ones. It's good practice to answer comments and set them "resolved" when pushing a new pa" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/790651 (https://phabricator.wikimedia.org/T306895) (owner: 10NOkafor) [10:13:47] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [POC] Use airflow-installed Spark3 for an Airflow job - https://phabricator.wikimedia.org/T308168 (10JAllemandou) [10:16:15] 10Analytics, 10Metrics-Platform: TypeError: navigator.sendBeacon is not a function - https://phabricator.wikimedia.org/T308311 (10AlexisJazz) [10:28:34] 10Analytics, 10Data-Engineering-Radar, 10Event-Platform, 10Metrics-Platform, 10Browser-Support-Microsoft-Edge: Problem with delay caused by intake-analytics.wikimedia.org - https://phabricator.wikimedia.org/T295427 (10AlexisJazz) >>! In T295427#7817708, @phuedx wrote: > The latter seems more likely to me... [10:30:40] 10Analytics, 10Metrics-Platform: TypeError: navigator.sendBeacon is not a function - https://phabricator.wikimedia.org/T308311 (10phuedx) a:03phuedx [10:41:46] 10Analytics, 10Metrics-Platform, 10Patch-For-Review: TypeError: navigator.sendBeacon is not a function - https://phabricator.wikimedia.org/T308311 (10phuedx) Thanks for the bug report, @AlexisJazz! [10:43:57] (03CR) 10Joal: "Idea for optimisation and some naming. Small things :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/791394 (https://phabricator.wikimedia.org/T300021) (owner: 10Snwachukwu) [10:44:50] 10Analytics, 10Data-Engineering-Radar, 10Event-Platform, 10Metrics-Platform, 10Browser-Support-Microsoft-Edge: Problem with delay caused by intake-analytics.wikimedia.org - https://phabricator.wikimedia.org/T295427 (10BTullis) This is very interesting to me because I'm currently investigating several iss... [10:45:53] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Upgrade Turnilo - https://phabricator.wikimedia.org/T301990 (10ayounsi) FYI, 1.36.0 is out but no significant changes. https://github.com/allegro/turnilo/releases/tag/1.36.0 [11:04:40] 10Analytics, 10Metrics-Platform, 10Patch-For-Review: TypeError: navigator.sendBeacon is not a function - https://phabricator.wikimedia.org/T308311 (10AlexisJazz) No problem, thanks for fixing! [11:25:41] 10Analytics, 10Data-Engineering-Radar, 10Event-Platform, 10Metrics-Platform, 10Browser-Support-Microsoft-Edge: Problem with delay caused by intake-analytics.wikimedia.org - https://phabricator.wikimedia.org/T295427 (10AlexisJazz) >>! In T295427#7926809, @BTullis wrote: > Any evidence of requests to these... [12:45:28] 10Data-Engineering, 10Epic: Data Infrastructure as a Service platform MVP - https://phabricator.wikimedia.org/T308317 (10BTullis) [13:02:53] 10Data-Engineering-Kanban, 10Airflow, 10Documentation: [Airflow] Kick off documentation in wikitech - https://phabricator.wikimedia.org/T302400 (10EChetty) [13:04:03] 10Data-Engineering, 10Data-Engineering-Kanban: Draft initial data storage platform and place budget hold for Q2 - https://phabricator.wikimedia.org/T308318 (10BTullis) [13:04:46] 10Data-Engineering, 10Data-Engineering-Kanban: Draft initial data storage platform and place budget hold for Q2 - https://phabricator.wikimedia.org/T308318 (10BTullis) p:05Triage→03High [13:05:43] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Adapt maxExecutors value by Dag - https://phabricator.wikimedia.org/T307447 (10JAllemandou) a:03JAllemandou [13:10:45] mforns: I talked to Andrew last night and here's where we got: [13:10:56] ok [13:11:14] we'd need a new repo in gitlab, or to migrate refinery to gitlab if it's going to be able to use the gitlab CI stuff [13:11:40] milimetric: another idea! [13:11:48] you could just use conda-dist to make your thing (in refinery?) [13:12:00] until then, we could make a folder in refinery and manually call workflow-utils' conda-dist to build the environment we need, and use it as follows: [13:12:05] upload it to archiva, and add it as a git fat artifact [13:12:18] i guess that's not that different than using the artifact syncing to do it [13:13:41] just put the zip file directly in airflow-dags/analytics/dags/my-new-ingestion-folder/ and pass it to skein by modifying SimpleSkeinOperator to take a files parameter [13:14:09] I just want to do whatever is closest to what we'll eventually do, and as I understood it, this way we are close in two ways: [13:14:26] 1.) the folder in refinery will eventually get taken care of automagically by CI [13:14:59] 2.) the new files= parameter would need to be there anyway, but it would take an artifact(<>) in the future, so that'll be a simple change once the gitlab CI magic is in place [13:15:01] thoughts? [13:15:43] using conda-dist to build the env from refinery makes sense to me [13:15:59] whay though should we put it in airflow-dags? [13:16:06] *why [13:18:00] as a temporary hold until we get the CI magic to put it in gitlab downloads [13:19:10] milimetric: you can put it in archiva [13:19:20] and either the artifact syncing or git fat stuff will work [13:19:25] manually? [13:19:27] yes [13:19:43] not worse then manually git-adding it in airflow-dags :) [13:20:14] are we long-term going to put it in archiva instead of gitlab downloads? [13:20:40] 'cause then I can make the artifacts.yaml work with that and the airflow code won't change much at all [13:21:15] I like this option a bit better, precisely because of what Dan says, DAG code won't change [13:21:37] If we move the artifact to GitLab at some point, we only need to change the artifact_config.yaml [13:21:57] ok, cool, doing it this way, code reviews incoming sometime today [13:22:14] plus we don't add artifacts to the airflow-dags code [13:22:44] ok, thanks for bringing this discussion! [13:23:07] (03PS1) 10Vivian Rook: Return 404 on query ids that do not exist [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/791606 (https://phabricator.wikimedia.org/T290874) [13:23:40] 10Quarry, 10Patch-For-Review, 10cloud-services-team (Kanban): Quarry returns 500 rather than 404 when asked for an invalid query ID - https://phabricator.wikimedia.org/T290874 (10rook) a:03rook [13:27:32] (03CR) 10jerkins-bot: [V: 04-1] Return 404 on query ids that do not exist [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/791606 (https://phabricator.wikimedia.org/T290874) (owner: 10Vivian Rook) [13:50:40] 10Data-Engineering, 10Data-Engineering-Kanban: Draft initial data storage platform and place budget hold for Q2 - https://phabricator.wikimedia.org/T308318 (10BTullis) == Storage node spec == === Head Node === The five servers would be a custom build since they do not closely match an existing entry from the... [13:58:38] 10Data-Engineering, 10Data-Engineering-Kanban: Draft initial data storage platform and place budget hold for Q2 - https://phabricator.wikimedia.org/T308318 (10BTullis) ==MVP Cluster Capacity== The configuration set out above would give a total capacity of: 50 x 3.8 = **190 TB** for the hot tier 60 x 18 = **1... [15:04:14] 10Data-Engineering, 10Data-Engineering-Kanban: Draft initial data storage platform and place budget hold for Q2 - https://phabricator.wikimedia.org/T308318 (10BTullis) Without having sought a quote from Dell on this, all I can do it to estimate the costs. My estimate is: * Head node = $8,000 each * JBOD excl... [15:06:03] 10Data-Engineering, 10Epic: Data Infrastructure as a Service MVP - https://phabricator.wikimedia.org/T308317 (10BTullis) [15:30:07] (03PS2) 10Vivian Rook: Return 404 on query ids that do not exist [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/791606 (https://phabricator.wikimedia.org/T290874) [15:30:43] 10Data-Engineering, 10Data-Engineering-Kanban: Draft initial data storage platform and place budget hold for Q2 - https://phabricator.wikimedia.org/T308318 (10EChetty) This MVP will hopefully Run in Q1, not Q2 [15:36:26] mforns: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/commit/62467d6612d96fb1135346083a91add5f3c1214a is the airflow side, haven't built the environment yet, but see if you see anything clearly broken there (don't worry I'll test to make sure all the details line up) [16:18:32] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10akosiaris) The 50% bump in capacity didn't make any noticeable difference this time around. :-( [16:27:22] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) >>! In T306181#7927769, @akosiaris wrote: > The 50% bump in capacity didn't make any... [16:30:02] 10Data-Engineering, 10Data-Engineering-Kanban: Draft initial data storage platform and place budget hold for Q2 - https://phabricator.wikimedia.org/T308318 (10BTullis) Moving to in-review while we share the draft spec and invite comments. [16:31:38] 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: User Experience: Authentication - https://phabricator.wikimedia.org/T307711 (10BTullis) John Joyce from DataHub has acknowledged our issue and is aware of a path to fix, but hasn't yet committed to a fix. {F35138630,width=100%} I've offered to re... [17:02:54] 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: User Experience: Authentication - https://phabricator.wikimedia.org/T307711 (10BTullis) I have created an upstream bug report, as per John's request: https://github.com/datahub-project/datahub/issues/4915 [17:19:47] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow, 10Patch-For-Review: Set up backups and monitoring of airflow instances - https://phabricator.wikimedia.org/T307102 (10BTullis) The airflow checks are now working correctly. {F35138742,width=70%} [17:20:11] OK, I'm off for now. Have great weekends folks. :-) [19:34:36] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10AlexisJazz) >>! In T306181#7927825, @BTullis wrote: >>>! In T306181#7927769, @akosiaris wrote... [19:44:15] 10Data-Engineering-Kanban, 10Data-Catalog: Custom Metadata ingestion - https://phabricator.wikimedia.org/T307714 (10Milimetric) p:05Triage→03High a:03Milimetric [20:09:55] (03PS1) 10Vivian Rook: Update stop status directly and catch error [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/791669 (https://phabricator.wikimedia.org/T290146) [20:47:07] 10Quarry: Escape special characters in results - https://phabricator.wikimedia.org/T308362 (10Bugreporter) [22:25:08] (03CR) 10Bartosz Dziewoński: [C: 03+2] DesktopUIActions/MobileUIActions: add pageToken field [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/789324 (https://phabricator.wikimedia.org/T307640) (owner: 10DLynch) [22:25:51] (03Merged) 10jenkins-bot: DesktopUIActions/MobileUIActions: add pageToken field [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/789324 (https://phabricator.wikimedia.org/T307640) (owner: 10DLynch) [23:07:24] (03PS4) 10Razzi: Upgrade to upstream version 1.35.0 [analytics/turnilo/deploy] - 10https://gerrit.wikimedia.org/r/777881 (https://phabricator.wikimedia.org/T301990) [23:34:10] (03PS5) 10Razzi: Upgrade to upstream version 1.35.0 [analytics/turnilo/deploy] - 10https://gerrit.wikimedia.org/r/777881 (https://phabricator.wikimedia.org/T301990)