[00:28:39] (03PS2) 10Sharvaniharan: New schema for edit history screen interactions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772934 (https://phabricator.wikimedia.org/T304336) [00:29:58] (03CR) 10Sharvaniharan: "Please review when you get a chance :)" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772934 (https://phabricator.wikimedia.org/T304336) (owner: 10Sharvaniharan) [00:30:42] (03CR) 10Sharvaniharan: "Please review when you get a chance :)" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772910 (https://phabricator.wikimedia.org/T304335) (owner: 10Sharvaniharan) [02:06:16] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [02:11:16] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [04:28:54] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: The network_internal druid load job fails if data is not present - https://phabricator.wikimedia.org/T302263 (10odimitrijevic) Now that drmrs dc is operational this should be resided upon. As part of the work let's ensure that the data is col... [04:30:28] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10MediaWiki-extensions-EventLogging: Non-deterministic unit test "streamInSample() - session sampling resets" - https://phabricator.wikimedia.org/T304379 (10odimitrijevic) [04:30:50] 10Data-Engineering-Radar, 10Event-Platform, 10MediaWiki-extensions-EventLogging: Non-deterministic unit test "streamInSample() - session sampling resets" - https://phabricator.wikimedia.org/T304379 (10odimitrijevic) [04:31:21] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of clarakosi - https://phabricator.wikimedia.org/T304065 (10odimitrijevic) p:05Triage→03High [04:35:07] 10Data-Engineering, 10Data-Engineering-Kanban: Archiva's disk partiton space is getting filled up - https://phabricator.wikimedia.org/T304224 (10odimitrijevic) p:05Medium→03High [04:41:08] 10Data-Engineering, 10SRE, 10Traffic, 10Trust-and-Safety, 10serviceops: Disable GeoIP Legacy Download - https://phabricator.wikimedia.org/T303464 (10odimitrijevic) [06:16:06] 10Data-Engineering, 10Data-Services, 10cloud-services-team (Kanban): Reimage WMCS db proxies to Bullseye - https://phabricator.wikimedia.org/T298940 (10Marostegui) Any ETA on this task or {T273278}? [06:16:26] 10Analytics, 10Data-Engineering: Upgrade dbstore100* hosts to Bullseye - https://phabricator.wikimedia.org/T299481 (10Marostegui) @odimitrijevic any ETA on this? Thanks! [06:29:19] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Data-Services, and 2 others: Recreate views for globaluser table - https://phabricator.wikimedia.org/T301674 (10Marostegui) 05Open→03Resolved a:03Marostegui I have finally done this myself. [07:10:40] 10Data-Engineering, 10Data-Services: Move wikireplicas dbproxy haproxy config to etcd - https://phabricator.wikimedia.org/T304478 (10razzi) [07:24:26] 10Data-Engineering, 10Data-Services: Move wikireplicas dbproxy haproxy config to etcd - https://phabricator.wikimedia.org/T304478 (10razzi) @Joe I'm interested in your input, since you mentioned etcd is the way to go here - does the above plan make sense? If so, for etcd data modeling, would it make sense to... [07:56:39] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Recreate views for globaluser table - https://phabricator.wikimedia.org/T301674 (10Majavah) 05Resolved→03Open I still see the gu_hidden and gu_enabled fields in the live tables: ` u21215@centralauth... [09:22:53] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Recreate views for globaluser table - https://phabricator.wikimedia.org/T301674 (10Marostegui) 05Open→03Resolved Fixed, looks like I ran the script right before merging the second patch. ` root@clo... [10:28:41] 10Analytics, 10Data-Engineering: Upgrade db1108 to Bullseye - https://phabricator.wikimedia.org/T304492 (10Marostegui) [10:29:06] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: The network_internal druid load job fails if data is not present - https://phabricator.wikimedia.org/T302263 (10BTullis) I have re-enabled the task to collect the sflow data. Once it has run we should be able to verify that the data is corre... [12:16:00] 10Data-Engineering, 10Data-Engineering-Kanban, 10Superset: Superset SQL Lab fails to stop query - https://phabricator.wikimedia.org/T293083 (10KCVelaga_WMF) @razzi sorry, I missed this ping. I have been using Superset consistently, and I no longer face this issue. I will report back (along with the query) if... [12:36:31] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: The network_internal druid load job fails if data is not present - https://phabricator.wikimedia.org/T302263 (10BTullis) 05Open→03Declined p:05Triage→03Medium I can see data in Turnilo for the network_flows_internal stream, so I think... [13:01:04] (03CR) 10Ottomata: New schema for edit history screen interactions (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772934 (https://phabricator.wikimedia.org/T304336) (owner: 10Sharvaniharan) [13:02:41] (03CR) 10Ottomata: New schema for measuring article screen interactions (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772910 (https://phabricator.wikimedia.org/T304335) (owner: 10Sharvaniharan) [13:13:59] milimetric: o/ could you comment here? [13:14:00] https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/772934/2/jsonschema/analytics/mobile_apps/android_edit_history_interaction/current.yaml [13:14:15] do we prefer calling things like 'en' as in en.wikipedia 'project'? [13:14:20] and do we have that documented anywhere? [13:30:09] we follow some loose standards, but they're not well documented. There's a task to make all these decisions standard but we haven't prioritized it. Specifically for 'project', we usually call the whole thing that, as in 'en.wikipedia', but I think it's called something else in canonical_data and split up into 'en' and 'wikipedia'. I'll find the task [13:34:19] https://phabricator.wikimedia.org/T241741 [13:47:40] (03CR) 10Ottomata: [C: 03+2] Metrics Platform event schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [13:48:46] (03CR) 10Ottomata: New schema for edit history screen interactions (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772934 (https://phabricator.wikimedia.org/T304336) (owner: 10Sharvaniharan) [14:33:32] joal: what do you think we should do about that Special page bug in pageviews? (it allows pageviews to get through if they don't have "special" in X-Analytics, even though it shouldn't) [14:34:16] hm - I thionk I don't exactly get it milimetric :) [14:36:47] milimetric: just read the task and code - indeed we use x-analytics as a proxy for "special:" - possibly we wish to also use pagetitle? [14:40:05] (03CR) 10Milimetric: New schema for edit history screen interactions (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772934 (https://phabricator.wikimedia.org/T304336) (owner: 10Sharvaniharan) [14:49:45] joal: oh! I forgot about the project_namespace_map, so we could pass the localized name for "Special" into the pageview definition and filter out any titles that start with that if X-Analytics is not set. [14:50:32] hi allll! heya joal/milimetric I'm experiencing some issues with maxmind, and the code that is failing is the same that ran the job successfully last month... Have there been any changes to maxmind these days? [14:51:27] I think we decided that we've been using the new version of maxmind so the migration was a "no-op", but what are the issues? [15:16:15] 10Data-Engineering, 10Data-Catalog, 10SRE, 10serviceops, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) For the Ingress part we will need to use two different names/discovery records for the services (as we can't distinguish by port). Maybe `datahub.disc... [15:16:29] 10Data-Engineering, 10Data-Catalog, 10SRE, 10serviceops, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) a:03JMeybohm [15:21:26] 10Data-Engineering, 10Data-Catalog, 10SRE, 10serviceops, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) >>! In T303049#7800172, @JMeybohm wrote: > For the Ingress part we will need to use two different names/discovery records for the services (as we can't... [15:27:53] 10Data-Engineering, 10Data-Catalog, 10SRE, 10serviceops, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) >>! In T303049#7800187, @BTullis wrote: > It's not going to affect the public-facing (but authenticated) URL of https://datahub.wikimedia.org for the... [15:37:24] milimetric: the same code that ran in prod last month is giving the following error when trying to apply the UDF (jar import and create temporary func seem to work fine): [15:37:38] Exception in thread "main" java.lang.NoSuchMethodError: com.maxmind.geoip2.DatabaseReader.(Lcom/maxmind/geoip2/DatabaseReader$Builder;Lcom/maxmind/geoip2/DatabaseReader$1;)V [16:04:33] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Troubleshoot MySQL connection issues - https://phabricator.wikimedia.org/T298893 (10Milimetric) a:03mforns [16:09:01] 10Data-Engineering, 10Data-Engineering-Kanban, 10Superset, 10Epic: Presto/Superset User Experience Improvement - https://phabricator.wikimedia.org/T294259 (10razzi) [16:09:03] 10Data-Engineering, 10Data-Engineering-Kanban, 10Superset: Superset SQL Lab fails to stop query - https://phabricator.wikimedia.org/T293083 (10razzi) 05Open→03Resolved Ok thanks for confirming @KCVelaga_WMF! @CMacholan I'm going to close this; if you run into it again feel free to reopen. [16:12:36] mforns: I see, looks like some problem with deployment or maybe the archiva artifacts? Try to add the geolocate udf manually in a hive session and see if it works (using the same jar version from the job) [16:12:54] it doesn;t it gives the same error [16:16:05] btullis: razzi elukey , okay if i skip ops sync today? i have lots of back to back meetings and would like to make lunch! [16:16:47] ottomata: I am very against it! [16:16:51] :P [16:17:02] np please enjoy some time for your lunch :) [16:17:20] :) [16:17:26] razzi,btullis if you don't have specific things to discuss we can skip the meeting [16:18:16] 10Data-Engineering, 10Data-Engineering-Kanban: Add alert for varnishkafka low/zero messages per second to alertmanager - https://phabricator.wikimedia.org/T300246 (10BTullis) [16:20:38] I don't have anything in particular [16:20:43] elukey: ottomata I created https://phabricator.wikimedia.org/T304478 to move wikireplicas haproxy config to etcd, could use some input on that; also https://phabricator.wikimedia.org/T299481 is on my radar to upgrade dbstore hosts to bullseye [16:26:20] sure we can discuss them! The first one is definitely difficult but you outlined some good steps, the latter is easier and we can brainbounce on what to do [16:32:03] I am in the meeting :) [16:32:40] hmm I'm getting this error "your meeting code has expired" [16:34:16] the tardis is working for me elukey https://meet.google.com/kti-iybt-ekv [16:40:21] 10Data-Engineering, 10Data-Engineering-Kanban: Hosting of GDI use case specific source-code - https://phabricator.wikimedia.org/T304539 (10ntsako) [16:40:34] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Hosting of GDI use case specific source-code - https://phabricator.wikimedia.org/T304539 (10ntsako) [17:29:08] 10Data-Engineering-Radar, 10Product-Analytics: Support on understanding traffic and behaviors for users on legacy browsers (somewhat timely) - https://phabricator.wikimedia.org/T303301 (10Mayakp.wiki) a:03STHart Consultation Hours with @STHart : - Provided context on anomalous IE traffic from certain co... [17:42:47] (03PS4) 10Eigyan: analytics/legacy/quicksurveyinitiation: Add editCountBucket property [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/768014 (owner: 10Phuedx) [18:10:21] 10Data-Engineering, 10SRE: Adding snwachukwu@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T304541 (10Ottomata) [18:11:15] (03CR) 10Ottomata: "You need to bump the schema version in current.yaml" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/767439 (https://phabricator.wikimedia.org/T301391) (owner: 10Jdrewniak) [18:11:55] (03CR) 10Ottomata: "https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To#Evolving" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/767439 (https://phabricator.wikimedia.org/T301391) (owner: 10Jdrewniak) [18:19:28] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10wmfdata-python: conda-create-stacked breaks wmfdata.presto - https://phabricator.wikimedia.org/T301734 (10nshahquinn-wmf) p:05Triage→03Low We have good workarounds, but there is a bug somewhere in our setup. [18:43:10] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, and 2 others: wmfdata.mariadb relies on analytics-mysql being available - https://phabricator.wikimedia.org/T292479 (10nshahquinn-wmf) p:05Medium→03Low Unclear whether or not we want this logic to live in Wmfdata-Py... [18:47:21] 10Data-Engineering, 10wmfdata-python, 10Product-Analytics (Kanban): `spark.memory.driver` option does not get applied with "client" deployment mode. - https://phabricator.wikimedia.org/T284630 (10nshahquinn-wmf) a:05nshahquinn-wmf→03Milimetric I'm about to go on sabbatical, so Dan will look into this whi... [18:47:24] 10Data-Engineering, 10wmfdata-python, 10Product-Analytics (Kanban): `spark.memory.driver` option does not get applied with "client" deployment mode. - https://phabricator.wikimedia.org/T284630 (10nshahquinn-wmf) [18:49:31] (03PS4) 10Jdrewniak: Updating desktopwebuiactionstracking with viewport buckets [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/767439 (https://phabricator.wikimedia.org/T301391) [18:53:47] (03CR) 10Jdrewniak: [C: 04-1] Updating desktopwebuiactionstracking with viewport buckets [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/767439 (https://phabricator.wikimedia.org/T301391) (owner: 10Jdrewniak) [19:03:09] (03PS5) 10Jdrewniak: Updating desktopwebuiactionstracking with viewport buckets [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/767439 (https://phabricator.wikimedia.org/T301391) [19:04:38] (03CR) 10Jdrewniak: Updating desktopwebuiactionstracking with viewport buckets (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/767439 (https://phabricator.wikimedia.org/T301391) (owner: 10Jdrewniak) [19:11:30] hey mforns - do you still have your issue? [19:11:43] heya joal yes [19:11:51] wanna batcave? [19:11:57] ok :] [19:24:25] (03CR) 10Ottomata: [C: 03+1] Updating desktopwebuiactionstracking with viewport buckets [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/767439 (https://phabricator.wikimedia.org/T301391) (owner: 10Jdrewniak) [19:40:10] milimetric: wanna chat about pageviews quickly? [19:40:29] omw cave joal [20:50:36] (03CR) 10Bearloga: New schema for edit history screen interactions (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772934 (https://phabricator.wikimedia.org/T304336) (owner: 10Sharvaniharan) [20:50:40] 10Data-Engineering, 10Patch-For-Review: Pageview definition relies on X-Analytics to determine special pages - https://phabricator.wikimedia.org/T304362 (10Milimetric) Investigation reveals that special is not set in the x analytics header when the user is logged out. Proof: ` presto:wmf> select count(1)... [21:22:55] (03CR) 10Sharvaniharan: New schema for edit history screen interactions (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772934 (https://phabricator.wikimedia.org/T304336) (owner: 10Sharvaniharan) [21:29:39] (03CR) 10Bearloga: New schema for edit history screen interactions (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772934 (https://phabricator.wikimedia.org/T304336) (owner: 10Sharvaniharan) [21:32:08] (03CR) 10Sharvaniharan: New schema for edit history screen interactions (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772934 (https://phabricator.wikimedia.org/T304336) (owner: 10Sharvaniharan) [21:50:16] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [22:00:16] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [22:07:16] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [22:12:16] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [22:13:19] 10Data-Engineering, 10wmfdata-python, 10Product-Analytics (Kanban): Update Wmfdata-Python documention to describe code stewardship - https://phabricator.wikimedia.org/T304545 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:28] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Release Wmfdata-Python 2.0 - https://phabricator.wikimedia.org/T300442 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:30] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Move Wmfdata-Python from Github to Gitlab - https://phabricator.wikimedia.org/T304544 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:32] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Set up Wmfdata-Python test suite to run automatically - https://phabricator.wikimedia.org/T304547 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:38] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Update run functions to accept a filepath as well as a string for the SQL command - https://phabricator.wikimedia.org/T273197 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:41] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python, 10Documentation: Create end-user documentation for Wmfdata-Python - https://phabricator.wikimedia.org/T298178 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:43] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Rerunning Spark functions with changed settings has no effect - https://phabricator.wikimedia.org/T273210 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:45] 10Data-Engineering, 10wmfdata-python, 10Product-Analytics (Kanban): Support importing a Parquet file into HDFS using wmfdata-python - https://phabricator.wikimedia.org/T273196 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:47] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Create a script that installs Wmfdata-Python in development mode - https://phabricator.wikimedia.org/T294668 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:49] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Remove Spark session timeout functionality from Wmfdata-Python - https://phabricator.wikimedia.org/T298179 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:51] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Support querying a range of hourly data partitions - https://phabricator.wikimedia.org/T294654 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:53] 10Data-Engineering, 10wmfdata-python, 10Product-Analytics (Kanban): Remove "master" terminology from wmfdata-python - https://phabricator.wikimedia.org/T272220 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:55] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: wmfdata.spark module should provide easy access to pyspark - https://phabricator.wikimedia.org/T293722 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:57] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Add sql_tuple function to wmfdata-python - https://phabricator.wikimedia.org/T293706 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:13:59] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Update all run functions to allow specifying date and index columns - https://phabricator.wikimedia.org/T273208 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:14:01] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Add utility functions for converting between various date representations - https://phabricator.wikimedia.org/T273209 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:14:03] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Code toggle should remove standard error and center all output - https://phabricator.wikimedia.org/T247442 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:14:05] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: wmfdata should display more progress information and metadata when running a query - https://phabricator.wikimedia.org/T259808 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:14:07] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Allow query results to be cached in the filesystem or HDFS - https://phabricator.wikimedia.org/T248739 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:14:09] 10Analytics-Radar, 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Consider rewriting wmfdata-python to use omniduct - https://phabricator.wikimedia.org/T275038 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:14:11] 10Analytics-Radar, 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: wmfdata cannot recover from a crashed Spark session - https://phabricator.wikimedia.org/T245713 (10nshahquinn-wmf) Adding #data-engineering to all #wmfdata-python tasks, as requested by Dan and Andrew. [22:15:46] 10Data-Engineering, 10Patch-For-Review: Pageview definition relies on X-Analytics to determine special pages - https://phabricator.wikimedia.org/T304362 (10awight) (Special:Watchlist returns an http status 302 for logged-out users which explains that specific query result. Can you try the same thing but with... [22:20:21] 10Data-Engineering, 10Phabricator, 10Product-Analytics, 10wmfdata-python: Herald rule to add Product Analytics and Data Engineering tags to Wmfdata-Python tasks - https://phabricator.wikimedia.org/T304572 (10nshahquinn-wmf) [22:23:32] 10Data-Engineering, 10Patch-For-Review: Pageview definition relies on X-Analytics to determine special pages - https://phabricator.wikimedia.org/T304362 (10Milimetric) >>! In T304362#7801952, @awight wrote: > (Special:Watchlist returns an http status 302 for logged-out users which explains that specific query... [22:23:54] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Pageview definition relies on X-Analytics to determine special pages - https://phabricator.wikimedia.org/T304362 (10Milimetric) [22:30:58] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): WikipediaPortal Event Platform Migration - https://phabricator.wikimedia.org/T282012 (10Jdlrobson) a:05Jdrewniak→03Edtadros [22:31:05] 10Data-Engineering, 10wmfdata-python, 10Product-Analytics (Kanban): `spark.memory.driver` option does not get applied with "client" deployment mode. - https://phabricator.wikimedia.org/T284630 (10nshahquinn-wmf) >>! In T284630#7801044, @nshahquinn-wmf wrote: > I'm about to go on sabbatical, so Dan will look... [22:31:16] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): WikipediaPortal Event Platform Migration - https://phabricator.wikimedia.org/T282012 (10Jdlrobson) a:05Edtadros→03Jdrewniak Is this ready for QA? If so could you add some QA steps. If not, please move to do... [22:33:19] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: `spark.memory.driver` option does not get applied with "client" deployment mode. - https://phabricator.wikimedia.org/T284630 (10nshahquinn-wmf) [22:34:15] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Pageview definition relies on X-Analytics to determine special pages - https://phabricator.wikimedia.org/T304362 (10awight) >>! In T304362#7801976, @Milimetric wrote: > yeah, I agree, special=unknown is still useful in at least pointing out t... [22:46:33] 10Data-Engineering, 10wmfdata-python, 10Product-Analytics (Kanban): Remove "master" terminology from wmfdata-python - https://phabricator.wikimedia.org/T272220 (10nshahquinn-wmf) [22:58:33] 10Data-Engineering, 10wmfdata-python: Remove "master" terminology from wmfdata-python - https://phabricator.wikimedia.org/T272220 (10nshahquinn-wmf) 05Open→03Stalled I've renamed the `master` branch to `main`. Unfortunately, Spark still hasn't changed their use of "master". Term is widely used in Spark do... [22:59:15] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Remove "master" terminology from wmfdata-python - https://phabricator.wikimedia.org/T272220 (10nshahquinn-wmf) p:05Medium→03Low [23:05:04] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Remove "master" terminology from wmfdata-python - https://phabricator.wikimedia.org/T272220 (10nshahquinn-wmf) [23:28:29] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python, 10GitLab (Project Migration): Move Wmfdata-Python from Github to Gitlab - https://phabricator.wikimedia.org/T304544 (10Aklapper) +GitLab (Project Migration) (please add appropriate tags so people can find tasks - thanks!)