[00:21:59] PROBLEM - Check unit status of monitor_refine_eventlogging_analytics on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_analytics https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [00:23:55] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: monitor_refine_eventlogging_analytics.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [04:10:57] PROBLEM - eventgate-analytics-external validation error rate too high on alert1001 is CRITICAL: 2.157 gt 2 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos [06:37:38] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/842403 (https://phabricator.wikimedia.org/T320898) (owner: 10Gerrit maintenance bot) [06:53:43] bonjour joal :) [06:53:53] Good morning elukey ! [07:02:45] joal: an-airflow1001 has ~2.5G in the root partition left to use [07:02:53] meh :( [07:03:16] we could in theory trim some logs, not sure if it is ok or not [07:03:28] it is very much ok elukey [07:04:29] ah I see /usr/local/bin/airflow-clean-log-dirs, it cleans up logs 30d+ stale [07:04:41] ok if I run the same for 15d+ ? [07:05:07] elukey: I think it is - I wonder why we have so much log :() [07:05:45] joal: not a lot of logs, some GBs, but it is the only thing that we can trim :( [07:06:00] right [07:06:03] hm [07:08:31] done, we have ~4G now, not much but should be ok for the moment [07:09:05] biggest dirs are [07:09:05] 6.5G /var [07:09:05] 11G /srv [07:09:05] 15G /usr [07:09:37] /srv could be moved to a separate vdisk/partition in theory [07:09:41] /usr is mostly due to the anaconda-wmf etc.. packages [07:09:45] and /var is logs [07:09:59] but the root partition is ~40G, so not huge [07:11:26] Thanks a milion for the action and summary elukey - I'll ask btullis what plan we'd rather follow for this [07:11:41] Moving /srv feels like a good idea [07:14:08] <3 [07:43:43] (03CR) 10Esanders: [C: 03+2] Include client_ip in EditAttemptStep schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/842452 (https://phabricator.wikimedia.org/T314178) (owner: 10DLynch) [07:44:25] (03Merged) 10jenkins-bot: Include client_ip in EditAttemptStep schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/842452 (https://phabricator.wikimedia.org/T314178) (owner: 10DLynch) [08:22:07] 10Data-Engineering, 10Machine-Learning-Team, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10fgiunchedi) (FYI) For this task and the work in {T314981} we now have Debian packages for Benthos available for Buster and Bullseye [08:26:13] 10Data-Engineering: wmf.webrequest: 'presto error: Corrupted statistics for column "[user_agent] optional binary " in Parquet file ...' - https://phabricator.wikimedia.org/T320926 (10Michael) [08:58:18] 10Data-Engineering-Kanban, 10Data Engineering Planning, 10SRE, 10serviceops, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10Clement_Goubert) In preparation of the redeploy, I lowered the TTL for service discovery to 30 seconds instead of 5 minutes s... [09:05:10] 10Data-Engineering: wmf.webrequest: 'presto error: Corrupted statistics for column "[user_agent] optional binary " in Parquet file ...' - https://phabricator.wikimedia.org/T320926 (10JAllemandou) Reading from here: https://github.com/prestodb/presto/issues/12338 It seems we can disable the failure on parquet sta... [10:07:26] (03PS2) 10Joal: Update mediawiki-history page computation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/842922 (https://phabricator.wikimedia.org/T318589) [11:07:49] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python, 10Data Pipelines (Sprint 03): Upgrade WMFData Python Package to use Spark3 - https://phabricator.wikimedia.org/T318587 (10EChetty) [11:26:35] 10Data-Engineering, 10Cassandra, 10Image-Suggestions: Section Level Image Suggestions - Data Persistence Request - https://phabricator.wikimedia.org/T320831 (10LSobanski) [11:41:54] 10Data-Engineering, 10Privacy Engineering, 10SRE-swift-storage: Swift for differential privacy data publication - https://phabricator.wikimedia.org/T307245 (10LSobanski) [12:27:44] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03), 10Patch-For-Review: Design Schema for page state and page state with content (enriched) streams - https://phabricator.wikimedia.org/T308017 (10lbowmaker) [12:27:59] 10Data-Engineering-Kanban, 10Data Engineering Planning, 10SRE, 10serviceops, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10lbowmaker) [12:28:25] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03), 10Spike: [SPIKE] Build simple stateless service using Flink SQL - https://phabricator.wikimedia.org/T318856 (10lbowmaker) [12:47:15] PROBLEM - eventgate-analytics-external validation error rate too high on alert1001 is CRITICAL: 2.185 gt 2 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos [13:16:08] 10Data-Engineering, 10Cassandra, 10Image-Suggestions, 10Section-Level-Image-Suggestions: Section Level Image Suggestions - Data Persistence Request - https://phabricator.wikimedia.org/T320831 (10CBogen) [13:34:53] RECOVERY - eventgate-analytics-external validation error rate too high on alert1001 is OK: (C)2 gt (W)1 gt 0.9601 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos [13:36:09] the errors seemed for the mediawiki.editattempt_block stream: ".country_code' should be string"" [13:37:24] there is aleady ticket :) [13:37:39] https://phabricator.wikimedia.org/T320938 [13:39:26] ah nice! [14:21:24] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGMT! thanks :]" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832524 (https://phabricator.wikimedia.org/T317525) (owner: 10Phuedx) [14:22:53] (03PS9) 10Mforns: Fix end-of-month/year allowed_interval issue [analytics/refinery] - 10https://gerrit.wikimedia.org/r/836295 (https://phabricator.wikimedia.org/T316746) [14:23:44] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Merging after discussions and reviews!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/836295 (https://phabricator.wikimedia.org/T316746) (owner: 10Mforns) [14:24:20] (03CR) 10Mforns: [V: 03+2 C: 03+2] Fix end-of-month/year allowed_interval issue (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/836295 (https://phabricator.wikimedia.org/T316746) (owner: 10Mforns) [14:26:08] (03Abandoned) 10Mforns: Added meta.wikidata to the pageview allow-list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/817323 (https://phabricator.wikimedia.org/T313834) (owner: 10NOkafor) [14:26:51] 10Data-Engineering: wmf.webrequest: 'presto error: Corrupted statistics for column "[user_agent] optional binary " in Parquet file ...' - https://phabricator.wikimedia.org/T320926 (10mpopov) @Michael: Until we get a fix, `AND length(user_agent) < 2` as a workaround might be OK? [14:27:13] (03CR) 10Mforns: "@Sandra, hi! Is this change still valid?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/793523 (owner: 10Snwachukwu) [14:28:00] (03CR) 10Mforns: [C: 03+1] "@Joal, can we now merge this?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/681682 (https://phabricator.wikimedia.org/T280649) (owner: 10Joal) [14:28:59] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03), 10Spike: [SPIKE] Build simple stateless service using PyFlink - https://phabricator.wikimedia.org/T318859 (10lbowmaker) [14:31:38] 10Data-Engineering, 10Event-Platform Value Stream: Prototype Flink job for content Dumps - https://phabricator.wikimedia.org/T320966 (10Milimetric) [14:32:42] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03): Prototype Flink job for content Dumps - https://phabricator.wikimedia.org/T320966 (10lbowmaker) [14:34:20] 10Data-Engineering, 10Event-Platform Value Stream, 10Spike: [SPIKE] Investigate using Flink Stateful Functions - https://phabricator.wikimedia.org/T318861 (10Ottomata) 05Open→03Declined Going to decline this based on a conversation I had with some Flink maintainers. Stateful Functions is not well mainta... [14:37:54] 10Data-Engineering, 10Machine-Learning-Team, 10observability, 10Event-Platform Value Stream (Sprint 03): Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10Ottomata) [14:38:12] 10Data-Engineering, 10Machine-Learning-Team, 10observability, 10Event-Platform Value Stream (Sprint 03): Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10lbowmaker) [14:46:46] 10Data-Engineering-Kanban, 10Data Engineering Planning, 10SRE, 10serviceops, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10Clement_Goubert) 05Open→03Resolved All eventgate services redeployed, including staging environments. TTL back at its nor... [14:54:54] found https://github.com/apache/incubator-streampark that may be interesting to explore [14:56:59] interesting --^ ! ottomata, gmodena --^ [14:59:00] also the name is so nice [15:01:14] (03Abandoned) 10Snwachukwu: Fix api hql file and Projectview hql file. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/793523 (owner: 10Snwachukwu) [15:01:32] (03CR) 10Snwachukwu: Fix api hql file and Projectview hql file. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/793523 (owner: 10Snwachukwu) [15:05:07] 10Data-Engineering, 10API Platform (Product Roadmap), 10Platform Engineering Roadmap, 10User-Eevans: Obtain a security review of AQS 2.0 - https://phabricator.wikimedia.org/T288663 (10VirginiaPoundstone) [15:08:17] 10Data-Engineering, 10API Platform (Product Roadmap), 10Platform Engineering Roadmap, 10User-Eevans: Obtain security review of uniqueDevices - https://phabricator.wikimedia.org/T320976 (10VirginiaPoundstone) [15:08:41] 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: Obtain security review of uniqueDevices - https://phabricator.wikimedia.org/T320976 (10VirginiaPoundstone) [15:09:06] 10Data-Engineering-Kanban, 10Data Engineering Planning, 10SRE, 10serviceops, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10Ottomata) Yeehaw thank you so much Clem! [15:09:12] 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: Obtain security review of uniqueDevices - https://phabricator.wikimedia.org/T320976 (10VirginiaPoundstone) a:03Atieno [15:14:03] 10Analytics-Kanban, 10Data-Engineering, 10Event-Platform Value Stream, 10Fundraising-Backlog, and 3 others: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned - https://phabricator.wikimedia.org/T282131 (10phuedx) [15:17:19] PROBLEM - eventgate-analytics-external validation error rate too high on alert1001 is CRITICAL: 2.7 gt 2 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos [15:33:20] (03CR) 10Joal: "We should actually do more than this and remove all cassandra-oozie code as it is now loaded using airflow. I'll update the patch." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/681682 (https://phabricator.wikimedia.org/T280649) (owner: 10Joal) [15:45:37] 10Data-Engineering, 10Data Pipelines: Reduce the number of files generated by geoeditors airflor jobs - https://phabricator.wikimedia.org/T304852 (10EChetty) [15:47:24] 10Data-Engineering, 10Data Pipelines: Reduce the number of files generated by geoeditors airflor jobs - https://phabricator.wikimedia.org/T304852 (10JAllemandou) [15:51:29] 10Data-Engineering, 10Equity-Landscape: Editorship Input Metrics - https://phabricator.wikimedia.org/T309274 (10ntsako) a:05ntsako→03JAnstee_WMF [16:01:57] 10Data-Engineering, 10Data Pipelines (Sprint 03): Reduce the number of files generated by geoeditors airflor jobs - https://phabricator.wikimedia.org/T304852 (10EChetty) [16:08:10] 10Data-Engineering, 10Equity-Landscape: Grants Metrics Transformation - https://phabricator.wikimedia.org/T306620 (10KCVelaga_WMF) 05Resolved→03Open reopening for data QA [16:08:12] 10Data-Engineering, 10Equity-Landscape: Milestone: Ingest and Transform Input Data - https://phabricator.wikimedia.org/T305475 (10KCVelaga_WMF) [16:08:33] 10Data-Engineering, 10Equity-Landscape: Grants Leadership Output Metrics transformation - https://phabricator.wikimedia.org/T306620 (10KCVelaga_WMF) [16:21:11] 10Data-Engineering, 10Equity-Landscape: split country data into regional classification and main country cata - https://phabricator.wikimedia.org/T320985 (10ntsako) [16:21:28] 10Data-Engineering, 10Equity-Landscape: split country data into regional classification and main country data - https://phabricator.wikimedia.org/T320985 (10ntsako) [16:22:23] 10Data-Engineering, 10Equity-Landscape: Split country data into regional classification and main country data - https://phabricator.wikimedia.org/T320985 (10ntsako) [16:43:43] 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10xcollazo) 05Open→03Resolved We discussed this and figured that @BTullis, and a new SRE that is joining us soon, shoul... [16:51:49] 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10Dzahn) 05Open→03Resolved [16:51:54] 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10Dzahn) 05Resolved→03Open Keeping the existing setup was the one possible outcome I had tried to prevent here :( [16:52:48] 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10Dzahn) @BTullis Is there any way we could get this out of the exim aliases? ...pleaaasse.... [17:04:13] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03): Refactor EventBus extension Hooks to use new hook system - https://phabricator.wikimedia.org/T320655 (10Ottomata) @lbowmaker I just learned that we don't need to do this for this sprint in order to do T311129. I can use the new HookHandler system... [17:08:00] 10Data-Engineering, 10Event-Platform Value Stream: Refactor EventBus extension Hooks to use new hook system - https://phabricator.wikimedia.org/T320655 (10lbowmaker) [17:09:58] 10Data-Engineering, 10Equity-Landscape: Editorship Output Rank Metrics - https://phabricator.wikimedia.org/T306618 (10JAnstee_WMF) @ntsako I have checked that the regional aggregates are aligned sufficiently to sign-off on the QA for the editorship outputs. I have begun mapping all our data labels here: https... [17:18:24] 10Data-Engineering, 10Equity-Landscape: Editorship Input Metrics - https://phabricator.wikimedia.org/T309274 (10JAnstee_WMF) @ntsako I can sign off on these metrics also - however some table column labels suggestions for the input data where I suggest maybe changing for clarity: from commons_column_ja to co... [17:21:17] 10Data-Engineering, 10Equity-Landscape: Grants input metric - https://phabricator.wikimedia.org/T309276 (10ntsako) Table renamed to ` SELECT * FROM ntsako.grants_leadership_input_metrics WHERE year=2021; ` Query that populates the table (run in Hue not Superset): -- NOTE Percent ranks aren't being calc... [17:21:51] 10Data-Engineering, 10Equity-Landscape: Grants input metric - https://phabricator.wikimedia.org/T309276 (10ntsako) a:05ntsako→03JAnstee_WMF [17:34:39] PROBLEM - eventgate-analytics-external validation error rate too high on alert1001 is CRITICAL: 2.041 gt 2 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos [17:36:22] 10Data-Engineering, 10Equity-Landscape: Readership Output Rank Metrics - https://phabricator.wikimedia.org/T306617 (10JAnstee_WMF) @ntsako @KCVelaga_WMF Signing off on thie data QA here - But it seems we should be consistent in labeling our standardized metric inputs - in the reader pipeline this may need to... [17:37:28] 10Data-Engineering, 10Equity-Landscape: Readership input metrics - https://phabricator.wikimedia.org/T309273 (10JAnstee_WMF) @ntsako Signing off on the data QA for GDI [17:40:12] (VarnishkafkaNoMessages) firing: (2) varnishkafka on cp2033 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:45:12] (VarnishkafkaNoMessages) resolved: (2) varnishkafka on cp2033 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:47:54] joal one question: I remember hearing that we already did make all our Spark2 jobs Spark3-ready syntax-wise. Is that correct? [18:01:11] mforns: yes! refinery-job is spark3 compatible, except for potential runtime-errors of not yet tested jobs [18:01:25] thanks joal! [18:08:45] PROBLEM - eventgate-analytics-external validation error rate too high on alert1001 is CRITICAL: 2.115 gt 2 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos [18:13:53] 10Data-Engineering, 10Equity-Landscape: Editorship Input Metrics - https://phabricator.wikimedia.org/T309274 (10JAnstee_WMF) a:05JAnstee_WMF→03ntsako [18:14:12] 10Data-Engineering, 10Equity-Landscape: Readership input metrics - https://phabricator.wikimedia.org/T309273 (10JAnstee_WMF) a:05JAnstee_WMF→03ntsako [18:14:25] 10Data-Engineering, 10Equity-Landscape: Readership Output Rank Metrics - https://phabricator.wikimedia.org/T306617 (10JAnstee_WMF) a:05JAnstee_WMF→03ntsako [18:14:40] 10Data-Engineering, 10Equity-Landscape: Editorship Output Rank Metrics - https://phabricator.wikimedia.org/T306618 (10JAnstee_WMF) a:05JAnstee_WMF→03ntsako [18:17:25] 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10BTullis) Ok @dzahn - I'm sorry, I didn't realise that moving this out of the Exim aliases was important to you. I though... [18:17:36] !log deleted Airflow DAGs for backfilling of Cassandra loading of unique devices [18:17:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:21:39] 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10Ladsgroup) [[https://phabricator.wikimedia.org/T315486#8172401|Again]], you can have it in mailman as well, relend alerts... [18:31:33] PROBLEM - eventgate-analytics-external validation error rate too high on alert1001 is CRITICAL: 2.181 gt 2 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos [19:28:23] RECOVERY - eventgate-analytics-external validation error rate too high on alert1001 is OK: (C)2 gt (W)1 gt 0.7848 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos [21:20:17] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: wmfdata.spark module should provide easy access to pyspark - https://phabricator.wikimedia.org/T293722 (10xcollazo) > Instead, it should expose it as wmfdata.spark.pyspark Could you elaborate on use cases for this? [21:44:47] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: wmfdata.spark module should provide easy access to pyspark - https://phabricator.wikimedia.org/T293722 (10nshahquinn-wmf) >>! In T293722#8323195, @xcollazo wrote: >> Instead, it should expose it as wmfdata.spark.pyspark > > Could you elaborate on us... [21:45:24] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Remove Spark session timeout functionality from Wmfdata-Python - https://phabricator.wikimedia.org/T298179 (10xcollazo) > If the kernel gets shut down, the driver and application master go with it, and this neatly frees up the resources used by the k... [21:46:36] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Rerunning Spark functions with changed settings has no effect - https://phabricator.wikimedia.org/T273210 (10xcollazo) Well defined semantics on what to expect from a method, and a method name that matches reality all make sense to me. +1. [21:58:34] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: wmfdata.spark module should provide easy access to pyspark - https://phabricator.wikimedia.org/T293722 (10xcollazo) Ah I see. Thanks for the example. A note: `spark.get_custom_session()` is the method that currently calls `findspark`. So a rearrange... [22:41:55] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: wmfdata.spark module should provide easy access to pyspark - https://phabricator.wikimedia.org/T293722 (10nshahquinn-wmf) >>! In T293722#8323328, @xcollazo wrote: > Having said that, `pyspark` will be available in the new conda-analytics environment.... [23:00:32] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Remove Spark session timeout functionality from Wmfdata-Python - https://phabricator.wikimedia.org/T298179 (10nshahquinn-wmf) >>! In T298179#8323285, @xcollazo wrote: >> If the kernel gets shut down, the driver and application master go with it, and...