[00:10:25] PROBLEM - Check unit status of hadoop-namenode-backup-fetchimage on an-master1002 is CRITICAL: CRITICAL: Status of the systemd unit hadoop-namenode-backup-fetchimage https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [00:39:21] RECOVERY - Check unit status of monitor_refine_eventlogging_legacy on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_legacy https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [01:17:09] RECOVERY - Check unit status of monitor_refine_event on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_event https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [01:30:49] PROBLEM - Check unit status of monitor_refine_event on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_event https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [08:38:02] hello folks [08:38:19] as FYI I am moving kafka main and jumbo in deployment prep to the fixed uid/gid for kafka [08:38:25] Morning elukey. [08:38:28] we have already done in prod a while ago [08:38:35] hello btullis :) [08:39:45] Great, thanks. Did I previously have the ticket to do jumbo, but didn't get around to it? [08:40:37] Ah, you said jumbo in deployment prep. Yes, I think that I still have jumbo in production to do. right? [08:41:55] Ah, you already did it in January: T296990 [08:41:55] T296990: Move kafka-jumbo to a fixed uid/gid - https://phabricator.wikimedia.org/T296990 [08:43:34] yeah the remaining thing for jumbo is to move to the new TLS certs [08:43:43] but we still have to move kafka logging first etc.. [08:43:45] so no hurry :) [08:51:12] done :) [13:12:35] hello team! [13:52:19] Hi mforns ) [13:52:34] hello! [14:03:19] hi all! [14:38:23] Hi! [14:48:36] btullis: o/ qq - do you get the notifications for new messages coming to analytics-announce@? [14:48:45] (just to know if the settings are working) [15:26:33] chicocvenancio: I've loaded the data and I'll paste a query that works here: [15:26:35] https://www.irccloud.com/pastebin/TDEy3Jmf/ [15:48:49] 10Data-Engineering, 10Airflow: Migrate 1+ reportupdater jobs - https://phabricator.wikimedia.org/T307540 (10mforns) a:03mforns [15:49:09] 10Data-Engineering, 10Airflow: Migrate 1+ Refine jobs - https://phabricator.wikimedia.org/T307505 (10mforns) a:03NOkafor-WMF [15:49:21] 10Data-Engineering, 10Airflow: SparkSubmitOperator should make it easier to use conda dist envs - https://phabricator.wikimedia.org/T307937 (10mforns) a:03Ottomata [15:51:04] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow, 10Patch-For-Review: Fix airflow interlanguage job - https://phabricator.wikimedia.org/T308766 (10mforns) a:03NOkafor-WMF [15:52:12] 10Data-Engineering-Kanban, 10Airflow, 10Documentation: [Airflow] Kick off documentation in wikitech - https://phabricator.wikimedia.org/T302400 (10EChetty) [15:52:58] 10Data-Engineering-Kanban, 10Airflow, 10Documentation: [Airflow] Kick off documentation in wikitech - https://phabricator.wikimedia.org/T302400 (10EChetty) p:05High→03Medium [15:56:52] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [POC] Use airflow-installed Spark3 for an Airflow job - https://phabricator.wikimedia.org/T308168 (10JAllemandou) a:05JAllemandou→03Antoine_Quhen [16:06:13] (03PS3) 10NOkafor: Updated two HQL jobs to match conventions - interlanguage_daily and browser_general [analytics/refinery] - 10https://gerrit.wikimedia.org/r/793507 (https://phabricator.wikimedia.org/T308766) [16:06:33] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Plan spark3 migration - possibly incrementally - https://phabricator.wikimedia.org/T306955 (10JAllemandou) decisions for Spark3: * We're gonna merge and release the refinery-source patch bumping Spark and Scala as is, changing refinery-source verison... [16:07:16] (03PS4) 10NOkafor: Updated two HQL jobs to match conventions - interlanguage_daily and browser_general [analytics/refinery] - 10https://gerrit.wikimedia.org/r/793507 (https://phabricator.wikimedia.org/T308766) [16:10:29] (03CR) 10Joal: [V: 03+2 C: 03+2] "LGTM - Thanks for the changes - Merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/793507 (https://phabricator.wikimedia.org/T308766) (owner: 10NOkafor) [16:38:52] 10Analytics, 10Product-Analytics, 10SDAW-MediaSearch, 10Structured-Data-Backlog (Current Work): No data from ptwikinews in event.mediawiki_mediasearch_interaction table - https://phabricator.wikimedia.org/T308815 (10CBogen) [16:39:10] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform, 10Patch-For-Review: [Shared Event Platform] Ability to use Event Platform streams in Flink without boilerplate - https://phabricator.wikimedia.org/T308356 (10Ottomata) Wow crazy. [16:57:01] 10Data-Engineering, 10Beta-Cluster-Infrastructure: deployment-kafka-jumbo-5 in deployment-prep without role - https://phabricator.wikimedia.org/T309006 (10Ottomata) 05Open→03Resolved a:03Ottomata Indeed! I suppose this was an oversight when I recreated these as buster nodes. But how did Kafka get appli... [17:03:05] (03CR) 10Ottomata: [C: 03+1] ":)" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/794023 (https://phabricator.wikimedia.org/T305575) (owner: 10Sharvaniharan) [17:26:59] (03CR) 10Mforns: [C: 03+1] "Left a very minor comment. Feel free to ignore it!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/797240 (https://phabricator.wikimedia.org/T309023) (owner: 10Snwachukwu) [17:33:00] elukey: yes thanks. I have been rejecting the spam for the party week or two. [17:33:26] s/party/last [17:36:01] ack super, I'll remove myself from the admins of the list :) [17:48:42] heya mforns - would you have a minute? [17:59:15] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform, 10Patch-For-Review: [Shared Event Platform] Ability to use Event Platform streams in Flink without boilerplate - https://phabricator.wikimedia.org/T308356 (10Ottomata) > AFAICT, the Kafka Table connector only works with one topic at a time Th... [18:05:11] joal: yees! [18:05:20] sorry for delay, was in a meeting [18:05:25] np mforns [18:05:26] bc? [18:05:41] Can I grab come of your time for an airflow job please? I found an issue [18:05:55] Joining mforns [18:05:57] ofc! [18:09:13] (03CR) 10Shay Nowick: [C: 03+1] New schema for android app breadcrumbs [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/794023 (https://phabricator.wikimedia.org/T305575) (owner: 10Sharvaniharan) [18:16:14] joal: what is the last data-point timestamp in the file, please? [18:16:38] mforns: 2022-5-16 -- 2022-5-22 [18:16:49] mforns: logical_date: 2022-5-16 [18:17:06] but that's a Monday no? [18:17:28] YES mforns - My bad - 2022-5-15 (we have [18:17:31] both) [18:17:40] ok ok, got it [18:18:13] I will then start the Airflow job at 2022-05-22. [18:20:48] mforns: file corrected [18:21:00] thank you joal :] [18:24:10] joal: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/61 [18:25:39] mforns: Will approve when CI is done :) [18:25:44] ofc [18:25:48] mforns: Can you log when you kill the oozie job please? [18:25:56] yes [18:27:32] !log killed mobile_apps-session_metrics-coord (Airflow job is taking over) [18:27:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:27:41] Thanks mforns [18:28:05] (03PS20) 10Joal: Update to spark-3 and scala-2.12 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/656897 [18:28:35] Hi ottomata - Would you mind confirming you're ok with me moving forward with https://phabricator.wikimedia.org/T306955#7950475 [18:37:06] joal: the fix is deployed, I cleared the history of the DAG, and it should run in a week. [18:37:19] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Plan spark3 migration - possibly incrementally - https://phabricator.wikimedia.org/T306955 (10Ottomata) Let's do it! [18:37:20] Awesome thank you mforns [18:37:29] thanks for the heads up! [18:37:56] Thanks ottomata - will proceed into the merges :) [18:53:07] (03PS21) 10Joal: Update to spark-3 and scala-2.12 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/656897 [18:57:22] (03CR) 10CI reject: [V: 04-1] Update to spark-3 and scala-2.12 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/656897 (owner: 10Joal) [18:59:06] ya joal lemme know if i can help [18:59:10] sorry was in meeting before [18:59:26] ottomata: no problem - I'm just gonna pull the trigger at some point :) [19:00:06] (03PS22) 10Joal: Update to spark-3 and scala-2.12 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/656897 [19:00:25] ottomata: when my patch passes CI, if you may review it quickly (for commit message at least) [19:04:19] actually ottomata - I'll merge and release tomorrow - you have time tonight if you wish to take a closer look [19:41:49] (03PS23) 10Joal: Update to spark-3 and scala-2.12 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/656897 [19:57:15] 10Data-Engineering: Airflow: pin dependency versions to prevent long installs - https://phabricator.wikimedia.org/T309046 (10Milimetric) [20:35:30] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10Patch-For-Review: Enable Cassandra encryption (inter-node & client) - https://phabricator.wikimedia.org/T307798 (10Eevans) >>! In T307798#7943894, @BTullis wrote: > I believe that https://gerrit.wikimedia.org/r/c/operations/puppet/+/791663 is no... [21:29:34] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform, 10Patch-For-Review: [Shared Event Platform] Ability to use Event Platform streams in Flink without boilerplate - https://phabricator.wikimedia.org/T308356 (10Ottomata) > implement our own version of KafkaDynamicTableFactory Everything is alw... [21:29:36] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform, 10Patch-For-Review: [Shared Event Platform] Ability to use Event Platform streams in Flink without boilerplate - https://phabricator.wikimedia.org/T308356 (10Ottomata) > implement our own version of KafkaDynamicTableFactory Everything is alw... [21:38:27] (03PS1) 10Gerrit maintenance bot: Add guw.wiktionary to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/797513 (https://phabricator.wikimedia.org/T309057)