[04:21:39] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [04:32:31] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: monitor_refine_event_sanitized_analytics_immediate.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [05:11:53] 10Analytics-Kanban, 10Data-Engineering-Planning, 10Pageviews-Anomaly: Article on Carles Puigdemont has inflated pageviews in many projects - https://phabricator.wikimedia.org/T263908 (10Anhn) This is my first WM-posting ever - pls tell me if something is wrong or in the wrong place. Regarding the inflated pa... [11:54:42] aqu: Hello :) [11:54:56] Do you have time now? I know it's lunch time :) [11:57:45] (03PS1) 10Phuedx: Remove EditorActivation allowlist entry [analytics/refinery] - 10https://gerrit.wikimedia.org/r/896326 (https://phabricator.wikimedia.org/T330766) [12:00:31] nfraison: Hello - would you have a minute for me? [12:06:53] (03PS3) 10Nmaphophe: GDI Equity Landscape Tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/895737 [12:22:58] joal yes I'm available [12:24:53] \o/ [12:24:56] batcave nfraison ? [12:26:14] please remind me the meet url for the batcave [12:27:17] https://meet.google.com/rxb-bjxn-nip [12:50:07] (03PS1) 10Jennifer Ebe: T330206 - Create Mediacounts Load Hourly HQL [analytics/refinery] - 10https://gerrit.wikimedia.org/r/896334 [13:45:34] elukey: fyi: I'm troubleshooting an oom of our flink app. You might see a couple of restart notifications pop up [13:59:12] gmodena: no problem! [15:10:49] hello folks, I was testing new streams for mediawiki.revision-score and I got this from eventgate main [15:10:52] of schema at /mediawiki/revision/score/2.0.0 destined to stream mediawiki.revision-score-goodfaith is not allowed in stream; mediawiki.revision-score-goodfaith is configured but does not have any settings [15:12:38] ah snap underscores [15:12:44] * elukey cries in a corner [15:26:53] * elukey blames ottomata for the underscores [15:26:54] :D [15:30:36] (03PS1) 10Barakat Ajadi: Navtiming: Add total longtask and total longtask duration [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/896370 (https://phabricator.wikimedia.org/T264032) [16:06:59] could i get someone to run a chown on hdfs for us? re: T331580 [16:07:00] T331580: Fix permissions in hdfs://analytics-hadoop/wmf/data/discovery - https://phabricator.wikimedia.org/T331580 [16:16:46] 10Data-Engineering, 10CirrusSearch, 10Discovery-Search (Current work): Fix permissions in hdfs://analytics-hadoop/wmf/data/discovery - https://phabricator.wikimedia.org/T331580 (10nfraison) [16:20:37] 10Data-Engineering, 10CirrusSearch, 10Discovery-Search (Current work): Fix permissions in hdfs://analytics-hadoop/wmf/data/discovery - https://phabricator.wikimedia.org/T331580 (10nfraison) Command to change right is runnning To prevent that we can remove write for the analytics-search-users group: ` From:... [16:21:31] 10Data-Engineering, 10CirrusSearch, 10Discovery-Search (Current work): Fix permissions in hdfs://analytics-hadoop/wmf/data/discovery - https://phabricator.wikimedia.org/T331580 (10nfraison) All rights update to analytics-search:analytics-search-users on hdfs://analytics-hadoop/wmf/data/discovery [16:22:52] ebernhardson: done [16:23:40] nfraison: thanks! [16:37:23] thanks nfraison for the perms fix - I had seen the ticket and didn't get to do it in time [17:58:53] 10Data-Engineering, 10Data Pipelines: Install conda-analytics on Airflow servers - https://phabricator.wikimedia.org/T331345 (10xcollazo) 05Open→03Resolved a:03xcollazo Confirmed that I can use spark3 from `an-airflow1004.eqiad.wmnet`: ` xcollazo@an-airflow1004:~$ sudo -u analytics-platform-eng bash ana... [19:24:02] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Migrate pageview dumps jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889532 (https://phabricator.wikimedia.org/T329646) (owner: 10Milimetric) [19:24:09] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Migrate pageview_hourly and related jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/887869 (https://phabricator.wikimedia.org/T324482) (owner: 10Milimetric) [19:25:46] I'm gonna deploy refinery and try to migrate the pageview jobs... bit risky but I'm around over the weekend so I'll handle any problems [20:04:05] !log deployed refinery with new pageview jobs, patched in a manual copy of static_data/pageview/whitelist/whitelist.tsv because that file was renamed in the most recent version and would have broken jobs otherwise [20:04:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:07:05] (03PS1) 10Milimetric: Move pageview allowlist to a proper folder [analytics/refinery] - 10https://gerrit.wikimedia.org/r/896425 [20:07:17] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Move pageview allowlist to a proper folder [analytics/refinery] - 10https://gerrit.wikimedia.org/r/896425 (owner: 10Milimetric) [20:16:55] (03PS1) 10Htriedman: add hql/differential_privacy/join_titles.hql for dp [analytics/refinery] - 10https://gerrit.wikimedia.org/r/896389 [20:22:15] (03CR) 10Milimetric: [V: 03+2 C: 03+2] add hql/differential_privacy/join_titles.hql for dp [analytics/refinery] - 10https://gerrit.wikimedia.org/r/896389 (owner: 10Htriedman)