[02:15:26] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10cloud-services-team (Kanban): Top edited pages list on enwiktionary contains nonexistent pages with titles made up of question marks - https://phabricator.wikimedia.org/T284623 (10Milimetric) p:05High→03Triage a:05Milimetric→03None [02:16:07] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10cloud-services-team (Kanban): Top edited pages list on enwiktionary contains nonexistent pages with titles made up of question marks - https://phabricator.wikimedia.org/T284623 (10Milimetric) Moving to paused as we are blocked until cloud services l... [02:19:18] (03PS1) 10Milimetric: Convert with utf8mb4 when sqooping varbinary cols [analytics/refinery] - 10https://gerrit.wikimedia.org/r/701220 (https://phabricator.wikimedia.org/T284623) [06:34:17] !log rename superset hiera role configs in puppet private repo (to match the role change done recently) + superset restart [06:34:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:38:34] !log drop hieradata/role/common/analytics_cluster/superset.yaml from puppet private repo (unused config, all the values dumplicated in the new hiera config) [06:38:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:39:36] superset works fine afaics, lemme know otherwise :) [06:48:04] Good morning - Thanks elukey [06:53:04] bonjour joal [07:19:28] https://towardsdatascience.com/lessons-on-ml-platforms-from-netflix-doordash-spotify-and-more-f455400115c7 [07:43:29] (03CR) 10Joal: [C: 03+1] "Thanks a lot Dan - I let you merge." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/701220 (https://phabricator.wikimedia.org/T284623) (owner: 10Milimetric) [11:07:15] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Replace Camus by Gobblin - https://phabricator.wikimedia.org/T271232 (10JAllemandou) Why not having a single job for all streams (stream-config declared streams first). I think it'll be a matter of scaling the workers accordingly :) [12:10:31] (03PS1) 10Kosta Harlan: link_suggestion_interaction: Add outdatedsuggestions_dialog interface [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/701376 (https://phabricator.wikimedia.org/T283109) [12:56:55] 10Analytics-Radar, 10EventStreams, 10MediaWiki-API, 10MW-1.37-notes (1.37.0-wmf.12; 2021-06-28), 10User-Urbanecm: Add user field to mediawiki/api/request - https://phabricator.wikimedia.org/T285113 (10Urbanecm) Okay, that's helpful. Thanks @ottomata! [13:44:49] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Replace Camus by Gobblin - https://phabricator.wikimedia.org/T271232 (10Ottomata) Ok, let's try that first. I guess we need a different job for the migrated legacy eventlogging streams, since they will go into a different output dir? [13:56:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Replace Camus by Gobblin - https://phabricator.wikimedia.org/T271232 (10JAllemandou) We need to have different jobs for different output folders yes. And also per configuration type (topics/eventstreamconfig). [13:58:50] 10Analytics, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), and 2 others: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10awight) @mforns Success, I see the data landing in Graphite! Thank you for this marathon of v... [14:02:14] Gone for kids - see you at standup :) [14:05:31] 10Analytics, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), and 2 others: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10mforns) @awight Awesome! I'm glad it works in the end. It's the first time I see the corrupted... [14:10:57] 10Analytics, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), and 2 others: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10awight) >>! In T273748#7175255, @mforns wrote: > @awight Awesome! I'm glad it works in the end... [14:43:39] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Top edited pages list on enwiktionary contains nonexistent pages with titles made up of question marks - https://phabricator.wikimedia.org/T284623 (10Milimetric) removing cloud services, debugging showed the problem to be fixable b... [14:48:36] 10Analytics, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), and 2 others: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10lilients_WMDE) a:03lilients_WMDE [15:42:35] 10Analytics, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure: [Metrics Platform] Define stream configuration syntax relevant to v1 release - https://phabricator.wikimedia.org/T273235 (10Ottomata) Hey all, we are moving forward with the `consumers` stream config setting to do {T2739... [15:47:12] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Analytics, 10Product-Data-Infrastructure: [MEP] Determine how stream configuration is authored and deployed - https://phabricator.wikimedia.org/T269774 (10Ottomata) wgEventStreams is getting unruly! [16:16:48] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Top edited pages list on enwiktionary contains nonexistent pages with titles made up of question marks - https://phabricator.wikimedia.org/T284623 (10JAllemandou) Does the patch above on `misctools/sql` mean that sqoop should work... [16:41:19] 10Analytics, 10observability: Need a list of AQS Kibana dashboards and searches - https://phabricator.wikimedia.org/T285318 (10odimitrijevic) p:05Triage→03High a:03JAllemandou [16:42:03] 10Analytics-Radar, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), and 2 others: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10odimitrijevic) [16:45:13] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Patch-For-Review: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10odimitrijevic) a:05Gilles→03odimitrijevic Olja to follow up on work related to event-logging e... [16:46:25] 10Analytics, 10Analytics-Kanban, 10Event-Platform: jsonschema-tools should allow skipping of repository tests for certain schemas. - https://phabricator.wikimedia.org/T285006 (10odimitrijevic) p:05Medium→03High [17:41:04] ottomata: shall we talk now or after diner/lunch? [17:41:48] can do now! [17:41:57] in bc [17:42:00] milimetric: about collation for sqoop I found that: https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-charsets.html [17:42:07] joining ottomata [17:45:50] thx joal! [17:54:57] milimetric: the connector we use is mysql-connector-java-5.1.49.jar [17:55:02] (03PS2) 10Milimetric: Convert with utf8mb4 when sqooping varbinary cols [analytics/refinery] - 10https://gerrit.wikimedia.org/r/701220 (https://phabricator.wikimedia.org/T284623) [17:55:20] yep, testing it now joal [17:59:26] PROBLEM - Hadoop NodeManager on an-worker1130 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [18:01:08] RECOVERY - Hadoop NodeManager on an-worker1130 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [18:01:34] Starting build #9 for job wikimedia-event-utilities-maven-release-docker [18:02:44] Project wikimedia-event-utilities-maven-release-docker build #9: 09SUCCESS in 1 min 11 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release-docker/9/ [18:05:26] joal: [18:05:26] https://archiva.wikimedia.org/#artifact/org.wikimedia/eventutilities/1.0.7 [18:05:48] also for you to revieww [18:05:49] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668124 [18:11:14] joal, for gobblin puppet; perhaps we can keep it super simple and just declare the jobs/systemd timers there? [18:11:27] could/should we keep gobblin job configs in refinery for now? [18:11:36] might make it easier to iterat [18:11:43] and also move to airflow eventually? [18:17:57] milimetric: wanna discuss custom xcomms and taskflow for 20 mins? [18:18:42] 10Analytics, 10Analytics-Kanban: Replace Camus by Gobblin - https://phabricator.wikimedia.org/T271232 (10Ottomata) @JAllemandou I've got your properties files on stat1008, can you give me an example of the command you run to launch? [18:31:57] ottomata: no problem for me keeping job config in refinery - Only concern I can see is auto-deploy, as the files will be changed by scap, but except for that no problem [18:38:03] 10Analytics, 10Analytics-Kanban: Replace Camus by Gobblin - https://phabricator.wikimedia.org/T271232 (10JAllemandou) Yes for sure, here are my test examples: ` # Webrequest HADOOP_USER_CLASSPATH_FIRST=true HADOOP_OPTS="-Dlog4j.configuration=file:///home/joal/etc_gobblin/log4j.properties -Dmapreduce.job.user.... [18:53:24] 10Analytics, 10observability: Need a list of AQS Kibana dashboards and searches - https://phabricator.wikimedia.org/T285318 (10JAllemandou) Hi @colewhite, we talked with the team and we can't think of a dashboard or special searches we would have saved in Kibana for AQS. Flagging @elukey to also confirm. About... [18:53:59] 10Analytics-Clusters: ROCm can't find clang on stat1005 - https://phabricator.wikimedia.org/T285495 (10EBernhardson) [18:55:39] 10Analytics-Clusters: ROCm can't find clang on stat1005 - https://phabricator.wikimedia.org/T285495 (10EBernhardson) [18:59:18] joal: at least you can run scap wtihout puppet +2 :) [18:59:33] yup :) [19:00:03] ottomata: seems that I can't push to gerrit :)( [19:00:26] oh to the new gobblin repo? [19:00:30] right lemme let you push [19:00:34] I tried a simple push of master after having added the remote and got: prohibited by Gerrit: update for creating new commit object not permitted [19:01:30] joal: try now [19:01:58] doing (will be long, no fiber at home) [19:06:09] sorry mforns, sick baby... I was thinking we were going to all talk about it, but I can talk tomorrow? [19:06:23] milimetric: of course! [19:06:39] ping me whenever :] [19:15:31] ottomata: I'm gonna make a patch in refinery for the gobblin config tomorrow - ok for you? [19:16:14] second thing ottomata - would you give me a minute to make sure I understand correctly the way to setup filters for consumers? [19:22:22] Arf ottomata - email address XX is not registered in your account, and you lack 'forge committer' permission. [19:47:28] ok - code ready to be pushed, then jar can be compiled and released [19:49:59] (03CR) 10Milimetric: [C: 03+1] "sqooped enwiktionary and a couple of other small wikis, confirmed no errors and that decoding works properly now. See:" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/701220 (https://phabricator.wikimedia.org/T284623) (owner: 10Milimetric) [19:50:11] \o/ [19:50:13] --^ [19:50:14] :) [19:50:18] thanks for testing milimetric [19:51:07] yeah, sorry so slow. I'm always running into the gauntlet of permissions errors as I slowly realize stat1004 is impossible and move to an-launcher. I am going to tattoo an-launcher1002 on my forehead, hope we don't change the machine name [19:51:27] :) [19:55:32] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Apache Beam go prototype code for DP evaluation - https://phabricator.wikimedia.org/T280385 (10Htriedman) @gmodena It might also be worth considering implementing HDFS in the Beam Go SDK ourselves — the template for doing it (whi... [19:57:28] MEH! ottomata, we forgot to merge https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/698723 before releasing EventUtilities [19:58:30] Let's fix all that on monday, then we'll be ready to pull the trigger :) [19:58:40] Ending my day now team - see you tomorrow [20:03:15] OH! [20:03:23] joal: i'll merge that now and just release 1.0.8 [20:03:25] easy enough [20:04:12] joal: added forge committer [20:04:20] joal: am not sure, but maybe you can edit access yourself? [20:04:21] https://gerrit.wikimedia.org/r/admin/repos/analytics/gobblin,access [20:26:20] Starting build #10 for job wikimedia-event-utilities-maven-release-docker [20:27:29] Project wikimedia-event-utilities-maven-release-docker build #10: 09SUCCESS in 1 min 8 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release-docker/10/ [20:34:34] 10Analytics, 10Analytics-Kanban: Replace Camus by Gobblin - https://phabricator.wikimedia.org/T271232 (10Ottomata) @JAllemandou https://archiva.wikimedia.org/#artifact/org.wikimedia/eventutilities/1.0.8 [20:38:13] Thank you ottomata for the release :) I'll finalize the java code tomorrow and will provide a config patch in refinery under gobblin/ folder [20:39:22] joal: i'm working on a gobblin/ config patch in refinery now, as well as a gobblin wrapper [20:39:31] needed something to see what systemd timers would be scheduled via puppet [20:39:37] will get that patch up for you to look at [20:41:52] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Top edited pages list on enwiktionary contains nonexistent pages with titles made up of question marks - https://phabricator.wikimedia.org/T284623 (10Milimetric) >>! In T284623#7175464, @JAllemandou wrote: > Does the patch above on... [20:43:08] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban: Import of MediaWiki tables into the Data Lakes mangles usernames - https://phabricator.wikimedia.org/T230915 (10Milimetric) I'm going to merge this into the other task, even though it's a totally different symptom the root cause is the same and is fi... [20:43:22] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban: Import of MediaWiki tables into the Data Lakes mangles usernames - https://phabricator.wikimedia.org/T230915 (10Milimetric) [20:43:27] actually joal will need a gobblin/job/.properties file [20:43:33] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Top edited pages list on enwiktionary contains nonexistent pages with titles made up of question marks - https://phabricator.wikimedia.org/T284623 (10Milimetric) [20:43:37] i'll add the common ones [20:45:37] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Top edited pages list on enwiktionary contains nonexistent pages with titles made up of question marks - https://phabricator.wikimedia.org/T284623 (10Milimetric) [20:46:41] ok... that seems pretty well put to bed... onto wikistats. Now... what's a UI? [21:06:04] 10Analytics, 10Product-Analytics: Investigate Hive & Hadoop permissions for users in same group - https://phabricator.wikimedia.org/T285503 (10mpopov) [21:07:48] 10Analytics, 10Product-Analytics: Investigate Hive & Hadoop permissions for users in same group - https://phabricator.wikimedia.org/T285503 (10mpopov) [21:37:09] (03PS1) 10Ottomata: Add bin/gobblin wrapper and initial gobblin/ common properties files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/701463 (https://phabricator.wikimedia.org/T271232) [21:40:20] (03CR) 10Ottomata: Add bin/gobblin wrapper and initial gobblin/ common properties files (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/701463 (https://phabricator.wikimedia.org/T271232) (owner: 10Ottomata) [22:37:39] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: mwgate-node12-docker consistently fails for EventLogging - https://phabricator.wikimedia.org/T285372 (10Krinkle) 05Open→03Resolved a:05hashar→03Krinkle