[09:52:07] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add 6 worker nodes to the HDFS Namenode config of the Analytics Hadoop cluster - https://phabricator.wikimedia.org/T275767 (10BTullis) I am making the change to `common.yaml` prior to changing the role of the servers. Patch here: https://gerrit.wi... [09:53:57] (03CR) 10Hnowlan: Add docker-compose environment with cassandra (031 comment) [analytics/aqs] - 10https://gerrit.wikimedia.org/r/679295 (https://phabricator.wikimedia.org/T257572) (owner: 10Hnowlan) [10:06:00] (03CR) 10David Caro: [C: 03+2] Add database autocompletion [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/713237 (https://phabricator.wikimedia.org/T287471) (owner: 10David Caro) [10:06:06] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add 6 worker nodes to the HDFS Namenode config of the Analytics Hadoop cluster - https://phabricator.wikimedia.org/T275767 (10BTullis) I have created the journal on each of the six new nodes, using the script that is shown here: https://wikitech.w... [10:07:42] (03Merged) 10jenkins-bot: Add database autocompletion [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/713237 (https://phabricator.wikimedia.org/T287471) (owner: 10David Caro) [10:59:53] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add 6 worker nodes to the HDFS Namenode config of the Analytics Hadoop cluster - https://phabricator.wikimedia.org/T275767 (10BTullis) Three of the six nodes (29,33,34) already had principals and keytabs. The remaining three (39,40,41) did not, so... [11:16:53] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add 6 worker nodes to the HDFS Namenode config of the Analytics Hadoop cluster - https://phabricator.wikimedia.org/T275767 (10BTullis) I have merged the change to the net topology and applied the change to an-master100[1-2], so now I need to carry... [12:14:23] (03CR) 10Ladsgroup: [C: 03+2] Track number of active items [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/713850 (https://phabricator.wikimedia.org/T286903) (owner: 10Lucas Werkmeister (WMDE)) [12:14:47] (03CR) 10Ladsgroup: [C: 03+2] Add comment to active_items.sql [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/713897 (https://phabricator.wikimedia.org/T286903) (owner: 10Lucas Werkmeister (WMDE)) [12:16:23] (03Merged) 10jenkins-bot: Track number of active items [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/713850 (https://phabricator.wikimedia.org/T286903) (owner: 10Lucas Werkmeister (WMDE)) [12:16:27] (03Merged) 10jenkins-bot: Add comment to active_items.sql [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/713897 (https://phabricator.wikimedia.org/T286903) (owner: 10Lucas Werkmeister (WMDE)) [12:16:47] (03PS1) 10Ladsgroup: Add comment to active_items.sql [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/714153 (https://phabricator.wikimedia.org/T286903) [12:16:52] (03CR) 10Ladsgroup: [C: 03+2] Add comment to active_items.sql [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/714153 (https://phabricator.wikimedia.org/T286903) (owner: 10Ladsgroup) [12:17:56] (03Merged) 10jenkins-bot: Add comment to active_items.sql [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/714153 (https://phabricator.wikimedia.org/T286903) (owner: 10Ladsgroup) [13:02:38] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add 6 worker nodes to the HDFS Namenode config of the Analytics Hadoop cluster - https://phabricator.wikimedia.org/T275767 (10BTullis) I am planning to run the cookbook at 14:00 UTC - I have reached out to product-analytics to let them know about... [13:31:33] (03PS5) 10Joal: [WIP] Add cassandra3 to oozie loading jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/706605 (https://phabricator.wikimedia.org/T280649) [13:58:55] I'm about to commence roll-restarting the hadoop analytics cluster master services, to pick up the new namenode configuration, prior to adding the six new worker nodes. [14:00:21] ack btullis - what i the conf you're updating? [14:01:15] https://phabricator.wikimedia.org/T275767 and this is the specific change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/714331 [14:02:27] ack btullis - thanks :) [14:03:57] My first ever hadoop master restart :) [14:06:27] btullis: let me know if you wish support (not that I could do anything, but I can be present with you :) [14:09:25] Thanks. Will do. I'm running the cookbook so hopefully all should be business as usual. I'm keeping an eye on yarn etc. Please let me know if anything seems untoward to you though. [14:10:07] ack! [14:12:51] hi team! :] [14:13:16] Hiya mforns. [14:13:34] hey btullis :] [14:14:36] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Metrics-Platform, 10Product-Data-Infrastructure: Client-side error logging should use Elastic Common Schema (ECS) fields when possible - https://phabricator.wikimedia.org/T267602 (10ldelench_wmf) p:05Medium→03Low [14:22:25] 10Analytics-Kanban, 10 Data-Engineering, 10Security-Team, 10Security: Add check to make sure deny-list countries aren't being passed through AQS - https://phabricator.wikimedia.org/T289279 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:22:45] 10Analytics, 10Analytics-Kanban, 10 Data-Engineering, 10Data-Engineering-Kanban: Change state to store project as an array - https://phabricator.wikimedia.org/T283624 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:23:01] 10Analytics, 10Analytics-Kanban: Change routing to accept a list of wikis in URL - https://phabricator.wikimedia.org/T283596 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:23:56] 10Analytics: Add ignore success flags option to pageview monthly dumps - https://phabricator.wikimedia.org/T283593 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:24:05] 10Analytics, 10Analytics-Wikistats: wikistats: montly pageview dumps are not bz2 files - https://phabricator.wikimedia.org/T287684 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:24:12] 10Analytics, 10Analytics-Kanban: Create monthly job for canonical pageviews - https://phabricator.wikimedia.org/T265732 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:24:15] 10Analytics, 10Analytics-Kanban: pagecounts-ez of month 2020-08 is incomplete - https://phabricator.wikimedia.org/T262141 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:24:18] 10Analytics, 10Analytics-Kanban, 10 Data-Engineering, 10Data-Engineering-Kanban: Wikistats should allow more than one project - https://phabricator.wikimedia.org/T283254 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:24:20] 10Analytics: Wikistats2 time related bugs - https://phabricator.wikimedia.org/T231248 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:24:28] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Creation of canonical pageview dumps for users to download - https://phabricator.wikimedia.org/T251777 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:24:30] 10Analytics, 10Analytics-Wikistats, 10Inuka-Team, 10Language-strategy, and 2 others: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:24:33] 10Analytics, 10Analytics-Kanban, 10 Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Expand Wikiselector to allow more than one wiki - https://phabricator.wikimedia.org/T285050 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:24:56] 10Analytics, 10Analytics-Kanban: Archive /home/ezachte data on stat1007 - https://phabricator.wikimedia.org/T238243 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:24:58] 10Analytics: Automate the deployment procedure of Wikistats 2 to Production - https://phabricator.wikimedia.org/T274126 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:25:00] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Migrate pagecounts-ez generation to hadoop - https://phabricator.wikimedia.org/T192474 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:25:03] 10Analytics, 10incubator.wikimedia.org: Create dashiki dashboard / small tool to track statistics about incubated wikis - https://phabricator.wikimedia.org/T237389 (10Aklapper) a:05fdans→03None (Resetting inactive assignee account) [14:26:55] fdans: good luck on your next adventures [14:28:10] Hadoop server rolling restart looks to have completed without incident. [14:28:49] \o/ [14:41:03] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add 6 worker nodes to the HDFS Namenode config of the Analytics Hadoop cluster - https://phabricator.wikimedia.org/T275767 (10BTullis) The cookbook ran successfully and the restart of the daemons appears to have been without incident. I have now p... [14:42:29] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Add analytics-presto.eqiad.wmnet CNAME for Presto coordinator failover - https://phabricator.wikimedia.org/T273642 (10BTullis) 05Open→03Resolved [14:42:31] 10Analytics: Analytics coordinator failover improvements - https://phabricator.wikimedia.org/T280905 (10BTullis) [14:43:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: jupyter notebook causing syslog/etc.. to fill up with error messages - https://phabricator.wikimedia.org/T287339 (10BTullis) 05Open→03Resolved [14:47:23] (03CR) 10Mforns: "LGTM overall! I left a couple very minor comments." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/706605 (https://phabricator.wikimedia.org/T280649) (owner: 10Joal) [15:02:09] (03CR) 10Joal: "Thank you for the review Marcel - More coordinators too come if I manage to overcome an issue I have in test." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/706605 (https://phabricator.wikimedia.org/T280649) (owner: 10Joal) [15:03:27] (03PS6) 10Joal: [WIP] Add cassandra3 to oozie loading jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/706605 (https://phabricator.wikimedia.org/T280649) [15:35:13] 10Quarry, 10cloud-services-team (Kanban): Write a blog post about our plans for Quarry - https://phabricator.wikimedia.org/T289503 (10Andrew) [15:41:24] 10Analytics-Kanban, 10 Data-Engineering, 10Privacy Engineering, 10Privacy, and 2 others: Add check to make sure deny-list countries aren't being passed through AQS - https://phabricator.wikimedia.org/T289279 (10sbassett) [15:45:42] 10Quarry, 10cloud-services-team (Kanban): Write a blog post about our plans for Quarry - https://phabricator.wikimedia.org/T289503 (10nskaggs) p:05Triage→03High [15:49:49] 10Quarry, 10Patch-For-Review: Add database selector - https://phabricator.wikimedia.org/T76466 (10nskaggs) [16:01:52] 10Quarry, 10Patch-For-Review: [quarry] ensure that the DB and the models are in sync - https://phabricator.wikimedia.org/T288523 (10nskaggs) [16:02:12] 10Quarry, 10Patch-For-Review, 10cloud-services-team (FY2021/2022-Q1): Develop Quarry tests - https://phabricator.wikimedia.org/T210359 (10nskaggs) [16:03:27] 10Analytics, 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 (10odimitrijevic) @jlinehan is this something that you can help with in @Ottomata's absense? [16:04:57] 10Analytics, 10 Data-Engineering, 10Product-Analytics: Reconstruct Hive & Hadoop permissions for shared database - https://phabricator.wikimedia.org/T288983 (10odimitrijevic) p:05Triage→03High [16:07:04] 10Analytics, 10 Data-Engineering, 10Growth-Team, 10Metrics-Platform, and 4 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10odimitrijevic) p:05Triage→03High [16:08:00] 10Analytics-Clusters: Move the Data Engineering infrastructure to Debian Bullseye - https://phabricator.wikimedia.org/T288804 (10odimitrijevic) [16:09:38] 10Analytics-Clusters: LVS in Analytics VLANs - https://phabricator.wikimedia.org/T288750 (10odimitrijevic) [16:10:02] 10Analytics-Clusters, 10 Data-Engineering: LVS in Analytics VLANs - https://phabricator.wikimedia.org/T288750 (10odimitrijevic) [16:12:15] 10Analytics, 10 Data-Engineering, 10Product-Analytics, 10Epic: Reconstruct Hive & Hadoop permissions for shared database - https://phabricator.wikimedia.org/T288983 (10ldelench_wmf) [16:15:42] 10Analytics, 10 Data-Engineering: Data Loss Check always shows false positives - https://phabricator.wikimedia.org/T288496 (10odimitrijevic) There is some research that needs to be done to understand late data arrival. Work can be done to make our jobs resilient to data arrival. [16:25:46] 10Analytics, 10 Data-Engineering: Make it possible to use anaconda + stacked conda envs for Airflow executors - https://phabricator.wikimedia.org/T288271 (10odimitrijevic) p:05Triage→03High We need to resolve computational resources for Airflow and discuss depending on the executors that are implemented. [16:32:07] 10Analytics, 10Traffic: Review use of realloc in varnishkafka - https://phabricator.wikimedia.org/T287561 (10odimitrijevic) Adding Traffic team to help assess the patch and push to production if appropriate. [16:32:44] 10Analytics-Radar, 10Product-Analytics (Kanban): Big increase in traffic for projects except 'wikipedia' family since Feb 14th - https://phabricator.wikimedia.org/T274823 (10cchen) 05Open→03Resolved [16:32:47] 10Analytics: Improve pageview automated traffic detection heuristics - https://phabricator.wikimedia.org/T280565 (10cchen) [16:34:44] 10Analytics, 10 Data-Engineering: Data structuring guidance request - https://phabricator.wikimedia.org/T287402 (10odimitrijevic) p:05Triage→03High There are broader ongoing conversations about an approach to bringing this data into the data lake. [16:37:11] 10Analytics: Update ROCm version on GPU instances. - https://phabricator.wikimedia.org/T287267 (10odimitrijevic) @elukey is this work that the ML team plans on implementing? [16:40:55] 10Analytics-Radar, 10Product-Analytics (Kanban): Hive table neilpquinn.toledo_pageviews missing almost all data - https://phabricator.wikimedia.org/T277781 (10nshahquinn-wmf) 05Open→03Resolved >>! In T277781#6935074, @nshahquinn-wmf wrote: > Since I'm confident this was my own error, no further investigati... [16:51:12] 10Analytics-Radar, 10Event-Platform, 10Metrics-Platform, 10Product-Analytics, 10Product-Data-Infrastructure: Draft of full process for instrumentation using new client libraries - https://phabricator.wikimedia.org/T275694 (10mpopov) a:05mpopov→03None [16:54:18] 10Analytics-Radar, 10Event-Platform, 10Metrics-Platform, 10Product-Analytics, 10Product-Data-Infrastructure: Draft of full process for instrumentation using new client libraries - https://phabricator.wikimedia.org/T275694 (10mpopov) [16:59:58] 10Analytics-Clusters, 10Analytics-Radar, 10SRE, 10SRE Observability (FY2021/2022-Q1): Move kafkamon hosts to Debian Buster - https://phabricator.wikimedia.org/T252773 (10BTullis) @herron - I have a quick question. In the task description it says: > Temporary fork role::kafka::monitoring to role::kafka::... [18:04:53] 10Analytics-Radar, 10Product-Analytics: How often do people try to edit on mobile devices, using the desktop site, at the English Wikipedia? - https://phabricator.wikimedia.org/T288972 (10ldelench_wmf) [18:04:59] 10Analytics, 10 Data-Engineering, 10Growth-Team, 10Metrics-Platform, and 4 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10mforns) a:03Mholloway Just for the record, some discussion has happened in https://gerrit.wikime... [18:09:21] 10Analytics-Radar, 10Product-Analytics: How often do people try to edit on mobile devices, using the desktop site, at the English Wikipedia? - https://phabricator.wikimedia.org/T288972 (10ldelench_wmf) Hi @Whatamidoing-WMF, we reviewed this at our board refinement meeting today - could you please provide some... [18:10:37] 10Analytics-Radar, 10Product-Analytics: How often do people try to edit on mobile devices, using the desktop site, at the English Wikipedia? - https://phabricator.wikimedia.org/T288972 (10Whatamidoing-WMF) [18:11:14] 10Analytics, 10Traffic: Review use of realloc in varnishkafka - https://phabricator.wikimedia.org/T287561 (10odimitrijevic) [18:11:24] 10Analytics-Radar, 10Product-Analytics: How often do people try to edit on mobile devices, using the desktop site, at the English Wikipedia? - https://phabricator.wikimedia.org/T288972 (10Whatamidoing-WMF) @ldelench_wmf, I have just updated the description. [18:24:15] 10Analytics-Clusters, 10Analytics-Radar, 10SRE, 10Patch-For-Review, 10SRE Observability (FY2021/2022-Q1): Move kafkamon hosts to Debian Buster - https://phabricator.wikimedia.org/T252773 (10herron) I opted to remove `role::kafka::monitoring` in favor of `role::kafka::monitoring_buster` so the config woul... [18:49:52] (03PS1) 10Joal: Update deny list for geoeditors [analytics/refinery] - 10https://gerrit.wikimedia.org/r/714402 [18:58:57] (03CR) 10Mforns: [C: 03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/714402 (owner: 10Joal) [19:33:38] (03CR) 10ODimitrijevic: "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/714402 (owner: 10Joal) [19:34:05] (03CR) 10ODimitrijevic: [C: 03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/714402 (owner: 10Joal) [20:26:44] 10Analytics, 10Data-release: Wikipedia Clickstream dataset. Programmatic Access - https://phabricator.wikimedia.org/T134231 (10Isaac) FYI in case this task is picked up, we had an Outreachy intern build an API and interface for exploring the clickstream dataset. I would still love to see the API moved to forma... [21:51:44] o/ hive is being really slow for me (both via command line and pyspark) and trying to see what's going on. for example, even the query `select * from wmf.webrequest where year = 2021 and month = 7 and day = 22 limit 1` was hanging for a long time and i haven't been able to get a simple select query that just looks at an hour of webrequest data to complete and it's going on at least 10 minutes [23:15:41] (03PS1) 10GoranSMilovanovic: T283575 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/714433 [23:15:52] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] T283575 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/714433 (owner: 10GoranSMilovanovic)