[06:20:41] !log `elukey@an-tool1011:~$ sudo systemctl reset-failed ifup@ens13.service` - T273026 [06:20:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:20:49] T273026: Errors for ifup@ens5.service after rebooting Ganeti VMs - https://phabricator.wikimedia.org/T273026 [06:30:18] hi folks, I left a comment in https://gerrit.wikimedia.org/r/c/operations/puppet/+/792116 [06:30:34] there are two alerts for the same timer, it may be removed in my opinion [10:51:37] 10Data-Engineering, 10Data-Engineering-Kanban: Add the conftool pooled/depooled status and weight into prometheus for each service - https://phabricator.wikimedia.org/T309189 (10BTullis) [10:52:31] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Add alert for varnishkafka low/zero messages per second to alertmanager - https://phabricator.wikimedia.org/T300246 (10BTullis) [10:53:28] 10Data-Engineering, 10Data-Engineering-Kanban: Add the conftool pooled/depooled status and weight into prometheus for each service - https://phabricator.wikimedia.org/T309189 (10BTullis) p:05Triage→03Medium [10:54:25] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Add alert for varnishkafka low/zero messages per second to alertmanager - https://phabricator.wikimedia.org/T300246 (10BTullis) Moving this ticket to paused, while I work on T309189 [11:34:22] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering, 10User-TheresNoTime, and 2 others: Wikistats Bug - easy to understand language for pageviews - https://phabricator.wikimedia.org/T263973 (10TheresNoTime) 05Stalled→03Resolved a:03Milimetric >>! In T263973#7922167, @Milimetric wrote: > @Kipala &... [13:10:50] (03CR) 10Ottomata: "I think you need a conda-environment.yaml file that specifies at minimum python as a dependency." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/792215 (https://phabricator.wikimedia.org/T307714) (owner: 10Milimetric) [13:11:37] (03CR) 10Ottomata: "Also, this is not an artifact, so should not go in the artifacts dir." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/792215 (https://phabricator.wikimedia.org/T307714) (owner: 10Milimetric) [14:16:20] 10Data-Engineering, 10Data-Engineering-Kanban: Draft initial data storage platform and place budget hold for Q2 - https://phabricator.wikimedia.org/T308318 (10Ottomata) Looks great! I added a comment asking for how this (in the future) could support Shared Data Platform Multi DC for prod. [14:17:30] (03PS2) 10Milimetric: Add datahub metadata ingestion CLI as a conda env [analytics/refinery] - 10https://gerrit.wikimedia.org/r/792215 (https://phabricator.wikimedia.org/T307714) [14:23:49] (03PS1) 10Luke Bowmaker: Add Schema for Enriched MW Streams [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/799351 (https://phabricator.wikimedia.org/T308017) [14:24:24] (03CR) 10CI reject: [V: 04-1] Add Schema for Enriched MW Streams [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/799351 (https://phabricator.wikimedia.org/T308017) (owner: 10Luke Bowmaker) [14:27:38] (03PS3) 10Milimetric: Add datahub metadata ingestion CLI as a conda env [analytics/refinery] - 10https://gerrit.wikimedia.org/r/792215 (https://phabricator.wikimedia.org/T307714) [14:31:54] (03PS2) 10Luke Bowmaker: Add Schema for Enriched MW Streams [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/799351 (https://phabricator.wikimedia.org/T308017) [14:32:25] 10Data-Engineering-Radar, 10MW-1.39-notes (1.39.0-wmf.7; 2022-04-11): Decommission the UploadWizard* instruments - https://phabricator.wikimedia.org/T305238 (10Ottomata) @phuedx Should we also remove the current tables and data in the event database? There doesn't seem to be much there anyway :) [14:32:39] 10Data-Engineering: Drop sanitized GettingStarted* data - https://phabricator.wikimedia.org/T307774 (10Ottomata) Should we also remove the tables and data from the event database? [14:35:26] 10Data-Engineering: Check home/HDFS leftovers of nikkin - https://phabricator.wikimedia.org/T307420 (10Ottomata) 05Open→03Resolved a:03Ottomata nikkin has no user data on stat boxes, hdfs, or hive. Resolving. ` ====== stat1004 ====== total 0 ====== stat1005 ====== total 0 ====== stat1006 ====== total... [14:37:16] 10Data-Engineering: Check home/HDFS leftovers of nikkin - https://phabricator.wikimedia.org/T307420 (10Ottomata) Removing HDFS and user home dirs: ` sudo -u hdfs kerberos-run-command hdfs hdfs dfs -rm -r /user/nikkin ` ` sudo cumin 'C:profile::analytics::cluster::client or C:profile::hadoop::master or C:profil... [14:38:22] 10Data-Engineering: Check home/HDFS leftovers of keepit-ssh - https://phabricator.wikimedia.org/T306415 (10Ottomata) 05Open→03Resolved a:03Ottomata Thanks Sandra. Since there is no data for this user, I am removing home dirs and resolving. ` sudo -u hdfs kerberos-run-command hdfs hdfs dfs -rm -r /user/ke... [14:40:36] 10Data-Engineering: Check home/HDFS leftovers of statwithlatte - https://phabricator.wikimedia.org/T307980 (10Ottomata) This user has one Untitled Jupyter Notebook on stat1005 last modified Oct 21 2021, but nothing else. I'm going to be bold and go ahead and remove this file, as well as the empty user home dirs... [14:40:49] 10Data-Engineering: Check home/HDFS leftovers of statwithlatte - https://phabricator.wikimedia.org/T307980 (10Ottomata) 05Open→03Resolved a:03Ottomata [14:44:26] 10Data-Engineering: Check home/HDFS leftovers of jdl - https://phabricator.wikimedia.org/T306412 (10Ottomata) 05Open→03Resolved a:03Ottomata Dropping hive db tables, removing data and homedirs: ` sudo -u hdfs kerberos-run-command hdfs hive DROP DATABASE jdl CASCADE; ` ` sudo -u hdfs kerberos-run-command... [14:48:15] 10Data-Engineering: Add projects to sqoop list when synced in clouddb - https://phabricator.wikimedia.org/T304632 (10Ottomata) @JAllemandou @Snwachukwu is this done? [14:50:09] 10Data-Engineering, 10SRE, 10Traffic-Icebox: Mobile redirects drop provenance parameters - https://phabricator.wikimedia.org/T252227 (10Ottomata) Moving back to incoming, this is not an Ops Week task. [14:50:24] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Platform Engineering, 10Product-Analytics: AQS `edited-pages/new` metric does not make clear that the value is net of deletions - https://phabricator.wikimedia.org/T240860 (10Ottomata) Moving back to incoming, this is not an ops week task. [14:57:59] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Platform Engineering, 10Product-Analytics: AQS `edited-pages/new` metric does not make clear that the value is net of deletions - https://phabricator.wikimedia.org/T240860 (10EChetty) Moving this back to Requests - as a request for documentation. [15:32:27] 10Data-Engineering, 10Equity-Landscape: Transformations Flowchart - https://phabricator.wikimedia.org/T306614 (10JAnstee_WMF) 05Open→03In progress [15:32:28] 10Data-Engineering, 10Equity-Landscape: Milestone: Transformation Definitions Complete: - https://phabricator.wikimedia.org/T305474 (10JAnstee_WMF) [16:02:10] 10Data-Engineering, 10Data-Engineering-Kanban: Draft initial data storage platform and place budget hold for Q2 - https://phabricator.wikimedia.org/T308318 (10BTullis) >>! In T308318#7957112, @Ottomata wrote: > Looks great! I added a comment asking for how this (in the future) could support Shared Data Platfo... [16:02:18] 10Data-Engineering, 10Data-Engineering-Kanban: Draft initial data storage platform and place budget hold for Q2 - https://phabricator.wikimedia.org/T308318 (10BTullis) 05Open→03Resolved [16:02:20] 10Data-Engineering, 10Data-Engineering-Kanban, 10Epic: Data Infrastructure as a Service MVP - https://phabricator.wikimedia.org/T308317 (10BTullis) [16:02:43] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra: Enable Cassandra encryption (inter-node & client) - https://phabricator.wikimedia.org/T307798 (10BTullis) 05Open→03Resolved [16:02:49] 10Data-Engineering-Radar, 10Cassandra, 10Generated Data Platform: AQS multi-datacenter cluster expansion - https://phabricator.wikimedia.org/T307641 (10BTullis) [16:03:04] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Split turnilo staging off of an-tool1005 - https://phabricator.wikimedia.org/T308597 (10BTullis) 05Open→03Resolved [16:03:24] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE Observability, and 2 others: Migrate the majority of the analytics cluster alerts from Icinga to AlertManager - https://phabricator.wikimedia.org/T293399 (10BTullis) 05Open→03Resolved [16:03:44] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Superset: Help with data that's not appearing on charts - https://phabricator.wikimedia.org/T301895 (10BTullis) 05In progress→03Resolved [16:08:05] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Services, 10Patch-For-Review: Move wikireplicas dbproxy haproxy config to etcd - https://phabricator.wikimedia.org/T304478 (10BTullis) a:05razzi→03BTullis [16:08:22] 10Data-Engineering-Kanban, 10Data-Services, 10cloud-services-team (Kanban): Reimage WMCS db proxies to Bullseye - https://phabricator.wikimedia.org/T298940 (10BTullis) a:05razzi→03BTullis [16:20:35] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for kcgwiki - https://phabricator.wikimedia.org/T305280 (10nskaggs) [16:24:53] 10Data-Engineering, 10Data-Engineering-Kanban: Create conda-base-env with last pyspark - https://phabricator.wikimedia.org/T309227 (10Antoine_Quhen) [16:27:22] 10Data-Engineering, 10Data-Engineering-Kanban: Create conda-base-env with last pyspark - https://phabricator.wikimedia.org/T309227 (10Antoine_Quhen) [16:27:24] 10Data-Engineering, 10Airflow: Install spark3 in analytics clusters - https://phabricator.wikimedia.org/T295072 (10Antoine_Quhen) [16:35:08] 10Data-Engineering-Radar, 10Cassandra, 10Generated Data Platform: AQS multi-datacenter cluster expansion - https://phabricator.wikimedia.org/T307641 (10Eevans) [16:40:21] 10Data-Engineering, 10Cassandra: Make Cassandra client encryption non-optional (AQS cluster) - https://phabricator.wikimedia.org/T309229 (10Eevans) [16:40:47] 10Data-Engineering, 10Cassandra: Make Cassandra client encryption non-optional (AQS cluster) - https://phabricator.wikimedia.org/T309229 (10Eevans) p:05Triage→03Medium [16:41:38] 10Data-Engineering, 10Cassandra: Make Cassandra client encryption non-optional (AQS cluster) - https://phabricator.wikimedia.org/T309229 (10Eevans) [17:36:34] Hi mforns - would you have a minute? [17:36:43] heya joal yes! [17:36:47] bc? [17:36:50] Yes! [17:36:55] omw [20:29:24] 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Custom Metadata ingestion - https://phabricator.wikimedia.org/T307714 (10Milimetric) Jobs are up for review at https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/63, tested in prod [20:33:48] !log Pausing aqs_hourly job in airflow test intil we fix the spark3 issue [20:33:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:57:26] 10Data-Engineering: Add projects to sqoop list when synced in clouddb - https://phabricator.wikimedia.org/T304632 (10JAllemandou) 05Open→03Resolved a:03JAllemandou It is! Closing the task [20:57:30] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for guwwiki - https://phabricator.wikimedia.org/T303761 (10JAllemandou) [20:57:38] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for shnwikivoyage - https://phabricator.wikimedia.org/T302798 (10JAllemandou) [21:09:36] !log Resume aqs_hourly job in airflow test [21:09:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:10:07] mforns: for when you have amoment, the spark3 PR passes CI and works in test-env! \o/ [21:10:11] aqu --^ [21:10:20] 👍 [21:18:01] joal: LGTM! approved it [21:19:46] 10Data-Engineering, 10SRE, 10Traffic-Icebox: Mobile redirects drop provenance parameters - https://phabricator.wikimedia.org/T252227 (10mpopov) Thank you @Milimetric for the ping! I missed this earlier in the month. > I did find one dataset where `wprov` is used, by #product-analytics, so perhaps @mpopov, w... [21:37:19] 10Analytics-Kanban, 10Data-Engineering, 10Event-Platform, 10Fundraising-Backlog, and 3 others: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned - https://phabricator.wikimedia.org/T282131 (10Etonkovidova) [21:51:35] (03PS1) 10Mayakpwiki: movement_metrics: Migrate Content Interactions tables and ETL [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/799417 (https://phabricator.wikimedia.org/T308695) [23:01:53] 10Data-Engineering: Check home/HDFS leftovers of keepit-ssh - https://phabricator.wikimedia.org/T306415 (10Snwachukwu) Thanks Andrew [23:02:14] 10Data-Engineering: Add projects to sqoop list when synced in clouddb - https://phabricator.wikimedia.org/T304632 (10Snwachukwu) Thanks Joseph.