[05:01:23] (03PS1) 10Joal: Update refinery-drop-mediawiki-snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/786448 (https://phabricator.wikimedia.org/T303988) [05:04:38] (03PS1) 10Joal: Add structured_data.commons_entity to purge [analytics/refinery] - 10https://gerrit.wikimedia.org/r/786452 [05:12:09] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics (Kanban): Request for March 2022 per-host traffic - https://phabricator.wikimedia.org/T306480 (10JAllemandou) 05Open→03Resolved [05:12:23] 10Data-Engineering, 10Data-Engineering-Kanban, 10Generated Data Platform, 10Image-Suggestion-API, 10Image-Suggestions: Update HiveToCassandra for variable substitution and HQL from files loading - https://phabricator.wikimedia.org/T297934 (10JAllemandou) 05Open→03Resolved [05:21:00] (03PS2) 10Joal: Update refinery-drop-mediawiki-snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/786448 (https://phabricator.wikimedia.org/T303988) [05:24:21] (03PS3) 10Joal: Update refinery-drop-mediawiki-snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/786448 (https://phabricator.wikimedia.org/T303988) [05:45:56] (03PS4) 10Joal: Update refinery-drop-mediawiki-snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/786448 (https://phabricator.wikimedia.org/T303988) [05:55:23] (03CR) 10Joal: [V: 03+1] "Tested in dry-run mode" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/786448 (https://phabricator.wikimedia.org/T303988) (owner: 10Joal) [06:16:59] (03CR) 10Joal: [V: 03+1] "Tested in dry-run/non-strict mode" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/786452 (owner: 10Joal) [06:24:16] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics: Improvements to mediawiki_geoeditors_monthly dimensions - https://phabricator.wikimedia.org/T302079 (10JAllemandou) I wonder whether the requested change should be done for the data in Druid only, or if it would be valuable to change the geo... [06:34:27] 10Data-Engineering, 10Product-Analytics, 10Research: Update HDFS links tables as Mediawiki changes - https://phabricator.wikimedia.org/T304979 (10JAllemandou) Thank you @Isaac for creating this task! I have reviewed the planned change and it will indeed impact our sqooping/usage of the various links tables.... [06:43:39] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Research: Update HDFS links tables as Mediawiki changes - https://phabricator.wikimedia.org/T304979 (10JAllemandou) a:03JAllemandou [06:52:26] 10Data-Engineering, 10Data-Engineering-Kanban: Plan spark3 migration - possibly incrementally - https://phabricator.wikimedia.org/T306955 (10JAllemandou) [06:53:13] 10Data-Engineering, 10Data-Engineering-Kanban: Plan spark3 jobs migration - possibly incrementally - https://phabricator.wikimedia.org/T306955 (10JAllemandou) [06:53:28] 10Data-Engineering, 10Data-Engineering-Kanban: Plan spark3 migration - possibly incrementally - https://phabricator.wikimedia.org/T306955 (10JAllemandou) [06:55:32] 10Data-Engineering, 10Data-Engineering-Kanban: Plan spark3 migration - possibly incrementally - https://phabricator.wikimedia.org/T306955 (10JAllemandou) [06:55:34] 10Analytics, 10Data-Engineering, 10Epic: Upgrade analytics-hadoop to Spark 3 + scala 2.12 - https://phabricator.wikimedia.org/T291464 (10JAllemandou) [06:59:21] Hi, I noticed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/786430 that they are still a lot of references to Analytics SREs, should these all be Data Engineering now? [07:03:58] 10Data-Engineering, 10Patch-For-Review: Upgrade Refinery Jobs to Spark 3 - https://phabricator.wikimedia.org/T291386 (10JAllemandou) [07:04:00] 10Data-Engineering, 10Data-Engineering-Kanban: Plan spark3 migration - possibly incrementally - https://phabricator.wikimedia.org/T306955 (10JAllemandou) [08:19:19] 10Data-Engineering, 10Data-Engineering-Kanban: Update ua-parser library for traffic data - https://phabricator.wikimedia.org/T306829 (10Antoine_Quhen) The contributor told me that he is going to update uap-java next week. [08:38:42] 10Data-Engineering, 10Airflow: Use airflow to load cassandra - https://phabricator.wikimedia.org/T306962 (10JAllemandou) [08:49:45] RhinosF1: Yes, they probably should all be changed now. [08:50:05] btullis: can you upload a patch? [08:50:20] RhinosF1: Yep, will do. [08:56:05] Thanks :) [08:57:31] Here we are: https://gerrit.wikimedia.org/r/c/operations/puppet/+/786848 [09:09:22] btullis: thanks! [09:09:41] RhinosF1: A pleasure. [09:11:27] :) [09:32:43] 10Data-Engineering: Crash of artifact-cache in scap deploy context - https://phabricator.wikimedia.org/T305868 (10Antoine_Quhen) How to reproduce manually and currently, on an-launcher1002: ` hdfs dfs -rm /wmf/cache/artifacts/airflow/org.wikimedia.analytics.refinery.hive_refinery-hive_jar_shaded_0.1.27 # 2 tim... [09:39:42] 10Data-Engineering: Crash of artifact-cache in scap deploy context - https://phabricator.wikimedia.org/T305868 (10Antoine_Quhen) https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags-scap-analytics/-/merge_requests/2 [10:00:01] I looked the refine_eventlogging_legacy failure with SandraEbele - but we weren't able to work out how to get it to complete. [10:05:25] Should we follow the instructions here? https://wikitech.wikimedia.org/wiki/Analytics/Systems/Refine#Rerunning_a_Refine_job_that_has_a_malformed_record [10:07:15] The yarn log doesn't mention 'malformed' but the job is complaining with: `Could not extract /$schema field from event, field does not exist` [10:18:23] (03CR) 10Kosta Harlan: homepagemodule: Add mentorship-optout/mentorship-optin actions (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/786268 (https://phabricator.wikimedia.org/T287915) (owner: 10Urbanecm) [10:57:54] 10Data-Engineering, 10Data-Catalog: Integrate Spark with DataHub - https://phabricator.wikimedia.org/T306896 (10BTullis) ==Spark commands supported​== Below is a list of Spark commands that are parsed currently: - InsertIntoHadoopFsRelationCommand - SaveIntoDataSourceCommand (jdbc) - Create... [11:00:46] 10Data-Engineering: Review refinery scripts so that they no longer depend on _SUCCESS files - https://phabricator.wikimedia.org/T306611 (10JAllemandou) [11:01:09] 10Data-Engineering: Review refinery scripts so that they no longer depend on _SUCCESS files - https://phabricator.wikimedia.org/T306611 (10JAllemandou) [11:01:12] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Refactor refinery-drop-mediawiki-snapshots so that it no longer uses a _SUCCESS file - https://phabricator.wikimedia.org/T303988 (10JAllemandou) [11:41:27] 10Data-Engineering, 10Data-Catalog: Integrate Spark with DataHub - https://phabricator.wikimedia.org/T306896 (10BTullis) ==Configuring Spark/DataHub integration in Notebooks== I executed the following code in a notebook in order to test the Spark/DataHub integration. ` import wmfdata spark = wmfdata.spark.ge... [11:47:23] 10Data-Engineering, 10Airflow, 10Data-Catalog: Integrate Airflow with DataHub - https://phabricator.wikimedia.org/T306977 (10BTullis) [11:47:54] 10Data-Engineering, 10Airflow, 10Data-Catalog: Integrate Airflow with DataHub - https://phabricator.wikimedia.org/T306977 (10BTullis) a:05BTullis→03None [11:51:40] 10Data-Engineering, 10Data-Catalog: Integrate Spark with DataHub - https://phabricator.wikimedia.org/T306896 (10BTullis) The configuration for spark-submit jobs should be very similar. https://datahubproject.io/docs/metadata-integration/java/spark-lineage#configuration-instructions-spark-submit [11:59:48] 10Data-Engineering, 10Data-Catalog: Integrate Spark with DataHub - https://phabricator.wikimedia.org/T306896 (10BTullis) [12:35:29] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Research: Update HDFS links tables as Mediawiki changes - https://phabricator.wikimedia.org/T304979 (10Ladsgroup) >>! In T304979#7883363, @JAllemandou wrote: > I've added myself as a subscriber of the migration task to follow implementat... [13:22:01] 10Data-Engineering, 10Airflow: Low Risk Oozie Migration: Mediawiki History Dumps - https://phabricator.wikimedia.org/T300344 (10JArguello-WMF) [13:23:17] 10Data-Engineering, 10Data-Engineering-Kanban: Crash of artifact-cache in scap deploy context - https://phabricator.wikimedia.org/T305868 (10Ottomata) [13:27:08] 10Data-Engineering, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10MNeisler) [13:31:19] 10Data-Engineering, 10Data-Engineering-Kanban: Implement one golang AQS microservice - https://phabricator.wikimedia.org/T299729 (10JArguello-WMF) 05Open→03Stalled p:05Triage→03Low This task is up for grabs since Dan's main focus right now is to help with the Airflow migration. [13:38:26] 10Data-Engineering, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10MNeisler) Note: This appears to be impacting all of #product-analytics at least and is currently blocking the following analyses that require access to the [[ https://www.med... [13:41:39] 10Data-Engineering, 10Data-Engineering-Kanban: Crash of artifact-cache in scap deploy context - https://phabricator.wikimedia.org/T305868 (10Ottomata) https://github.com/fsspec/filesystem_spec/issues/874#issuecomment-1111015333 [13:44:18] 10Data-Engineering, 10Data-Engineering-Kanban: Crash of artifact-cache in scap deploy context - https://phabricator.wikimedia.org/T305868 (10Ottomata) Seems to be a bug with fsspec + the new pyarrow API. I think we have to go back to not using the new pyarrow API for now. We can just avoid calling `fsspec_us... [13:58:01] 10Data-Engineering, 10Airflow: [Airflow] Organize hackathon - https://phabricator.wikimedia.org/T295204 (10mforns) [14:25:12] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10BTullis) p:05Triage→03Unbreak! Looking into this for you now with the highest priority. [14:36:52] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10BTullis) It looks to me like the databases have gone from this host. ` btullis@stat1008:~$ mysql -h x1-analytics-replica.eqiad.wmnet -P 3320 Welc... [14:37:05] 10Data-Engineering, 10Airflow, 10Data-Catalog: Integrate Airflow with DataHub - https://phabricator.wikimedia.org/T306977 (10mforns) [14:42:04] 10Data-Engineering, 10Airflow: Use airflow to load cassandra - https://phabricator.wikimedia.org/T306962 (10mforns) @JAllemandou Thanks for working on this! I think an Operator will be enough no? Or is there more than 1 action to do? IIUC, the only thing we need to do is to run the Spark-Scala code that reads... [14:47:29] (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/786452 (owner: 10Joal) [15:12:11] hey folks. anyone doing something on dbstore1005? just saw an alert for it [15:12:47] (03CR) 10Mforns: "Code looks good to me!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/786448 (https://phabricator.wikimedia.org/T303988) (owner: 10Joal) [15:21:01] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10BTullis) It looks like this is a missing grant issue. I haven't worked out quite how/why it was deleted, but I'm working on adding it back now. [16:02:35] kormat: it's probably related to this ticket: https://phabricator.wikimedia.org/T306984 [16:05:24] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) Hello, I have triaged this and it is now available. I am unsure what is going on and I will need more time (and probably) do... [16:06:00] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) ` research@dbstore1005.eqiad.wmnet[(none)]> show databases; +---------------------------+ | Database | +---... [16:07:48] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) p:05Unbreak!→03Medium Decreasing from UBN to normal as the user works again. I will wait for a date where the reports ar... [16:10:15] 10Data-Engineering-Kanban, 10Airflow, 10Documentation: [Airflow] Kick off documentation in wikitech - https://phabricator.wikimedia.org/T302400 (10mforns) [16:11:29] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Organize hackathon - https://phabricator.wikimedia.org/T295204 (10mforns) a:03mforns [16:14:38] (03CR) 10D3r1ck01: [C: 03+1] "LTGM!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/770059 (owner: 10EpicPupper) [16:16:29] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) Interestingly, I have been able to get `research` user back on its original `research_role` by dropping and recreating that... [16:19:52] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10BTullis) Thanks @Marostegui - That's excellent! [16:20:09] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) @MNeisler please confirm if you can access everything on x1 again. Thanks [16:29:18] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10BTullis) >>! In T306984#7884904, @Marostegui wrote: > Hello, I have triaged this and it is now available. I am unsure what is going on a... [16:32:55] ottomata: wanna sre sync? [16:33:05] k [16:33:16] elukey: invited too :) [16:36:29] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) If everything works fine again there's no need to get downtime, I can drop the temporary role without any. Let's wait for @M... [16:44:53] razzi: snap sorry I totally missed it, I had to go afk for a sec [16:45:02] still there? If I am needed I'll join :) [16:47:32] elukey: yeah, we're in the batcave :) [16:49:45] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10EChetty) p:05Medium→03Unbreak! [16:51:34] (03CR) 10Sharvaniharan: [C: 03+2] "Merging this, thank you for the review :-)" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 (owner: 10Sharvaniharan) [16:52:10] (03Merged) 10jenkins-bot: Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 (owner: 10Sharvaniharan) [16:55:36] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10EChetty) @Marostegui & @BTullis , @Iflorez (and I assume others) are still having access issues. Are there multiple places where we mana... [16:56:20] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) No, they might need to reset the connection though [17:01:25] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) I am successfully using the research user as it can be seen here: ` research@dbstore1005.eqiad.wmnet[wikishared]> SELECT CU... [17:08:00] 10Data-Engineering, 10Data-Engineering-Kanban, 10DC-Ops, 10Infrastructure-Foundations: clouddb1021 missing network firmware bnx2x/bnx2x-e2-7.13.21.0.fw in Debian 11 Bullseye - https://phabricator.wikimedia.org/T306148 (10razzi) 05Open→03Resolved a:03razzi [17:31:30] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Iflorez) Hello, I'm unable to access x1, s7, and ToolForge. The error message I'm getting: ` Unable to connect to host x1-analytics-r... [17:39:33] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Iflorez) x1 and s7 are not providing error details. This is the ToolForge error detail: ` Used command: /usr/bin/ssh -v -N -S none -o... [17:40:21] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Ladsgroup) I did a `FLUSH PRIVILEGES;` right now. If it's still not working, can you give the exact command you run? [17:41:48] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Ladsgroup) I don't think `tools.db.svc.wikimedia.cloud` is dbstore1005. [17:41:59] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) @Iflorez that's an SSH error not a MySQL error. [17:42:35] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Ladsgroup) It's `clouddb1001.clouddb-services.eqiad1.wikimedia.cloud` [17:44:23] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Ladsgroup) >>! In T306984#7885455, @Marostegui wrote: > @Iflorez that's an SSH error not a MySQL error. Yeah. I think @Iflorez has two... [17:57:42] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10MNeisler) Confirming I can now access dbs on x1 via `analytics-mysql` and on stat8 Jupyter notebook with the wmfdata package. Thanks all! [18:06:23] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Majavah) @Iflorez I am seeing this in the logs: ` Apr 27 17:35:09 tools-sgebastion-07 sshd[23227]: Failed publickey for iflorez from x.x... [18:10:26] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Ladsgroup) >>! In T306984#7885509, @Majavah wrote: >>>! In T306984#7885456, @Ladsgroup wrote: >> It's `clouddb1001.clouddb-services.eqia... [18:22:48] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering-Radar, 10Product-Analytics, and 3 others: Wikistats pageview data missing counts for Mobile App pageviews on Commons, going back to 2020-11 - https://phabricator.wikimedia.org/T299439 (10JTannerWMF) [18:38:51] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10User-Eevans: Properly add aqsloader user (w/ secrets) - https://phabricator.wikimedia.org/T305600 (10Eevans) >>! In T305600#7877875, @JAllemandou wrote: > I support keeping two users to separate loading from accessing but it's not a strong opini... [19:06:52] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Services: View wb_changes_dispatch in commonswiki_p shows an error - https://phabricator.wikimedia.org/T304591 (10razzi) 05Open→03Resolved This is done. Since none of the views were changing but passing `--replace-all` was hanging since some views wer... [19:37:01] !log restarting airflow services on all airflow instances after installing updated airflow debian package [19:37:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:43:54] 10Data-Engineering, 10Data-Engineering-Kanban: Crash of artifact-cache in scap deploy context - https://phabricator.wikimedia.org/T305868 (10Ottomata) Okay, fixed and deployed. All artifacts should be synced now. The fixes and improvements are in this MR: https://gitlab.wikimedia.org/repos/data-engineering/w... [19:45:04] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Refactor jobs to not use DAG factories - https://phabricator.wikimedia.org/T302391 (10Ottomata) Don't the anomoly detection dags still use a dag factory? [19:51:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [20:13:00] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Marostegui) @iflorez you might want to open a different task for your issue, as the MySQL one seems solved (pending me to clean up the o... [20:16:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [20:17:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [20:22:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [21:10:21] 10Data-Engineering, 10Data-Catalog, 10Product-Analytics: Propagate field descriptions from event schemas to metastore - https://phabricator.wikimedia.org/T307040 (10mpopov) [21:16:54] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10Iflorez) I'm now able to access x1 on sequel pro, the password for sequel pro to access the stat machines had to be reset. I'm now able... [21:22:29] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Product-Analytics: Denied access error when querying wikishared dbs - https://phabricator.wikimedia.org/T306984 (10EChetty) p:05Unbreak!→03Medium Thank you all! [23:48:58] 10Data-Engineering-Radar, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Reimage WMCS db proxies to Bullseye - https://phabricator.wikimedia.org/T298940 (10razzi) If we merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/779915/ that @BTullis and I came up with, we can do the f...