[11:42:11] Getting an error in superset, > presto error: line 25:1: Function array_frequency not registered [11:42:43] Do I have to do something in my script to access the array functions? Or maybe I'm looking at the wrong API version? [11:43:17] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Recreate analytics-meta replica on db1108 from master on an-coord1001 - https://phabricator.wikimedia.org/T295312 (10BTullis) Thanks again @jcrespo - Lots to take in and consider there. As things stand, I... [11:43:23] awight: Oh, I've not seen that before. Are you in the SQLLLab part of it? [11:45:32] awight: Yes, I think that you might need the older version of the Presto API: https://prestodb.io/docs/0.246/functions/array.html [11:46:02] awight: `array_frequency` isn't mentioned on that page :-( [11:48:44] btullis: Thank you! [11:48:50] * awight squeezes eyes and wishes for a polyfill [11:52:36] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Recreate analytics-meta replica on db1108 from master on an-coord1001 - https://phabricator.wikimedia.org/T295312 (10jcrespo) > I think that I can resolve this ticket of course! > only having a weekly ba... [12:03:54] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Recreate analytics-meta replica on db1108 from master on an-coord1001 - https://phabricator.wikimedia.org/T295312 (10BTullis) > Note you not only have that- you should have also available binlogs for point... [12:05:32] Too bad: "histogram" isn't a good substitute, it uses 40 GB of memory and dies O_O [12:09:33] awight: Hmm, yeah that's a blow. If you have a compelling case for an upgrade you could make a new ticket and link to this as its parent: https://phabricator.wikimedia.org/T294259 - Can't promise to make it happen quickly, but it all helps. [12:35:20] Filed as T295589 :-) [12:35:20] T295589: Upgrade Presto to access UDF library improvements - https://phabricator.wikimedia.org/T295589 [12:37:05] Great, thanks. Will try to get it discussed soon. [12:45:30] Thanks! I'll work around it this time so no rush, but would be nice to have on the next round. [14:20:24] 10Analytics-Radar, 10Event-Platform, 10WMF-JobQueue, 10Wikibase change dispatching scripts to jobs, and 2 others: Queuing jobs is extremely slow - https://phabricator.wikimedia.org/T292048 (10Ottomata) I think you are right, looking at eventgate -> kafka latencies, etc, it does look like the topic balance... [14:22:24] 10Analytics-Radar, 10Data-Engineering, 10Event-Platform, 10SRE, 10Patch-For-Review: Allow kafka clients to verify brokers hostnames when using SSL - https://phabricator.wikimedia.org/T291905 (10Ottomata) https://wikitech.wikimedia.org/wiki/PKI/Cloud ? maybe we can make it work? [14:30:02] 10Analytics-Radar, 10Data-Engineering, 10Event-Platform, 10SRE, 10Patch-For-Review: Allow kafka clients to verify brokers hostnames when using SSL - https://phabricator.wikimedia.org/T291905 (10elukey) >>! In T291905#7500211, @Ottomata wrote: > https://wikitech.wikimedia.org/wiki/PKI/Cloud ? maybe we ca... [17:17:02] mforns: we have scap! [17:17:12] gotta update docs, but scap deploy -e hadoop-test [17:17:14] wowowowowoooooo [17:17:23] will deploy to an-test-client1001 [17:17:32] \o/ [17:17:33] if you are ready to have the dags folder on an-launcher changed [17:17:35] i can do it there too [17:18:03] what do you mean with the dags folder changed? [17:18:11] you mean wiped? yes, it can be wiped! [17:19:38] BTW ottomata :] I've been fighting with conda and stacked conda envs within my ru_dev_instance.sh script, and could not manage to make it work. I'm spending a lot of time with that, and I'm sure I'm doing something wrong... [17:21:45] yes wiped [17:21:46] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Refactor analytics-meta MariaDB layout to use an-db100[12] - https://phabricator.wikimedia.org/T284150 (10BTullis) [17:21:54] mforns: hmmm [17:22:00] ru_dev_instance.sh ? [17:22:06] anyway i can help but i can't bc right atm [17:22:07] what's up? [17:22:13] what's wrong? [17:22:24] ottomata: yes, the script that spins up an airflow instance for development [17:22:32] what I try is the folowing: [17:22:39] ah [17:22:58] create a stacked conda environment with: /home/mforns/Projects/airflow-dags/run_dev_instance.sh [17:23:03] uop, not that [17:23:48] this: /usr/lib/anaconda-wmf/bin/conda-create-stacked airflow_dev_instance [17:24:02] okay [17:24:12] and then activate it with: source /usr/lib/anaconda-wmf/bin/conda-activate-stacked airflow_dev_instance [17:24:19] okay [17:24:36] then: export https_proxy='http://webproxy.eqiad.wmnet:8080' [17:24:47] and: conda install -c conda-forge apache-airflow[hdfs,kerberos] [17:24:57] this works fine [17:24:59] kay [17:25:13] but when I try to: airflow db init [17:25:25] I see lots of errors of lib compatibility [17:25:47] oh [17:25:51] It seems to me that the versions of the libraries in conda-forge are not the latest [17:25:57] that might be true... [17:26:06] mforns: can you just use the existent airflow env? [17:26:10] and/or process that we use to make it? [17:26:14] before trying conda, I tried pip, same approach, and it worked well [17:26:17] if you were doing this on e.g. an-test-client or an-launcher [17:26:23] you could just use the existing conda env [17:26:31] yes! [17:26:39] instead of anaconda-wmf [17:26:44] without having to install anything [17:26:59] but then we force people to use a machine that has that, is that OK? [17:27:05] v [17:27:13] welll we can install it elsewhere if we need to [17:27:21] k! [17:27:22] there is debian package they can install if they run debian anywhere [17:27:24] but [17:27:34] here is how the env is created [17:27:35] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/airflow/ [17:28:30] speaking of lib versions [17:28:31] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/airflow/+/refs/heads/debian/build/profile.sh#163 [17:28:35] OK, I will try to just activate the airflow env in an-launcher1002! [17:29:01] that gets files like [17:29:02] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/airflow/+/refs/heads/debian/pip-constraints.txt [17:29:07] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/airflow/+/refs/heads/debian/pip-requirements.txt [17:29:13] where everythign is specified exactly [17:29:22] ok, makes sense [17:29:53] we use pip here to get them because of what you say: conda deps aren't what airflow upstream uses to build [17:30:01] aha [17:30:05] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/airflow/+/refs/heads/debian#conda-vs-pip-packages [17:30:27] yep [17:30:45] i betcah if you wanted [17:30:46] 10Analytics: Data drifts between superset_production on an-coord1001 and db1108 - https://phabricator.wikimedia.org/T279440 (10BTullis) To close this off, I performed a validation using `pt-table-checksum` in T295551 and I've found that it's the values in `INFORMATION_SCHEMA.TABLES` that are wrong. From here: ht... [17:30:51] you could clone that deb git repo [17:31:04] run source ./build/profile.sh [17:31:22] and then just run [17:31:23] build_conda_prep_env [17:31:45] and you'd have a working conda env [17:32:36] that would be, for when the airflow env is not in the machine right? [17:32:41] yes [17:33:20] and/or we coudl install the airflow debian package on other machines, like stat boxes [17:33:28] without any running managed airflow instnaces [17:33:39] OK, I'll try first with the existing env in an-launcher and then when all works, will see if I can do that! [17:33:58] yes, also an option [17:34:19] kay [17:34:23] thanks a lot!! [17:34:27] yup! [17:35:15] 10Analytics: Data drifts between superset_production on an-coord1001 and db1108 - https://phabricator.wikimedia.org/T279440 (10Ottomata) Phewf!!!! [17:36:09] btullis: that is good good news [17:37:16] (EventgateLoggingExternalLatency) firing: (2) Elevated latency for GET events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org [17:37:25] Yes, much happier about that DB situation now, plus the backuips. [17:38:30] Ah, this alert --^ is the eventgate latency thing that I mentioned. I was expecting this: https://phabricator.wikimedia.org/T294911 [17:40:10] 10Analytics: Data drifts between superset_production on an-coord1001 and db1108 - https://phabricator.wikimedia.org/T279440 (10elukey) Nice! [17:42:15] 10Analytics-Radar, 10Observability-Alerting, 10Patch-For-Review: Apparent latency warning in 90th centile of eventgate-logging-external - https://phabricator.wikimedia.org/T294911 (10BTullis) I have merged the change to the alerts repo and the alert did fire as expected. {F34743545} I can acknowledge it for... [17:43:07] nice btullis [17:43:54] Thanks. It looks like I can only ack it for 15 minutes in Karma. Still... It shows that the system works :-) [17:43:59] :) [17:47:15] (EventgateLoggingExternalLatency) firing: (2) Elevated latency for GET events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org [18:24:13] 10Analytics, 10Event-Platform, 10Observability-Alerting: Apparent latency warning in 90th centile of eventgate-logging-external - https://phabricator.wikimedia.org/T294911 (10colewhite) IIUC the other half of this task is to determine whether the alerting thresholds are appropriate. Given EventGate produces... [19:10:47] mforns: how goes? [19:11:06] slowwly, but moving :] [19:12:12] seems to be wooorking so far :] [19:12:56] that's good! [19:25:24] mforns: okay, what else can I do atm? [21:51:24] 10Analytics, 10Data-Engineering: Analytics-test-hadoop Spark3 package upgrade - https://phabricator.wikimedia.org/T291465 (10Ottomata) [21:51:54] 10Analytics, 10Data-Engineering: Analytics-test-hadoop Spark3 package upgrade - https://phabricator.wikimedia.org/T291465 (10Ottomata) [21:51:56] 10Analytics, 10Data-Engineering, 10Epic: Upgrade analytics-hadoop to Spark 3 + scala 2.12 - https://phabricator.wikimedia.org/T291464 (10Ottomata) [22:26:39] 10Analytics-Data-Quality, 10Analytics-EventLogging, 10Analytics-Radar, 10Product-Analytics, and 3 others: WikiEditor records all edits as platform = desktop in EventLogging - https://phabricator.wikimedia.org/T249944 (10Jdlrobson) +1 to having a better definition if "isMobile" based on user agent string in... [22:56:48] 10Analytics, 10Product-Analytics, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): Lower sampling rate for MobileWebUIClickTracking on English Wikipedia before wmf8 is on English Wikipedia - https://phabricator.wikimedia.org/T295432 (10Jdlrobson) a:03nray Currently we only see 4 events a second on this schema...