[00:01:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [00:31:54] RECOVERY - Check unit status of monitor_refine_eventlogging_legacy on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_legacy https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [01:13:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [01:18:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [05:45:02] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Persistence (Consultation), 10Data-Services, and 2 others: Toolforge db: View 'fiwiki_p.flaggedrevs' references invalid table/column/rights to use them - https://phabricator.wikimedia.org/T302233 (10Marostegui) Maybe it was done manually before - I just... [07:54:26] (03Abandoned) 10Aqu: Fix Airflow job session length daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/771730 (https://phabricator.wikimedia.org/T300029) (owner: 10Aqu) [08:53:01] 10Data-Engineering, 10Project-Admins: Archive Analytics tag - https://phabricator.wikimedia.org/T298671 (10Aklapper) @odimitrijevic: Any thoughts on the last two comments? Thanks! [09:39:42] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Research, discuss and decide on DAG/task dependencies VS. success/failure files (Oozie style) - https://phabricator.wikimedia.org/T301568 (10Antoine_Quhen) Some good points about using DAG dependencies are : - When 1 job triggers another one... [09:52:43] 10Analytics-Radar, 10DBA: mariadb on dbstore hosts, and specifically dbstore1004, possible memory leaking - https://phabricator.wikimedia.org/T270112 (10Marostegui) 05Open→03Declined dbstore1004 is no more, and for dbstore1007 we have {T290841} dbstore1003 and dbstore1005 look fine, closing this. [09:54:14] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Persistence (Consultation): dbstore1007 is swapping heavilly, potentially soon killing mysql services due to OOM error - https://phabricator.wikimedia.org/T290841 (10Marostegui) [10:01:10] (03CR) 10Emil Chetty: [C: 03+1] "Lets do this." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [11:04:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [11:09:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [12:03:36] 10Analytics, 10Data-Engineering, 10SRE: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373 (10Ottomata) This would be easier if {T276972} was done, but it doesn't look like there's enthusiasm for it. I'd love to be able to automate ingestion fro... [12:21:34] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Research, discuss and decide on DAG/task dependencies VS. success/failure files (Oozie style) - https://phabricator.wikimedia.org/T301568 (10Ottomata) > Should we use them always, by default? I'd say let's not. Let's add them as needed. +1.... [13:24:59] 10Analytics, 10Beta-Cluster-Infrastructure: Clarify how users can opt-out of intake-analytics - https://phabricator.wikimedia.org/T304426 (10AlexisJazz) [13:27:25] 10Analytics: Clarify how users can opt-out of intake-analytics - https://phabricator.wikimedia.org/T304426 (10hashar) [13:36:20] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Persistence (Consultation): dbstore1007 is swapping heavilly, potentially soon killing mysql services due to OOM error - https://phabricator.wikimedia.org/T290841 (10Ladsgroup) At this point in my career, I came to zen and accept mem... [13:37:08] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Push Gobblin import metrics to Prometheus and add alerts on some critical imports - https://phabricator.wikimedia.org/T286503 (10Ottomata) a:05JAllemandou→03Ottomata [13:41:07] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Metrics-Platform, 10Browser-Support-Microsoft-Edge: Problem with delay caused by intake-analytics.wikimedia.org - https://phabricator.wikimedia.org/T295427 (10AlexisJazz) For me it's just annoying: T304426 but apparently this causes problems on product... [13:59:09] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): WikipediaPortal Event Platform Migration - https://phabricator.wikimedia.org/T282012 (10EYener) Thank you @Jdrewniak and @Ottomata! This is a schema that I've worked with before to have th... [14:10:34] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): WikipediaPortal Event Platform Migration - https://phabricator.wikimedia.org/T282012 (10Ottomata) This migration is being done in a backwards compatible way, so the same system is sanitizi... [14:13:01] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): WikipediaPortal Event Platform Migration - https://phabricator.wikimedia.org/T282012 (10EYener) Great @Ottomata yes it is in that yaml file! [14:15:17] 10Data-Engineering, 10Code-Health-Objective, 10Epic, 10Platform Engineering Roadmap, and 2 others: [DISCUSS]: Problem details for HTTP APIs (rfc7807) - https://phabricator.wikimedia.org/T302536 (10nnikkhoui) New problem type proposal: - Do not include "type" and "instance" and don't validate "type" and "in... [14:36:45] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): WikipediaPortal Event Platform Migration - https://phabricator.wikimedia.org/T282012 (10ovasileva) a:05Jdrewniak→03None [14:56:01] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Send some existing Gobblin metrics to prometheus - https://phabricator.wikimedia.org/T294420 (10Ottomata) Calling this done. The dashboard works. [14:56:23] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Update refinery gobblin jars to use new gobblin-wmf jars and update puppet gobblin jobs - https://phabricator.wikimedia.org/T297939 (10Ottomata) Done. updated docs at https://wikitech.wikimedia.org/wiki/Analytics/System... [14:56:59] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Push Gobblin import metrics to Prometheus and add alerts on some critical imports - https://phabricator.wikimedia.org/T286503 (10Ottomata) Done! Alerts have been merged. [15:22:13] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10Patch-For-Review, and 2 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Milimetric) As I understand, the patch just adds the columns but is there now a plan to migrate the dat... [15:28:41] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10Patch-For-Review, and 2 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Zabe) >>! In T233004#7796847, @Milimetric wrote: > As I understand, the patch just adds the columns but... [15:29:46] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering-Radar, 10Product-Analytics, and 4 others: Wikistats pageview data missing counts for Mobile App pageviews on Commons, going back to 2020-11 - https://phabricator.wikimedia.org/T299439 (10Milimetric) @mpopov ooh, if you talk about that, maybe also inclu... [15:55:10] 10Data-Engineering, 10Beta-Cluster-Infrastructure, 10Event-Platform: Upgrade event platform related VMs in deployment-prep to Debian bullsye (or buster) - https://phabricator.wikimedia.org/T304433 (10Ottomata) [16:07:25] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Set up karapace instance for datahub - https://phabricator.wikimedia.org/T301562 (10razzi) [16:14:56] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Low Risk Oozie Migration: APIs - https://phabricator.wikimedia.org/T300028 (10Snwachukwu) [16:15:06] 10Analytics, 10Data-Engineering, 10SRE: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373 (10jbond) p:05Triage→03Medium [16:17:00] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of accraze - https://phabricator.wikimedia.org/T303928 (10BTullis) 05Open→03Resolved [16:17:24] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of ppchelko - https://phabricator.wikimedia.org/T303924 (10BTullis) 05Open→03Resolved [16:17:37] 10Data-Engineering, 10Data-Engineering-Kanban: Set SparkmaxPartitionBytes to 256MB - https://phabricator.wikimedia.org/T300299 (10Ottomata) 05Open→03Resolved [16:17:43] 10Analytics-Clusters, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Set hive.warehouse.subdir.inherit.perms to false - https://phabricator.wikimedia.org/T291664 (10Ottomata) 05Open→03Resolved [16:17:45] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Conda's CPPFLAGS may not be correct when pip installing a package that needs c/cpp compilation - https://phabricator.wikimedia.org/T292699 (10Ottomata) 05Open→03Resolved [16:17:47] 10Analytics, 10Analytics-Kanban, 10Data-Engineering-Kanban, 10wmfdata-python, 10Product-Analytics (Kanban): wmfdata-python's Hive query output includes logspam - https://phabricator.wikimedia.org/T275233 (10Ottomata) [16:18:20] 10Data-Engineering-Kanban, 10Airflow: Tooling for Deploying Conda Environments - https://phabricator.wikimedia.org/T296543 (10Ottomata) 05Open→03Resolved [16:18:33] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Airflow-triggered Spark-jobs produce hdfs-files belonging to the wrong hdfs-user-group - https://phabricator.wikimedia.org/T303201 (10Ottomata) 05Open→03Resolved [16:18:42] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Use arrow_hdfs:// fsspec protocol in workflow_utils artifact syncing - https://phabricator.wikimedia.org/T300876 (10Ottomata) 05Open→03Resolved [16:19:11] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Standardize the stats system user uid - https://phabricator.wikimedia.org/T291384 (10Ottomata) 05Open→03Resolved [16:19:22] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform: Users should run explicit commands to materialize schema versions, rather than using magic git hooks - https://phabricator.wikimedia.org/T290074 (10Ottomata) 05Open→03Resolved [16:19:28] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Un-fork analytics/gobblin - https://phabricator.wikimedia.org/T292396 (10Ottomata) [16:19:30] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Update refinery gobblin jars to use new gobblin-wmf jars and update puppet gobblin jobs - https://phabricator.wikimedia.org/T297939 (10Ottomata) 05Open→03Resolved [16:19:34] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Send some existing Gobblin metrics to prometheus - https://phabricator.wikimedia.org/T294420 (10Ottomata) 05Open→03Resolved [16:19:36] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Push Gobblin import metrics to Prometheus and add alerts on some critical imports - https://phabricator.wikimedia.org/T286503 (10Ottomata) [16:19:40] 10Data-Engineering-Kanban, 10SRE, 10SRE-Access-Requests: Requesting access to DataEngineering Team Resources for NOkafor - https://phabricator.wikimedia.org/T303516 (10BTullis) 05Open→03Resolved [16:20:11] 10Data-Engineering-Kanban, 10LDAP-Access-Requests: Grant Access to LDAP wmf group for NOkafor - https://phabricator.wikimedia.org/T303512 (10BTullis) 05Open→03Resolved [16:20:13] 10Data-Engineering-Kanban, 10SRE, 10SRE-Access-Requests: Requesting access to DataEngineering Team Resources for NOkafor - https://phabricator.wikimedia.org/T303516 (10BTullis) [16:20:33] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: The analytics-meta's binlogs are full of airflow sla-related UPDATE statements - https://phabricator.wikimedia.org/T302777 (10BTullis) 05Open→03Resolved [16:21:06] 10Analytics-Clusters, 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, and 2 others: Switch over the Cassandra AQS cluster to the new hosts - https://phabricator.wikimedia.org/T297803 (10BTullis) 05Open→03Resolved [16:21:12] 10Analytics-Clusters, 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, and 2 others: Cassandra3 migration for Analytics AQS - https://phabricator.wikimedia.org/T249755 (10BTullis) [16:23:35] o/ where can I configure the retention of some table in "events"? (I thought the data there would be kept for ever but apparently not) [16:26:02] ah no my bad even mediawiki.revision-create is being clean up [16:26:38] latest event is december 2021 [16:29:49] 10Data-Engineering, 10Data-Engineering-Kanban: Resume Webrequest Data Purge Job - https://phabricator.wikimedia.org/T303977 (10Milimetric) 05Open→03Resolved [16:30:11] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics: 22 small wikis missing from the mediawiki_history dataset - https://phabricator.wikimedia.org/T299548 (10Milimetric) 05Open→03Resolved [16:37:00] dcausse: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Event_Sanitization [16:37:36] see also the event_sanitized database for old data [16:38:43] ottomata: oh did not know that there was another db, thanks! [17:04:21] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): WikipediaPortal Event Platform Migration - https://phabricator.wikimedia.org/T282012 (10LGoto) a:03Jdrewniak [17:07:22] I need to reboot the servers with GPUs in them to pick up a new kernel version. I've sent an email to analytics-announce about stat1005 and stat1008 suggesting that I do this work on Friday morning at 10:00 UTC. [17:08:58] There are also six Hadoop workers which need rebooting: an-worker[1096-1101] [17:09:41] Do I need to give anyone a particular heads-up about these reboots, since they also have the GPUs? [17:15:22] btullis: sending a message to research and search should be enough [17:15:30] btullis: or at least I think :) [17:15:39] btullis: obviously I'm forgetting ML team [17:15:57] joal: Many thanks. Will do so now. [17:27:04] joal DID you see the new gobblin dashboard OR WUT?! [17:27:20] https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin [17:27:52] ottomata: I quickly scanned it yesterday - in meeting now, will ping you when done :) [17:28:03] okay! :) [17:28:23] ottomata: I refreshed the page, and it SO GREAT! I owe you :) [17:28:44] ottomata: It's fantastic. [17:30:24] thank you thank you i was fishing for compliments [17:31:28] ottomata: I have questions - will ping ou when out of meetings [17:31:33] mkkay! [17:31:43] ottomata: thank you again - this is awesome <3 [17:31:55] haha yw! i got to make the dashboard which was the fun part [17:32:22] ottomata: 🏅 (The details panel gets a bit heavy when all of the topics are selected - any way to lighten the load I wonder?) [17:32:49] yeah, you can filter on specific topics up above [17:32:57] or you can view hte whole panel all at once [17:33:08] i originally made the dash while looking at event_default_test [17:33:19] but ya i agree it i sbusy [17:33:25] if you have any ideas feel free to try em! [17:40:07] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Create debian package of karapace - https://phabricator.wikimedia.org/T301565 (10razzi) It turns out the .deb, as I created it, overwrites the system python, which we don't want; this caused the following error: ` Mar 22 17... [17:40:54] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Create debian package of karapace - https://phabricator.wikimedia.org/T301565 (10razzi) (useful command: view the files that were installed by a deb) ` dpkg-deb -c /var/cache/apt/archives/karapace_2.1.3-py3.7-1_amd64.deb ` [17:47:48] ottomata: Nothing comes to mind at the moment, but I'll think about it. Great work both of you. :-) [18:05:41] (03PS1) 10Sharvaniharan: New schema for measuring article screen interactions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772910 [18:06:08] (03CR) 10jerkins-bot: [V: 04-1] New schema for measuring article screen interactions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772910 (owner: 10Sharvaniharan) [18:07:16] (03PS2) 10Sharvaniharan: New schema for measuring article screen interactions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772910 [18:09:38] (03PS3) 10Sharvaniharan: New schema for measuring article screen interactions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772910 [18:11:11] (03PS4) 10Sharvaniharan: New schema for measuring article screen interactions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772910 (https://phabricator.wikimedia.org/T304335) [18:19:54] ottomata: batcave? [18:20:13] joal ok ii have 5 mins! [18:20:20] ack [18:20:29] btullis: the .deb that I built overwrote the system python so I'm going to reimage karapace1001. ottomata helped me fix it and the new deb is uploaded to apt (thanks ottomata !!) [18:48:18] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Research, discuss and decide on DAG/task dependencies VS. success/failure files (Oozie style) - https://phabricator.wikimedia.org/T301568 (10JAllemandou) I like the idea of not relying on Airflow for jobs dependency as this would put us in a... [18:59:25] 10Data-Engineering, 10Code-Health-Objective, 10Epic, 10Platform Engineering Roadmap, and 2 others: [DISCUSS]: Problem details for HTTP APIs (rfc7807) - https://phabricator.wikimedia.org/T302536 (10Eevans) >>! In T302536#7796646, @nnikkhoui wrote: > New problem type proposal below. > Changes include: > - `d... [19:11:34] yw! [19:12:44] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Research, discuss and decide on DAG/task dependencies VS. success/failure files (Oozie style) - https://phabricator.wikimedia.org/T301568 (10Ottomata) Ya, which is maybe a reason to do both! Especially for the final output of a DAG. Some jo... [19:28:06] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Set up karapace instance for datahub - https://phabricator.wikimedia.org/T301562 (10razzi) The server is running; hostname is karapace1001.eqiad.wmnet and port is 8081. ` curl karapace1001.eqiad.wmnet:8081 {} ` Looks like the monitoring is work... [19:31:49] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Set up karapace instance for datahub - https://phabricator.wikimedia.org/T301562 (10razzi) Currently this is connected to kafka-test1006.eqiad.wmnet. It is my understanding we will use the "jumbo" kafka cluster. The following netcat times out; we... [19:32:08] 10Data-Engineering, 10SRE: Create conda .deb and docker image - https://phabricator.wikimedia.org/T304450 (10Ottomata) [19:34:09] 10Data-Engineering, 10SRE: Create conda .deb and docker image - https://phabricator.wikimedia.org/T304450 (10Ottomata) @MoritzMuehlenhoff advice? Can I import [[ https://docs.conda.io/projects/conda/en/latest/user-guide/install/rpm-debian.html | conda's official .deb ]] into our apt repo, or would you prefer... [19:34:16] (03PS2) 10Kosta Harlan: Help panel: update schema to reflect new post-edit experience [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/769769 (https://phabricator.wikimedia.org/T301603) (owner: 10MewOphaswongse) [19:34:36] (03CR) 10Kosta Harlan: [C: 03+2] "PS2 removes trailing whitespace from a line" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/769769 (https://phabricator.wikimedia.org/T301603) (owner: 10MewOphaswongse) [19:35:21] (03Merged) 10jenkins-bot: Help panel: update schema to reflect new post-edit experience [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/769769 (https://phabricator.wikimedia.org/T301603) (owner: 10MewOphaswongse) [19:35:49] ottomata: yt? need help with dev instance :S [19:46:01] mforns: yes! [19:46:03] 15 mins [19:46:05] til meeting [19:46:10] in bc [19:47:01] ok, omw [20:17:00] 10Quarry: Add 'download as SQL' option - https://phabricator.wikimedia.org/T71191 (10rook) What form would this take? Would this be the equivalent of a mysqldump? If so how would metadata for table creation, int, double, date, etc be determined? [20:18:03] ottomata:, I found the thing... was a PEBCAK, sorry for bothering you [20:18:42] 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog, 10Epic: [EPIC] Count unique iOS & Android users precisely and in a privacy conscious manner that does not require opt in to send data - https://phabricator.wikimedia.org/T202664 (10LGoto) 05Open→03Dec... [20:19:38] mforns: identify PEBCAK! [20:19:39] :) [20:20:16] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [20:20:26] ottomata: you want me to explain the issue, or what pebcak is? xD [20:21:56] The issue was with the start_dat being too late for the @monthly job to start... blunder from my side [20:23:59] 10Analytics-Radar, 10Data-Engineering, 10Pageviews-API, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Large increase on 404s from the Wikipedia IOS app - https://phabricator.wikimedia.org/T203688 (10LGoto) 05Open→03Declined [20:25:16] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [20:25:58] 10Analytics, 10Pageviews-Anomaly: Manipulation of pageview statistics German Wikipedia - https://phabricator.wikimedia.org/T232992 (10LGoto) [20:38:24] the issue! ohhh nice [20:38:32] well we found that worker log server port thing [20:38:35] that probably needs fixed [21:47:38] (03PS1) 10Sharvaniharan: New schema for edit history screen interactions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/772934 [22:15:18] 10Analytics: Clarify how users can opt-out of intake-analytics - https://phabricator.wikimedia.org/T304426 (10AlexisJazz) [22:19:55] 10Analytics: Clarify how users can opt-out of intake-analytics - https://phabricator.wikimedia.org/T304426 (10AlexisJazz)