[00:03:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [02:16:16] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [02:21:16] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [02:21:45] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [02:26:45] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [03:44:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [03:49:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [04:29:16] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [04:39:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [06:52:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [06:57:16] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [07:08:16] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [07:13:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [07:24:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [07:44:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [08:14:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [08:19:16] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [08:37:49] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [08:40:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [08:40:34] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Superset: Help with data that's not appearing on charts - https://phabricator.wikimedia.org/T301895 (10BTullis) It sounds like upstream are working on a CLI too to migrate charts and dashboards away from the deprecated types, but it isn'... [08:50:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [08:55:16] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [08:57:42] (03PS3) 10Snwachukwu: Create Hive Query to generate Wikidata CoEditors metrics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/780749 (https://phabricator.wikimedia.org/T306177) [09:35:06] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) I suggest that we increase the `connect_timeout` value for the `local_service` clust... [09:59:27] 10Data-Engineering-Radar, 10Growth-Team, 10MediaWiki-extensions-GuidedTour, 10Patch-For-Review: Finish decommissioning the legacy GuidedTour schemas - https://phabricator.wikimedia.org/T303712 (10kostajh) >>! In T303712#7803247, @kostajh wrote: >>>! In T303712#7783123, @phuedx wrote: >>>>! In T303712#77810... [10:47:04] 10Data-Engineering, 10Data-Engineering-Kanban, 10MediaWiki-extensions-EventLogging: Generate $wgEventLoggingSchemas from $wgEventStreams - https://phabricator.wikimedia.org/T303602 (10phuedx) >>! In T303602#7903589, @Ottomata wrote: > I think doing having EventLogging code do this with filtering via EventStr... [11:47:41] 10Data-Engineering, 10Data-Catalog: Integrate Superset with DataHub - https://phabricator.wikimedia.org/T306903 (10BTullis) I did a little work on this, but it is a little complicated by the way that we have authentication configured for Superset. With superset we set `AUTH_TYPE = REMOTE_USER` [[https://githu... [12:09:46] (03PS4) 10Snwachukwu: Create Hive Query to generate Wikidata CoEditors metrics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/780749 (https://phabricator.wikimedia.org/T306177) [13:04:26] (03CR) 10Snwachukwu: Create Hive Query to generate Wikidata CoEditors metrics (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/780749 (https://phabricator.wikimedia.org/T306177) (owner: 10Snwachukwu) [13:08:20] (03CR) 10Joal: Create Hive Query to generate Wikidata CoEditors metrics (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/780749 (https://phabricator.wikimedia.org/T306177) (owner: 10Snwachukwu) [13:18:19] (03PS5) 10Snwachukwu: Create Hive Query to generate Wikidata CoEditors metrics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/780749 (https://phabricator.wikimedia.org/T306177) [13:21:10] (03CR) 10Snwachukwu: Create Hive Query to generate Wikidata CoEditors metrics (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/780749 (https://phabricator.wikimedia.org/T306177) (owner: 10Snwachukwu) [13:21:13] 10Data-Engineering, 10Data-Catalog, 10SRE, 10serviceops, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10Ottomata) > For the duration of the upgrade plus some safety time windows before/after the upgrade, traffic will always be served from codfw and thus have the w... [13:21:58] (03CR) 10Joal: [V: 03+2 C: 03+2] "Good for me! Thank you Sandra :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/780749 (https://phabricator.wikimedia.org/T306177) (owner: 10Snwachukwu) [13:25:58] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10Ottomata) If that fixes it, at the very least it will give us some information about what is... [13:26:57] 10Data-Engineering, 10Data-Engineering-Kanban, 10MediaWiki-extensions-EventLogging: Generate $wgEventLoggingSchemas from $wgEventStreams - https://phabricator.wikimedia.org/T303602 (10Ottomata) Wouldn't it be nice of EventStreamConfig was a GraphQL API? :p [13:28:02] 10Data-Engineering, 10Data-Engineering-Kanban, 10MediaWiki-extensions-EventLogging: Generate $wgEventLoggingSchemas from $wgEventStreams - https://phabricator.wikimedia.org/T303602 (10Ottomata) > Sure! We'll have to patch StreamConfig::matchesSettings to a recursive, associative array intersection, which sho... [13:28:47] (03CR) 10Ottomata: [C: 03+1] "Thank you Joseph!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/789797 (https://phabricator.wikimedia.org/T307779) (owner: 10Joal) [13:49:08] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) There's some discussion ongoing in [[https://wm-bot.wmflabs.org/libera_logs/%23wikim... [13:59:42] 10Data-Engineering-Kanban, 10Airflow: Migrate the Clickstream jobs to Airflow - https://phabricator.wikimedia.org/T305843 (10Antoine_Quhen) https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/52 [14:05:10] (03PS1) 10Aqu: Migrate clickstream/monthly to Airflow [analytics/refinery] - 10https://gerrit.wikimedia.org/r/790373 (https://phabricator.wikimedia.org/T305843) [14:06:57] (03PS2) 10Aqu: Migrate clickstream/monthly to Airflow [analytics/refinery] - 10https://gerrit.wikimedia.org/r/790373 (https://phabricator.wikimedia.org/T305843) [14:23:12] (03CR) 10Vivian Rook: [C: 03+2] separate stop and submit buttons [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/789196 (https://phabricator.wikimedia.org/T290146) (owner: 10Vivian Rook) [14:28:17] (03Merged) 10jenkins-bot: separate stop and submit buttons [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/789196 (https://phabricator.wikimedia.org/T290146) (owner: 10Vivian Rook) [15:06:31] !log killed ‘apis-coord' oozie job and started corresponding airflow job ‘apis_metrics_to_graphite’ [15:06:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:34:02] (03Abandoned) 10Ottomata: Repository structure adjusted. [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/647752 (owner: 10Jason Linehan) [15:52:21] (03Abandoned) 10Vivian Rook: Remove stop query function [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/788719 (https://phabricator.wikimedia.org/T290146) (owner: 10Vivian Rook) [15:54:30] 10Data-Engineering, 10Airflow: SparkSubmitOperator should make it easier to use conda dist envs - https://phabricator.wikimedia.org/T307937 (10Ottomata) [15:55:18] 10Data-Engineering, 10Airflow: Use airflow to load cassandra - https://phabricator.wikimedia.org/T306962 (10JAllemandou) [15:58:06] 10Data-Engineering, 10Airflow: SparkSubmitOperator should make it easier to use conda dist envs - https://phabricator.wikimedia.org/T307937 (10Ottomata) The MR is ready to merge and use for production envs now. However, development environments created with `run_dev_instance.sh` will not work with this as is.... [16:00:15] hi all. quick question: is https://phabricator.wikimedia.org/T300164 captured somewhere on wikitech? with the team we were looking at https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Pageviews#Events_and_known_problems_since_2015-05-01 for it but maybe this is not the right place? [16:00:59] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10akosiaris) Looking at https://grafana-rw.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&viewPane... [17:09:22] 10Analytics-Kanban, 10Event-Platform, 10MediaWiki-extensions-EventLogging, 10Goal, and 2 others: Modern Event Platform - https://phabricator.wikimedia.org/T185233 (10Ottomata) [17:19:36] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform, 10Epic: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP. - https://phabricator.wikimedia.org/T307944 (10Ottomata) [17:23:20] joal: can you answer leilas q above? ^^^ [18:07:06] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform, 10Epic: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP. - https://phabricator.wikimedia.org/T307944 (10Ottomata) [18:13:37] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform, 10Epic: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP - https://phabricator.wikimedia.org/T307944 (10Ottomata) [18:15:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [18:20:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [18:20:37] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform, 10Epic: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP - https://phabricator.wikimedia.org/T307944 (10Ottomata) [18:21:43] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP - https://phabricator.wikimedia.org/T307944 (10Ottomata) [18:24:20] 10Analytics-Kanban, 10Data-Engineering, 10Event-Platform, 10MediaWiki-extensions-EventLogging, and 3 others: Modern Event Platform - https://phabricator.wikimedia.org/T185233 (10lbowmaker) [18:27:43] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP - https://phabricator.wikimedia.org/T307944 (10Ottomata) [18:36:59] thanks ottomata for the ping - Hi leila - The problem has not been documented on wikitech - it shall be! I'm making qa note and will follow up on this [18:37:54] ottomata: wanna chat for a minute? [18:52:03] joal: thanks a lot. ( mgerlach fyi ) [18:58:08] I’m off to a doctor appointment [19:19:12] 10Data-Engineering, 10Product-Analytics (Kanban): Make analytics-product the owner of canonical_data - https://phabricator.wikimedia.org/T307749 (10mpopov) Thank you @JAllemandou! [19:46:36] joal: hi jusut saw ping [19:46:38] here if you still are! [19:48:26] 10Data-Engineering, 10Event-Platform, 10Generated Data Platform: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP - https://phabricator.wikimedia.org/T307944 (10Ottomata) [21:18:53] (03CR) 10Jdlrobson: [C: 03+1] DesktopUIActions/MobileUIActions: add pageToken field [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/789324 (https://phabricator.wikimedia.org/T307640) (owner: 10DLynch) [22:19:16] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [22:24:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [23:15:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [23:20:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency