[01:27:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:27:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [09:43:59] 06Data-Engineering, 06Traffic: Request for a new request dataset for caching research - https://phabricator.wikimedia.org/T401331#11218567 (10GGoncalves-WMF) Hi, just a quick update after my chat with Sukhbir. We should do this, not only for the value of the dataset itself, but also because it will be an excel... [10:02:46] 06Data-Engineering: Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11218616 (10Zabe) Please note that this will also cause `archive.ar_sha1` to be dropped, as the archive table contains the deleted revisions if you also use this column. [10:53:50] 06Data-Engineering, 06Traffic: Reduce noise from duplicate sequence-gap alerts on HaProxy-webrequests - https://phabricator.wikimedia.org/T401383#11218788 (10Clement_Goubert) These changes seem to cause some puppet CI failures due to the `lua` `utf8` module not being present in the test environment: ` 12:45:52... [12:36:07] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06Product Safety and Integrity (Sprint Apfel Strudel (September 29 - October 17)), and 2 others: AbuseFilter abuse_filter_log table: Store IP addresses as hex values - https://phabricator.wikimedia.org/T395612#11219060 (10OKryva-WMF) [12:55:55] 06Data-Engineering, 06Data-Persistence, 10GlobalBlocking, 06Product Safety and Integrity, and 2 others: globalblocks table: SQL in extension and production have different type for gb_address - https://phabricator.wikimedia.org/T395669#11219198 (10OKryva-WMF) [13:09:22] 06Data-Engineering: Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11219355 (10mforns) [13:10:41] 06Data-Engineering: Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11219363 (10mforns) [13:50:43] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work, 10Event-Platform, 13Patch-For-Review: Build Flink docker image on bookworm - https://phabricator.wikimedia.org/T400600#11219514 (10Gehel) [13:50:51] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Do performance testing of a big Hadoop Table hosted by Ceph - https://phabricator.wikimedia.org/T381416#11219530 (10Gehel) [13:50:59] 06Data-Engineering, 06Data-Engineering-Radar, 06cloud-services-team, 06Data-Persistence, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11219522 (10Gehel) [13:51:25] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), and 2 others: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#11219547 (10Gehel) [13:53:24] 06Data-Engineering, 06Data-Engineering-Radar, 06Discovery-Search, 06Infrastructure-Foundations, and 3 others: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#11219594 (10Gehel) [13:53:37] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Move the dumps_v1 DAGs from the Airflow test_k8s instance to the main instance - https://phabricator.wikimedia.org/T404084#11219605 (10Gehel) [13:54:09] 06Data-Engineering-Radar, 10Research-engineering, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Add analytics-research user to stat boxes - https://phabricator.wikimedia.org/T403207#11219619 (10Gehel) [13:54:23] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Requesting Kerberos access for sd - https://phabricator.wikimedia.org/T405219#11219613 (10Gehel) [13:54:42] 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 4 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#11219629 (10Gehel) [13:56:50] 06Data-Engineering, 10Technical-blog-posts, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Write a blog post about the recent Airflow migration to Kubernetes - https://phabricator.wikimedia.org/T393603#11219670 (10Gehel) [13:56:57] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: Blunderbuss: Move Hadoop/HDFS XML configuration into Helm deployment chart - https://phabricator.wikimedia.org/T402323#11219672 (10Gehel) [13:58:33] 06Data-Engineering, 06Data-Platform-SRE (2025.09.26 - 2025.10.17), 07Essential-Work: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#11219705 (10Gehel) [14:00:34] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product Safety and Integrity, 06serviceops, 06SRE: Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset - https://phabricator.wikimedia.org/T303464#11219764 (10OKryva-WMF) [14:28:02] 06Data-Engineering, 06Data-Persistence, 10MediaWiki-Core-Revision-backend, 10Wikidata, and 3 others: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#11220037 (10JTweed-WMF) [14:30:19] 06Data-Engineering, 06Data-Persistence, 10MediaWiki-Core-Revision-backend, 10Wikidata, and 3 others: Rethink rev_sha1 field - https://phabricator.wikimedia.org/T389026#11220044 (10JTweed-WMF) Adding the Wikidata tag as its explicitly mentioned above. Please try to add the Wikidata umbrella tag if a ticket... [14:46:15] 06Data-Engineering, 10AQS2.0, 07Documentation: Adding a AQS 2.0 endpoint guide - https://phabricator.wikimedia.org/T356748#11220100 (10TBurmeister) [15:07:21] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10service-utils, 10Event-Platform, 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Migrate and re-deploy eventgate-wikimedia using new service-utils - https://phabricator.wikimedia.org/T403169#11220227 (10tchin) Fully deployed to... [15:24:39] 06Data-Engineering, 06Data-Engineering-Icebox, 06serviceops, 06SRE, 06Trust-and-Safety: Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset - https://phabricator.wikimedia.org/T303464#11220338 (10OKryva-WMF) [15:31:48] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11220364 (10mforns) [16:09:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [16:11:17] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10service-utils, 10Event-Platform, 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Migrate and re-deploy eventgate-wikimedia using new service-utils - https://phabricator.wikimedia.org/T403169#11220495 (10tchin) Deployed to `even... [16:19:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [16:30:51] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11220569 (10Dzahn) [16:43:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [16:48:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [17:09:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [17:19:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [17:42:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [17:47:17] RESOLVED: [2x] EventgateLatency: Elevated latency for POST events on eventgate-analytics in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [17:50:43] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update Gobblin documentation - https://phabricator.wikimedia.org/T405780 (10amastilovic) 03NEW [18:09:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [18:14:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [18:16:08] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11220918 (10Dzahn) Thank you as well! All looks good. Only pending on the approval. Please note the people handling these request... [18:24:01] 06Data-Engineering, 06Traffic: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783 (10CDanis) 03NEW [18:24:47] 06Data-Engineering, 06Traffic: improved x-analytics data on Edge Uniques status - https://phabricator.wikimedia.org/T405783#11220967 (10CDanis) See also T402994, T391411 [18:25:12] (03PS5) 10Snwachukwu: CREATE HQL SCRIPT TO UPDATE SCOOP WIKI LIST DATA FILE. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1191398 [18:27:52] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026#11220997 (10tchin) In the logs I spotted another offender ` {"@timest... [18:42:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [18:43:48] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Investigate reasons for remaining inconsistencies - https://phabricator.wikimedia.org/T385112#11221038 (10xcollazo) >>! In T385112#11216686, @xcollazo wrote: > Currently re-running the september full recon... [18:52:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [19:09:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [19:14:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [19:15:41] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10service-utils, 10Event-Platform, 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), 13Patch-For-Review: Migrate and re-deploy eventgate-wikimedia using new service-utils - https://phabricator.wikimedia.org/T403169#11221108 (10tchin) Hmmm ok everythin... [19:34:36] (03CR) 10Mforns: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [19:42:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [19:44:43] (03CR) 10Snwachukwu: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [19:47:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [19:54:01] (03PS8) 10Snwachukwu: 1.Add a closed flag to the project namespace map dataset 2. Add a whether to sqoop flag by checking if wikidb exists in cloud replica. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1125184 (https://phabricator.wikimedia.org/T365203) (owner: 10Milimetric) [20:10:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [20:15:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [20:42:55] 06Data-Engineering (Q1 FY25/26 July 1st - September 30th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11221275 (10nshahquinn-wmf) FYI, assuming that the May backfill finished at the end of yesterday and using @mforns's estimate that a month takes... [20:43:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [20:48:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [21:12:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [21:17:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [21:42:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [21:47:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [22:12:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [22:17:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [22:42:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [22:47:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [23:10:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [23:15:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [23:42:17] FIRING: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency [23:47:17] RESOLVED: EventgateLatency: Elevated latency for POST events on eventgate-analytics-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLatency