[00:44:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [00:54:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:09:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:17:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [01:22:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:19:09] (03CR) 10Snwachukwu: Update referer classification patterns (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198313 (https://phabricator.wikimedia.org/T406531) (owner: 10Joal) [05:51:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:01:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:03:44] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11305329 (10JAllemandou) The impact on DE seems small IMO. I'm not an expert but my uinderstanding is that some databases/tabl... [08:05:23] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10AbuseFilter, 06DBA, 07Schema-change-in-production: Drop the afl_ip column and the afl_ip_timestamp index from the abuse_filter_log table - https://phabricator.wikimedia.org/T407997#11305354 (10JAllemandou) No direct impact on us, we don't sqoop... [08:33:50] 06Data-Engineering, 06Movement-Insights: Improve referrer tracking/classification using `utm_source` URL parameter - https://phabricator.wikimedia.org/T408185 (10JAllemandou) 03NEW [08:35:16] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 07Essential-Work, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11305426 (10JAllemandou) Thank you everyone for c... [08:41:31] 06Data-Engineering: Requesting Kerberos access for Jmoore111 - https://phabricator.wikimedia.org/T408165#11305459 (10BTullis) [08:41:50] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Requesting Kerberos access for Jmoore111 - https://phabricator.wikimedia.org/T408165#11305461 (10BTullis) a:03BTullis [10:20:17] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Product-Analytics, 10Event-Platform: [Event Platform] Disable default collection of user agent for analytics streams - https://phabricator.wikimedia.org/T384964#11305660 (10JMonton-WMF) a:03JMonton-WMF [10:23:04] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 13Patch-For-Review: Requesting Kerberos access for Jmoore111 - https://phabricator.wikimedia.org/T408165#11305671 (10BTullis) p:05Triage→03High [11:05:24] 06Data-Engineering, 06Data-Platform-SRE: Increase the size of the Druid broker cache size from 2GB to 4GB - https://phabricator.wikimedia.org/T408189 (10BTullis) 03NEW [11:28:21] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Update MediaWiki Content History SLO draft for SRE review - https://phabricator.wikimedia.org/T401892#11305781 (10APizzata-WMF) What about the source data we ingest? I was thinking that a possible completeness measure should include also the source `e... [11:55:14] 06Data-Engineering: Requesting Kerberos access for fabfur - https://phabricator.wikimedia.org/T408193 (10Fabfur) 03NEW [12:01:18] 06Data-Engineering, 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Increase the size of the Druid broker cache size from 2GB to 4GB - https://phabricator.wikimedia.org/T408189#11305870 (10Gehel) p:05Triage→03Medium [12:38:57] 06Data-Engineering: Requesting Kerberos access for fabfur - https://phabricator.wikimedia.org/T408193#11306021 (10Fabfur) 05Open→03Resolved a:03Fabfur Password has been reset by @ssingh for me, thanks anyway [13:00:54] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 First analysis review - https://phabricator.wikimedia.org/T407103#11306082 (10Antoine_Quhen) [13:00:56] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 Import 2 new datasets of Spurs in a parquet table - https://phabricator.wikimedia.org/T407104#11306081 (10Antoine_Quhen) [13:01:12] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 Import 2 new datasets of Spurs in a parquet table - https://phabricator.wikimedia.org/T407104#11306084 (10Antoine_Quhen) 05In progress→03Resolved [14:10:37] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 Prepare dataset of IPs crossing Spur, legacy bot detection and HAProxy signals - https://phabricator.wikimedia.org/T408210 (10Antoine_Quhen) 03NEW [14:11:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 Prepare dataset of IPs crossing Spur, legacy bot detection and HAProxy signals - https://phabricator.wikimedia.org/T408210#11306344 (10Antoine_Quhen) 05Open→03In progress [14:11:24] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 Prepare dataset of IPs crossing Spur, legacy bot detection and HAProxy signals - https://phabricator.wikimedia.org/T408210#11306345 (10Antoine_Quhen) [14:11:25] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 First analysis review - https://phabricator.wikimedia.org/T407103#11306346 (10Antoine_Quhen) [14:12:13] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 SPUR bot detection analysis - https://phabricator.wikimedia.org/T407103#11306349 (10Antoine_Quhen) [14:41:06] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 Prepare dataset of IPs crossing Spur, legacy bot detection and HAProxy signals - https://phabricator.wikimedia.org/T408210#11306457 (10Antoine_Quhen) All September is imported. ` desc aqu.20251023_bot_ips_study ; col_name data_type... [14:41:23] 06Data-Engineering, 10Dumps-Generation, 06Wikibase Reuse Team, 10Wikidata, and 3 others: No Wikidata dumps for Week 40 of 2025 (recurring issue) - https://phabricator.wikimedia.org/T406429#11306458 (10BTullis) 05Open→03Resolved I'm going to resolve this now, as the wikibase dumps are back up to spe... [14:42:25] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): SDS 1.3.6 Import 2 new datasets of Spurs in a parquet table - https://phabricator.wikimedia.org/T407104#11306468 (10Antoine_Quhen) All Septembre is imported: ` desc wmf_traffic.spur_feed ; col_name data_type comment ip string NUL... [14:44:02] (03CR) 10Xcollazo: Update referer classification patterns (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198313 (https://phabricator.wikimedia.org/T406531) (owner: 10Joal) [15:11:45] 06Data-Engineering, 06Data-Platform-SRE: Improve housekeeping of jar files in /tmp on Hadoop workers - https://phabricator.wikimedia.org/T396582#11306612 (10Gehel) It looks like not only jar files take space. At the moment, an.worker1161 is low on disk space, with a single directory taking 3.4G of what looks l... [15:18:22] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06Product Safety and Integrity: FYI: Changes to the cuc_agent column in the cu_changes table - https://phabricator.wikimedia.org/T361210#11306687 (10OKryva-WMF) [15:21:31] 06Data-Engineering, 06Product Safety and Integrity, 06Product-Analytics: Mediawiki history has no data on IP blocks - https://phabricator.wikimedia.org/T211627#11306743 (10OKryva-WMF) [15:21:35] 06Data-Engineering, 06Product Safety and Integrity, 06Product-Analytics: Distinguish between types of block events in the Mediawiki user history table - https://phabricator.wikimedia.org/T213583#11306744 (10OKryva-WMF) [15:22:12] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Update MediaWiki Content History SLO draft for SRE review - https://phabricator.wikimedia.org/T401892#11306831 (10xcollazo) Great @APizzata-WMF, let's discuss your query, and perhaps study those 1523 missing events at our next sync up. [15:36:00] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Certain languages on wmf_raw.mediawiki_project_namespace_map appear wrong - https://phabricator.wikimedia.org/T407902#11306943 (10xcollazo) Opened T408220 to chase this bug upstream. For our purposes though, we should be good with the workaround. [15:36:51] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Certain languages on wmf_raw.mediawiki_project_namespace_map appear wrong - https://phabricator.wikimedia.org/T407902#11306946 (10xcollazo) a:03xcollazo [15:45:48] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Implement the data layout, UI, and documentation for the XML file export - https://phabricator.wikimedia.org/T401022#11307002 (10xcollazo) From [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1764 | MR 1764 ]]: > I... [15:54:59] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Implement the data layout, UI, and documentation for the XML file export - https://phabricator.wikimedia.org/T401022#11307073 (10xcollazo) [15:55:37] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content: Compare the exported content between File Export and DumpV1 - https://phabricator.wikimedia.org/T407649#11307074 (10xcollazo) [16:44:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:49:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [17:24:53] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06DBA, and 2 others: Apply ce_event_contributions schema changes in production (x1) - https://phabricator.wikimedia.org/T407587#11307444 (10Daimona) >>! In T407587#11307355, @Marostegui wrote: > @Daimona can this be done or not yet? Yes sorry... [17:30:51] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06DBA, and 2 others: Apply ce_event_contributions schema changes in production (x1) - https://phabricator.wikimedia.org/T407587#11307482 (10Marostegui) Thank you! [18:03:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:13:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:18:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [18:30:50] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content: Compare the exported content between File Export and DumpV1 - https://phabricator.wikimedia.org/T407649#11307829 (10Ahoelzl) p:05Triage→03High [18:39:58] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11307866 (10Eevans) I've created a draft mer... [19:43:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:53:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:02:56] 07Analytics-Data-Problem, 06Data-Engineering, 06Movement-Insights, 06Product-Analytics, 13Patch-For-Review: webrequest dataset sets referer_class "unknown" instead of "external (search engine)" for origin-based referer values - https://phabricator.wikimedia.org/T383088#11308163 (10Mayakp.wiki) +1 to @Isa... [20:47:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:52:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:15:19] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content: Compare the exported content between File Export and DumpV1 - https://phabricator.wikimedia.org/T407649#11308321 (10Ahoelzl) Great findings. I recommend running a revision count per page comparisons for some of the big wikis... [21:21:12] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 07Essential-Work, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11308340 (10Ahoelzl) a:05Mayakp.wiki→03JAllema... [21:22:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=codfw%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:25:23] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 07Essential-Work, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11308343 (10Ahoelzl) @Isaac @Mayakp.wiki thoughts... [21:27:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:38:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:53:12] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Prepare data engineering infrastructure for drop of rev_sha1 - https://phabricator.wikimedia.org/T405503#11308371 (10Ahoelzl) 05Open→03Resolved [21:53:14] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Modify code to dump all slots - https://phabricator.wikimedia.org/T384945#11308373 (10Ahoelzl) 05In progress→03Resolved [21:53:16] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Experimentation Lab: NEW/CHANGE FEATURE REQUEST: make available the centralauth.globaluser table in Data Lake - https://phabricator.wikimedia.org/T389666#11308376 (10Ahoelzl) 05Open→03Resolved [21:53:24] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform: Deploy mediawiki-event-enrichment Flink jobs running 1.20 - https://phabricator.wikimedia.org/T401725#11308379 (10Ahoelzl) 05Open→03Resolved [21:53:29] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform: mediawiki.content_history: flink applications experiencing frequent restarts due to JobManager OOMs - https://phabricator.wikimedia.org/T397330#11308378 (10Ahoelzl) 05Open→03Resolved [21:53:33] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Dumps-Generation: possible data dump error, in category related tables - https://phabricator.wikimedia.org/T407435#11308386 (10Ahoelzl) 05Open→03Resolved [21:53:36] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Adapt mediawiki_history to the removal of mediawiki revision.rev_sha1 - https://phabricator.wikimedia.org/T406000#11308384 (10Ahoelzl) 05In progress→03Resolved [21:53:38] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Adapt MW Content pipelines to the removal of upstream revision.rev_sha1 - https://phabricator.wikimedia.org/T405641#11308383 (10Ahoelzl) 05In progress→03Resolved [21:53:39] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07Essential-Work, 13Patch-For-Review: [Data Quality] Implement wiki completeness check for MediaWiki History - https://phabricator.wikimedia.org/T365203#11308388 (10Ahoelzl) 05Open→03Resolved [21:53:44] 07Analytics-Data-Problem, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Unique devices data uses non-standard domains for Wikidata, Wikifunctions, and MediaWiki.org - https://phabricator.wikimedia.org/T405533#11308390 (10Ahoelzl) 05Open→03Resolved [21:53:47] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Backfill datasets affected by automated traffic detection issues - https://phabricator.wikimedia.org/T405667#11308392 (10Ahoelzl) 05Open→03Resolved [21:53:47] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:53:50] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Mitigate consequences of Gobblin hiccups generating late events and alerts - https://phabricator.wikimedia.org/T402324#11308394 (10Ahoelzl) 05Open→03Resolved [21:53:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Prepare webrequest derived Airflow DAGs for large scale re-runs - https://phabricator.wikimedia.org/T402645#11308393 (10Ahoelzl) 05Open→03Resolved [21:53:59] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Movement-Insights, 13Patch-For-Review: NEW BUG REPORT : Referer Name field is Null on Druid pageviews_daily - https://phabricator.wikimedia.org/T406530#11308391 (10Ahoelzl) 05Open→03Resolved [22:03:47] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:04:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:09:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:11:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:16:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:46:25] (03PS1) 10Aleksandar Mastilovic: Fix issues with mediawiki_history_reduced table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1198607 (https://phabricator.wikimedia.org/T406263)