[01:12:45] 06Data-Engineering-Icebox, 06Data-Engineering-Icebox-DEPRECATED, 10Observability-Logging: eventgate-analytics-external logs field explosion - https://phabricator.wikimedia.org/T333729#10446686 (10Ottomata) [02:37:56] 06Data-Engineering, 06Data-Engineering-Icebox, 06Privacy Engineering: Fix CI/CD issues in the differential-privacy repository - https://phabricator.wikimedia.org/T358601#10446764 (10Ottomata) [02:37:57] 06Data-Engineering, 06Data-Engineering-Icebox: Use hive metastore when registering views - https://phabricator.wikimedia.org/T350617#10446765 (10Ottomata) [02:37:58] 06Data-Engineering, 06Data-Engineering-Icebox: Allow retry on Airflow druid_load_webrequest_sampled_128_daily.remove_temporary_directory - https://phabricator.wikimedia.org/T345232#10446767 (10Ottomata) [02:37:59] 06Data-Engineering, 06Data-Engineering-Icebox: [SPIKE] Replicating EventGate Validation Errors in Local Environment - https://phabricator.wikimedia.org/T349018#10446766 (10Ottomata) [02:37:59] 06Data-Engineering, 06Data-Engineering-Icebox: Spark tries to create database in wrong place - https://phabricator.wikimedia.org/T340787#10446770 (10Ottomata) [02:38:00] 06Data-Engineering, 06Data-Engineering-Icebox: [opsweek] Airflow DAGs with Spark jobs should always include Spark tuning variables - https://phabricator.wikimedia.org/T343154#10446769 (10Ottomata) [02:38:03] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Investigate why we consume empty partitions from webrequests - https://phabricator.wikimedia.org/T343238#10446768 (10Ottomata) [02:38:07] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: TikTok referral data gaps - https://phabricator.wikimedia.org/T340677#10446771 (10Ottomata) [02:38:12] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: [Airflow] Simplify application and java_class parameters in SparkSqlOperator - https://phabricator.wikimedia.org/T338036#10446774 (10Ottomata) [02:38:16] 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE, 06Traffic: Webrequest x_analtics `wprov` value is incorrectly formatted - https://phabricator.wikimedia.org/T339910#10446772 (10Ottomata) [02:38:24] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Growth-Team, 07Browser-Support-Apple-Safari: Investigate how Safari in iOS 17 and macOS Sonoma will impact URLs generated in Wikimedia sites - https://phabricator.wikimedia.org/T338571#10446773 (10Ottomata) [02:38:30] 06Data-Engineering, 06Data-Engineering-Icebox: Increase webrequest_sampled_live Druid datasource's retention - https://phabricator.wikimedia.org/T337460#10446776 (10Ottomata) [02:38:34] 06Data-Engineering, 06Data-Engineering-Icebox, 10AQS2.0: Weekly views API - https://phabricator.wikimedia.org/T337085#10446777 (10Ottomata) [02:38:38] 06Data-Engineering, 06Data-Engineering-Icebox, 06Structured-Data-Backlog: Instrument {{Delete ...} template adding/removing on Commons and create a historical dataset - https://phabricator.wikimedia.org/T336955#10446778 (10Ottomata) [02:38:42] 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE, 06Traffic, and 3 others: Include User-Agent Client Hints in WebRequest logs - https://phabricator.wikimedia.org/T337947#10446775 (10Ottomata) [02:38:50] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: HDFS utils on Airflow to handle actions on hdfs files - https://phabricator.wikimedia.org/T336771#10446779 (10Ottomata) [02:38:54] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Make sure all partitions sensors are using the Dataset helpers - https://phabricator.wikimedia.org/T336741#10446780 (10Ottomata) [02:38:58] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Refactor our existing Airflow dags to use EasyDAG & DagProperties - https://phabricator.wikimedia.org/T336738#10446782 (10Ottomata) [02:39:03] 06Data-Engineering, 06Data-Engineering-Icebox: Codex, Graph, and Wikistats walk into a bar graph - https://phabricator.wikimedia.org/T336544#10446783 (10Ottomata) [02:39:07] 06Data-Engineering, 06Data-Engineering-Icebox, 10Observability-Logging: eventgate-analytics-external logs field explosion - https://phabricator.wikimedia.org/T333729#10446786 (10Ottomata) [02:39:11] 06Data-Engineering, 06Data-Engineering-Icebox, 06Privacy Engineering: The soon-to-be-released pageview datasets should be linked from dumps page - https://phabricator.wikimedia.org/T335958#10446784 (10Ottomata) [02:39:15] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 06Product-Analytics: Delete the leading question mark from uri_query in the webrequest table - https://phabricator.wikimedia.org/T334495#10446785 (10Ottomata) [02:39:19] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 07Epic, 13Patch-For-Review: Post Oozie -> Airflow migration refactorings - https://phabricator.wikimedia.org/T336739#10446781 (10Ottomata) [02:39:23] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 06Product-Analytics: Add log_search to monthly sqoop list - https://phabricator.wikimedia.org/T332621#10446788 (10Ottomata) [02:39:27] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Investigate dangling tables after Airflow 2.5.1 upgrade of an-airflow1004.eqiad.wmnet - https://phabricator.wikimedia.org/T332820#10446787 (10Ottomata) [02:39:31] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Airflow skein hook shouldn't fail when not managing to gather yarn logs - https://phabricator.wikimedia.org/T332215#10446789 (10Ottomata) [02:39:35] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Use RDD checkpointing in Mediawiki-History spark job - https://phabricator.wikimedia.org/T331003#10446790 (10Ottomata) [02:39:39] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Refactor analytics_test/dags/custom_operators_tryout_dag.py - https://phabricator.wikimedia.org/T327869#10446793 (10Ottomata) [02:39:43] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Update HiveToDruid job -to not use intermediary data - https://phabricator.wikimedia.org/T329257#10446792 (10Ottomata) [02:39:48] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Airflow operator to manage old data deletion - https://phabricator.wikimedia.org/T326826#10446794 (10Ottomata) [02:39:52] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Load wmf.unique_editors_by_country_monthly into Druid for access in Turnilo & Superset - https://phabricator.wikimedia.org/T330436#10446791 (10Ottomata) [02:39:56] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Migrate custom gitlab runner that runs Dockerfiles to releng's new production infra - https://phabricator.wikimedia.org/T326570#10446795 (10Ottomata) [02:40:00] 06Data-Engineering, 06Data-Engineering-Icebox: Add redirects in bot detection features - https://phabricator.wikimedia.org/T326337#10446797 (10Ottomata) [02:40:04] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Use uap-core browser-family for bot detection - https://phabricator.wikimedia.org/T326339#10446796 (10Ottomata) [02:40:08] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Update refinery-source PageviewDefinition to better handle `Special:` pages - https://phabricator.wikimedia.org/T325544#10446799 (10Ottomata) [02:40:12] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 06Product-Analytics: Add TikTok's in-app browser to ua-parser library - https://phabricator.wikimedia.org/T325611#10446798 (10Ottomata) [02:40:16] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Set up a repository to generate packaged conda environments via CI for Jupyter notebooks - https://phabricator.wikimedia.org/T325195#10446801 (10Ottomata) [02:40:20] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Increase mypy coverage in airflow-dags - https://phabricator.wikimedia.org/T325213#10446800 (10Ottomata) [02:40:24] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Improve job to prune old dataset partitions - https://phabricator.wikimedia.org/T322754#10446804 (10Ottomata) [02:40:28] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: NEW FEATURE REQUEST: sqoop (all) user properties from mariadb to wmf_raw.mediawiki_user_properties - https://phabricator.wikimedia.org/T323456#10446803 (10Ottomata) [02:40:32] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Improve docs around JupyterLab and conda-analytics - https://phabricator.wikimedia.org/T324025#10446802 (10Ottomata) [02:40:36] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Add support for repository artifacts in Airflow - https://phabricator.wikimedia.org/T322690#10446805 (10Ottomata) [02:40:40] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: MVP for Notebook Scheduler - https://phabricator.wikimedia.org/T322533#10446806 (10Ottomata) [02:40:44] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: wmf.virtualpageview_hourly's language_variant field is corrupted - https://phabricator.wikimedia.org/T322545#10446807 (10Ottomata) [02:40:49] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 07Epic: Notebook Scheduler for Product Analytics - https://phabricator.wikimedia.org/T322532#10446808 (10Ottomata) [02:40:53] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Puppetize custom gitlab runner that can launch Docker containers - https://phabricator.wikimedia.org/T322251#10446809 (10Ottomata) [02:40:57] 06Data-Engineering, 06Data-Engineering-Icebox, 06Experimentation Lab, 10Experimentation Lab Roadmap, 07Epic: Bot Detection - https://phabricator.wikimedia.org/T321707#10446811 (10Ottomata) [02:41:01] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Make launcher an explicit parameter on SparkSubmitOperator() - https://phabricator.wikimedia.org/T319688#10446813 (10Ottomata) [02:41:05] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Make mediawiki-history page and user sorting complete for denormalization - https://phabricator.wikimedia.org/T321493#10446812 (10Ottomata) [02:41:11] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 10Wikidata: Back-fill Wikidata reliability Graphite metrics - https://phabricator.wikimedia.org/T321838#10446810 (10Ottomata) [02:41:15] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Convert to pure Docker the gitlab CI pipeline to build debianized conda - https://phabricator.wikimedia.org/T315475#10446815 (10Ottomata) [02:41:19] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 06Product-Analytics: Improved edit summary data in mediawiki_history - https://phabricator.wikimedia.org/T318010#10446814 (10Ottomata) [02:41:23] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Suspicious user pageview activity in India during June from Android mobile web browsers - https://phabricator.wikimedia.org/T315267#10446816 (10Ottomata) [02:41:27] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Sanitize network_flows_internal dataset - https://phabricator.wikimedia.org/T312915#10446817 (10Ottomata) [02:41:31] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Bug: Deleted pages are accidentally excluded from mediawiki_history_reduced - https://phabricator.wikimedia.org/T313955#10446818 (10Ottomata) [02:41:35] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: [Iceberg] Migrate event_santised to Iceberg - https://phabricator.wikimedia.org/T311765#10446819 (10Ottomata) [02:41:39] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: [Iceberg] Modify superset dashboards to use timestamp based (instead of partition based) queries - https://phabricator.wikimedia.org/T311764#10446820 (10Ottomata) [02:41:44] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: [Iceberg] Migrate event_sanitized_iceberg to event_sanitized - https://phabricator.wikimedia.org/T311737#10446823 (10Ottomata) [02:41:48] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Catalog, 06Data-Platform-SRE: Re-enable Public Druid metadata ingestion - https://phabricator.wikimedia.org/T311547#10446824 (10Ottomata) [02:41:54] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 07Epic: [Iceberg] Epic: Icebergify event_sanitized database - https://phabricator.wikimedia.org/T311743#10446821 (10Ottomata) [02:41:58] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 13Patch-Needs-Improvement: [Iceberg] Update Refine Sanitize to insert into Iceberg tables - https://phabricator.wikimedia.org/T311739#10446822 (10Ottomata) [02:42:02] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Generate data to count langswitches for every article - https://phabricator.wikimedia.org/T310975#10446827 (10Ottomata) [02:42:06] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Drop MediaViewer and MultimediaViewer* tables - https://phabricator.wikimedia.org/T311229#10446825 (10Ottomata) [02:42:10] 06Data-Engineering, 06Data-Engineering-Icebox: Improve Bot Detection Heuristics - https://phabricator.wikimedia.org/T310846#10446828 (10Ottomata) [02:42:14] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Drop ArticleCreationWorkflow data - https://phabricator.wikimedia.org/T310863#10446826 (10Ottomata) [02:42:18] 06Data-Engineering, 06Data-Engineering-Icebox: Late events in wdqs-external.sparql-query? - https://phabricator.wikimedia.org/T310790#10446829 (10Ottomata) [02:43:33] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Fundraising-Backlog: Identify source of discrepancy between HUE query in Count of event.impression and druid queries via turnilo/superset - https://phabricator.wikimedia.org/T204396#10447036 (10Ottomata) [02:43:37] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10ChangeProp, and 2 others: Consider the possibility of separating ChangeProp and JobQueue on Kafka level - https://phabricator.wikimedia.org/T199431#10447037 (10Ottomata) [02:45:35] 06Data-Engineering, 06Data-Engineering-Icebox: Sesssion reconstruction - evaluate privacy threat - https://phabricator.wikimedia.org/T194058#10447040 (10Ottomata) [02:45:39] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: AQS edits API should not allow queries without time bounds - https://phabricator.wikimedia.org/T189623#10447042 (10Ottomata) [02:45:43] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10draftquality-modeling, and 2 others: Productionize monthly article quality prediction datasets - https://phabricator.wikimedia.org/T194741#10447039 (10Ottomata) [02:45:49] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10observability, 07Puppet: Upgrade prometheus-jmx-exporter on all services using it - https://phabricator.wikimedia.org/T192948#10447041 (10Ottomata) [02:45:57] 06Data-Engineering, 06Data-Engineering-Icebox: Mediawiki History: moves counted twice in Revision - https://phabricator.wikimedia.org/T189044#10447044 (10Ottomata) [02:46:01] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-release, and 2 others: An expert panel to produce recommendations on open data sharing for public good - https://phabricator.wikimedia.org/T189339#10447043 (10Ottomata) [02:46:05] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE, 06Traffic-Icebox: Investigate and fix odd uri_host values - https://phabricator.wikimedia.org/T188804#10447045 (10Ottomata) [02:46:13] 06Data-Engineering, 06Data-Engineering-Icebox, 10MediaWiki-Core-AuthManager, 10MediaWiki-Platform-Team (Radar): Clear site data on MediaWiki log out - https://phabricator.wikimedia.org/T179752#10447047 (10Ottomata) [02:46:19] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Privacy Engineering, and 2 others: Publishing project anomaly data for censorship researchers. Evaluate privacy threats - https://phabricator.wikimedia.org/T183990#10447046 (10Ottomata) [02:46:27] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Provide unqiues estimate/offset breakdowns in AQS - https://phabricator.wikimedia.org/T164593#10447049 (10Ottomata) [02:46:31] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Enable nested on-wiki config pages in mediawiki-storage - https://phabricator.wikimedia.org/T163725#10447050 (10Ottomata) [02:46:35] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10CirrusSearch, 06Discovery-Search: Load cirrussearch data into druid - https://phabricator.wikimedia.org/T156037#10447051 (10Ottomata) [02:46:39] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06cloud-services-team, 10Data-Services: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#10447048 (10Ottomata) [02:46:45] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: [REQUEST] Extract search queries from HTTP_REFERER field for a Wikibook - https://phabricator.wikimedia.org/T144714#10447053 (10Ottomata) [02:46:49] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Decide on JSON validation library - https://phabricator.wikimedia.org/T147137#10447052 (10Ottomata) [02:46:55] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Research-consulting, 06Research-Freezer: Report on Wikimedia's industry ranking - https://phabricator.wikimedia.org/T141117#10447055 (10Ottomata) [02:46:59] 06Data-Engineering, 06Data-Engineering-Icebox, 06Research-Freezer: [Open question] Improve bot identification at scale - https://phabricator.wikimedia.org/T138207#10447056 (10Ottomata) [02:47:03] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10MediaWiki-Page-editing, and 2 others: statistics about edit conflicts according to page type - https://phabricator.wikimedia.org/T139019#10447054 (10Ottomata) [02:47:13] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-release: Wikipedia Clickstream dataset. Programmatic Access - https://phabricator.wikimedia.org/T134231#10447058 (10Ottomata) [02:47:17] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Add json linting test for schemas in mediawiki/event-schemas - https://phabricator.wikimedia.org/T124319#10447059 (10Ottomata) [02:47:24] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10MediaWiki-Action-API, 13Patch-For-Review: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables - https://phabricator.wikimedia.org/T137321#10447057 (10Ottomata) [02:47:30] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Create a tool that can read the elements of a wiki template and call the API {kudu} - https://phabricator.wikimedia.org/T117290#10447061 (10Ottomata) [02:47:34] 06Data-Engineering, 06Data-Engineering-Icebox: Add alarms for high volume of views to pages with replacement characters - https://phabricator.wikimedia.org/T117945#10447060 (10Ottomata) [02:47:38] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10MediaWiki-Action-API: Add Application errors for Mediawiki API to x-analytics - https://phabricator.wikimedia.org/T116658#10447062 (10Ottomata) [02:47:44] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Research-consulting: Too few page views for June/July 2015 - https://phabricator.wikimedia.org/T106034#10447065 (10Ottomata) [02:47:48] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 07patch-welcome: Validate JsonSchemaContent using MediaWIki core's handling - https://phabricator.wikimedia.org/T76432#10447067 (10Ottomata) [02:47:52] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE, and 2 others: Add request_id to webrequest logs as well as other event records ingested into Hadoop - https://phabricator.wikimedia.org/T113817#10447064 (10Ottomata) [02:48:00] 06Data-Engineering, 06Data-Engineering-Icebox, 06Data-Platform-SRE: Misconfigured proxies on data-engineering hosts - https://phabricator.wikimedia.org/T326302#10447068 (10Ottomata) [02:48:06] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics, and 3 others: there should be a comparison of clicks count on interlanguage links on different platforms - https://phabricator.wikimedia.org/T78351#10447066 (10Ottomata) [02:48:10] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines: Fix mediawiki-history page computation for deleted pages having the same title - https://phabricator.wikimedia.org/T320860#10447069 (10Ottomata) [02:48:14] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 06Product-Analytics: [REQUEST] Add new Fundraising dimensions to druid.pageviews_daily & druid.pageviews_hourly - https://phabricator.wikimedia.org/T304571#10447070 (10Ottomata) [02:48:18] 06Data-Engineering, 06Data-Engineering-Icebox: Implement one golang AQS microservice - https://phabricator.wikimedia.org/T299729#10447071 (10Ottomata) [02:48:22] 06Data-Engineering, 06Data-Engineering-Icebox: Fill holes in pageview-complete dumps using pageview-count-raw - https://phabricator.wikimedia.org/T290060#10447073 (10Ottomata) [02:48:26] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics, 10Editing-team (Tracking): How often do people try to edit on mobile devices, using the desktop site, at the English Wikipedia? - https://phabricator.wikimedia.org/T288972#10447074 (10Ottomata) [02:48:30] 06Data-Engineering, 06Data-Engineering-Icebox, 06Data-Platform-SRE: LVS in Analytics VLANs - https://phabricator.wikimedia.org/T288750#10447075 (10Ottomata) [02:48:36] 06Data-Engineering, 06Data-Engineering-Icebox, 06serviceops-radar, 10Wikidata Change Dispatching & Watchlists: Better observability/visualization for MediaWiki jobs - https://phabricator.wikimedia.org/T291620#10447072 (10Ottomata) [02:48:40] 06Data-Engineering, 06Data-Engineering-Icebox: Sqoop image metadata - https://phabricator.wikimedia.org/T285783#10447076 (10Ottomata) [02:48:44] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: [event sanitization] Add a "mask" functionality to purge string fields - https://phabricator.wikimedia.org/T281144#10447077 (10Ottomata) [02:48:48] 06Data-Engineering, 06Data-Engineering-Icebox: Easy dimensional data visualization - https://phabricator.wikimedia.org/T280029#10447078 (10Ottomata) [02:48:52] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467#10447080 (10Ottomata) [02:48:56] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Improve Sonar job for analytics-refinery-source - https://phabricator.wikimedia.org/T279841#10447079 (10Ottomata) [02:49:00] 06Data-Engineering, 06Data-Engineering-Icebox, 06Data-Platform-SRE: Turnilo "Display Druid query" gives "general error" - https://phabricator.wikimedia.org/T273685#10447082 (10Ottomata) [02:49:06] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Persistence-Backup: Implement production zookeeper backups - https://phabricator.wikimedia.org/T274808#10447081 (10Ottomata) [02:49:12] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Feature request: Keeping track of time spent in phases of edits for users - https://phabricator.wikimedia.org/T268385#10447084 (10Ottomata) [02:49:16] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Chart data from analytics.wikimedia.org do not fully specify macOS or Windows versions. - https://phabricator.wikimedia.org/T269722#10447083 (10Ottomata) [02:49:20] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox, 07Mobile: analytics.wikimedia.org is less readable on smartphones - https://phabricator.wikimedia.org/T266132#10447087 (10Ottomata) [02:49:24] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Stats dashboards on analytics.wikimedia.org are poorly navigable on mobile devices - https://phabricator.wikimedia.org/T266142#10447086 (10Ottomata) [02:49:28] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Analyze differences between checksum-based and revert-tag based reverts in mediawiki_history - https://phabricator.wikimedia.org/T266374#10447085 (10Ottomata) [02:49:32] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox, 07Browser-Support-Apple-Safari: On analytics.wikimedia.org, Safari on iOS ignores collapsed tabs interaction - https://phabricator.wikimedia.org/T266122#10447088 (10Ottomata) [02:49:36] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: History: mismatched historical and latest values - https://phabricator.wikimedia.org/T258967#10447091 (10Ottomata) [02:49:40] 06Data-Engineering, 06Data-Engineering-Icebox: page_id is null where it shouldn't be in mediawiki history - https://phabricator.wikimedia.org/T259823#10447090 (10Ottomata) [02:49:44] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Create a tool checking HDFS data size - https://phabricator.wikimedia.org/T256644#10447092 (10Ottomata) [02:49:48] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox, 13Patch-For-Review: Quick data exploration CLI - https://phabricator.wikimedia.org/T265765#10447089 (10Ottomata) [02:49:52] 06Data-Engineering, 06Data-Engineering-Icebox: Idea: Add 'top X bigger than Y' sanitization method to EL-to-Druid - https://phabricator.wikimedia.org/T251145#10447095 (10Ottomata) [02:49:56] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Editing-team, and 2 others: EditPage save hooks pass an entire `EditPage` object - https://phabricator.wikimedia.org/T251588#10447093 (10Ottomata) [02:50:04] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Dumps-Generation, 13Patch-Needs-Improvement: page_restrictions field incomplete in current and historical dumps - https://phabricator.wikimedia.org/T251411#10447094 (10Ottomata) [02:50:08] 06Data-Engineering, 06Data-Engineering-Icebox: Release a public dataset about percentage of referrers in wikipedia traffic - https://phabricator.wikimedia.org/T250840#10447096 (10Ottomata) [02:50:12] 06Data-Engineering, 06Data-Engineering-Icebox: Join slot, content, revision, and page once on load - https://phabricator.wikimedia.org/T240413#10447097 (10Ottomata) [02:50:16] 06Data-Engineering, 06Data-Engineering-Icebox, 10incubator.wikimedia.org: Create dashiki dashboard / small tool to track statistics about incubated wikis - https://phabricator.wikimedia.org/T237389#10447098 (10Ottomata) [02:50:20] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Add page protection status to MediaWiki history tables - https://phabricator.wikimedia.org/T230044#10447099 (10Ottomata) [02:50:24] 06Data-Engineering, 06Data-Engineering-Icebox: Set entropy alarm in editors per country per wiki - https://phabricator.wikimedia.org/T227809#10447100 (10Ottomata) [02:50:28] 06Data-Engineering, 06Data-Engineering-Icebox: Aggregate pageviews to Wikidata entities - https://phabricator.wikimedia.org/T215438#10447104 (10Ottomata) [02:50:32] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10MobileFrontend, 10XAnalytics: MobileFrontend should use XAnalytics extension - https://phabricator.wikimedia.org/T217859#10447103 (10Ottomata) [02:50:38] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox, 10MediaWiki-Core-JobQueue, and 2 others: showJobs.php maintenance script useless and misleading in production - https://phabricator.wikimedia.org/T221224#10447101 (10Ottomata) [02:50:44] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Research: add participant list to some of AQS edit api operations - https://phabricator.wikimedia.org/T206137#10447105 (10Ottomata) [02:50:48] 06Data-Engineering, 06Data-Engineering-Icebox: Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users - https://phabricator.wikimedia.org/T204950#10447106 (10Ottomata) [02:50:52] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Approximate mediawiki-history user creation dates using user-id/registration-date coherence - https://phabricator.wikimedia.org/T202899#10447107 (10Ottomata) [02:50:56] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Sqoop more tables for mediawiki in monthly schedule - https://phabricator.wikimedia.org/T198983#10447109 (10Ottomata) [02:51:00] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox, 10Datasets-General-or-Unknown: Add checksums pageviews dataset - https://phabricator.wikimedia.org/T199461#10447108 (10Ottomata) [02:51:04] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10MediaWiki-Vagrant: role::spark should compile into a catalogue without dependency cycles - https://phabricator.wikimedia.org/T184151#10447112 (10Ottomata) [02:51:10] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Publish data on seen page previews - https://phabricator.wikimedia.org/T193524#10447111 (10Ottomata) [02:51:14] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Count the number of video plays - https://phabricator.wikimedia.org/T198628#10447110 (10Ottomata) [02:51:18] 06Data-Engineering, 06Data-Engineering-Icebox: Implement digest-only mediawiki_history_reduced dataset in spark - https://phabricator.wikimedia.org/T181703#10447114 (10Ottomata) [02:51:22] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Enhance mediawiki-history page reconstruction with best historical information possible - https://phabricator.wikimedia.org/T179692#10447115 (10Ottomata) [02:51:26] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10ChangeProp, 10MediaWiki-Core-JobQueue: Allow easy tuning of the jobqueue concurrency. - https://phabricator.wikimedia.org/T175800#10447116 (10Ottomata) [02:51:32] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Productionize analysis of editcount vs per_user_revision_count - https://phabricator.wikimedia.org/T168648#10447118 (10Ottomata) [02:51:36] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Provide historical redirect flag in Data Lake edit data - https://phabricator.wikimedia.org/T161146#10447119 (10Ottomata) [02:51:40] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Machine-Learning-Team, and 3 others: [Investigate] Use PMML for prediction model serialization - https://phabricator.wikimedia.org/T173244#10447117 (10Ottomata) [02:51:46] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Sort inconsistency in AQS timestamp behavior - https://phabricator.wikimedia.org/T160311#10447120 (10Ottomata) [02:51:50] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Import 2001 wikipedia data - https://phabricator.wikimedia.org/T155014#10447121 (10Ottomata) [02:51:54] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: de-duplicate archive records matching revision records in mediawiki_history - https://phabricator.wikimedia.org/T152546#10447122 (10Ottomata) [02:51:58] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Collect information about how we collect user statistics in one place - https://phabricator.wikimedia.org/T132405#10447123 (10Ottomata) [02:52:02] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06WMDE-Analytics-Engineering: Provide machine readable directory indexes on http://datasets.wikimedia.org/aggregate-datasets/ - https://phabricator.wikimedia.org/T117480#10447125 (10Ottomata) [02:52:08] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Better redirect handling for pageview API - https://phabricator.wikimedia.org/T121912#10447124 (10Ottomata) [02:52:12] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Analyze difference in Edit Schema "bounce rates" across wikis {lion} - https://phabricator.wikimedia.org/T89726#10447126 (10Ottomata) [02:52:16] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Investigate getting redirect_page_id as an x_analytics field using the X analytics extension. {pika} - https://phabricator.wikimedia.org/T89397#10447128 (10Ottomata) [02:52:20] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Fundraising-Backlog: Dashboard for CentralNotice impression rates using Druid, centralnotice_analytics and CN events - https://phabricator.wikimedia.org/T254792#10447129 (10Ottomata) [02:52:26] 06Data-Engineering, 06Data-Engineering-Icebox: Release wikimedia history dumps sorted by user ID and page ID - https://phabricator.wikimedia.org/T232844#10447130 (10Ottomata) [02:52:30] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06cloud-services-team, 10Cloud-VPS: Report page views for labs instances - https://phabricator.wikimedia.org/T103726#10447127 (10Ottomata) [02:52:36] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Machine-Learning-Team, 10ORES: Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545#10447132 (10Ottomata) [02:52:44] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Commons, and 2 others: Provide download statistics of files on Wikimedia Commons - https://phabricator.wikimedia.org/T218076#10447131 (10Ottomata) [02:52:52] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Machine-Learning-Team, 10ORES: Purge ORES scores from Hadoop and begin backfill when model version changes - https://phabricator.wikimedia.org/T209742#10447134 (10Ottomata) [02:52:58] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Dumps-Generation, and 5 others: Decide whether we will include raw features - https://phabricator.wikimedia.org/T211069#10447133 (10Ottomata) [02:53:04] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Machine-Learning-Team, 10ORES: Include feature values in ORES changeprop stream - https://phabricator.wikimedia.org/T209734#10447136 (10Ottomata) [02:53:10] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Machine-Learning-Team, 10ORES: Wire ORES recent_score events into Hadoop - https://phabricator.wikimedia.org/T209732#10447137 (10Ottomata) [02:53:17] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Dumps-Generation, and 2 others: Produce dump files for ORES scores - https://phabricator.wikimedia.org/T209739#10447135 (10Ottomata) [02:53:23] 06Data-Engineering, 06Data-Engineering-Icebox: Add legacy per-article pagecounts data (prior to 2015) - https://phabricator.wikimedia.org/T193759#10447139 (10Ottomata) [02:53:27] 06Data-Engineering, 06Data-Engineering-Icebox: Automate refinery jar cleanup - https://phabricator.wikimedia.org/T159337#10447141 (10Ottomata) [02:53:31] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Add pagecounts by article and top pagecounts to AQS - https://phabricator.wikimedia.org/T173720#10447140 (10Ottomata) [02:53:37] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10[DEPRECATED] wdwb-tech, and 7 others: [Epic] Make ORES scores for wikidata available as a dump - https://phabricator.wikimedia.org/T209611#10447138 (10Ottomata) [02:53:43] 06Data-Engineering, 06Data-Engineering-Icebox, 07Epic: Sanitize pageview_hourly - https://phabricator.wikimedia.org/T114675#10447143 (10Ottomata) [02:53:47] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Add CI job Oozie XML stylesheet validation for the analytics/refinery repository - https://phabricator.wikimedia.org/T147072#10447142 (10Ottomata) [02:53:51] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Provide regular cross-wiki reports on flagged revisions status - https://phabricator.wikimedia.org/T44360#10447144 (10Ottomata) [02:54:03] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10draftquality-modeling, and 2 others: Productionize monthly article quality prediction datasets - https://phabricator.wikimedia.org/T194741#10447145 (10Ottomata) 05Open→03Declined [03:12:03] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Add CI job Oozie XML stylesheet validation for the analytics/refinery repository - https://phabricator.wikimedia.org/T147072#10447163 (10Ottomata) 05Open→03Declined [03:12:52] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10MediaWiki-Vagrant: role::spark should compile into a catalogue without dependency cycles - https://phabricator.wikimedia.org/T184151#10447164 (10Ottomata) 05Open→03Invalid [03:13:39] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 07patch-welcome: Validate JsonSchemaContent using MediaWIki core's handling - https://phabricator.wikimedia.org/T76432#10447166 (10Ottomata) 05Open→03Declined [04:52:51] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-Page-deletion, 10Event-Platform, 10MediaWiki-Platform-Team (Radar): Undelete of page with same title leads to unexpected results - https://phabricator.wikimedia.org/T351411#10447210 (10Pppery) Is there actually anything that needs to change here?... [08:23:43] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox: Create cron on 1002 to remove CirrusSearchRequest partitions - https://phabricator.wikimedia.org/T119897#10447279 (10dcausse) 05Open→03Declined this dataset is cleaned by airflow now [08:46:43] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10CirrusSearch, 06Discovery-Search: Load cirrussearch data into druid - https://phabricator.wikimedia.org/T156037#10447311 (10dcausse) 05Open→03Declined we have this data in relforge so I don't think we'll need this in druid, but f... [09:51:39] btullis, brouberol o/ - I ran apt-get clean on various (10+) an-worker nodes, the root partition was 100% used and in some of them the node manager failed (due to space issues) [09:52:03] I noticed that some home dirs have been used, and also tmp files [09:52:23] non sure if it is due to a new job that allocates more tmp files on the root of the workers [09:53:25] elukey: Thanks ever so much. I created this yesterday T383320 and I'm chasing up with fab: [09:53:26] T383320: Low disk space on the root partition for several Hadoop workers - https://phabricator.wikimedia.org/T383320 [09:54:25] Also yes, I think that we might need to implement some housekeeping for `/tmp` since we have some large artifacts there and I think that they might be left behind from some tasks, but I'm not sure which, yet. [09:55:21] btullis: ack yes it was my impression as well, mind if I run apt-get clean on all an-workers? [09:55:33] just to free space and be safer [09:57:49] Yes, please do go ahead. <3 [10:02:47] done! [10:39:00] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2025.01.11 - 2025.01.31), 13Patch-For-Review: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0 - https://phabricator.wikimedia.org/T338057#10447485 (10Gehel) [10:39:36] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Research, 10Data-Platform-SRE (2025.01.11 - 2025.01.31), 03Discovery-Search (Current work): Low available space on Hadoop / HDFS - https://phabricator.wikimedia.org/T381707#10447515 (10Gehel) [10:39:48] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation, 10Data-Platform-SRE (2025.01.11 - 2025.01.31), 13Patch-For-Review: Switch dumps 1.0 processes to use the analytics MariadB replicas (dbstore100[7-9]) - https://phabricator.wikimedia.org/T382947#10447513 (10Gehel) [10:40:01] 06Data-Engineering, 06Data-Engineering-Radar, 10Cassandra, 10Data Pipelines, and 2 others: Create puppet resource for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#10447523 (10Gehel) [10:40:09] 06Data-Engineering, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Draft a project plan for the Hadoop version 3 upgrade - https://phabricator.wikimedia.org/T379748#10447531 (10Gehel) [10:40:42] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Airflow UI sometimes shows no response for a DAG run task with many mapped tasks - https://phabricator.wikimedia.org/T381479#10447545 (10Gehel) [10:40:56] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Data Platform access streamlining for WMDE staff - https://phabricator.wikimedia.org/T381824#10447541 (10Gehel) [10:41:22] 06Data-Engineering, 06Data-Engineering-Radar, 10CirrusSearch, 10Structured Data Engineering, and 3 others: Migrate image recommendation to use page_weighted_tags_changed stream - https://phabricator.wikimedia.org/T372912#10447561 (10Gehel) [10:41:46] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Test if an existing conda environment with Spark 3.1.2 clients works fine with Spark 3.5.3 - https://phabricator.wikimedia.org/T380417#10447565 (10Gehel) [10:42:16] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Discovery-Search, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Upload an image with flink-k8s-operator version that supports flink 1.20 - https://phabricator.wikimedia.org/T377137#10447577 (10Gehel) [10:42:28] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Some wikibase tables not available in commonswiki_p - https://phabricator.wikimedia.org/T298452#10447583 (10Gehel) [10:42:46] Thanks elukey! [10:42:47] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Discovery-Search, 10Dumps 2.0, 10Data-Platform-SRE (2025.01.11 - 2025.01.31), 13Patch-For-Review: Add relevant kafka clusters to defined airflow connections in puppet - https://phabricator.wikimedia.org/T379676#10447581 (10Gehel) [12:46:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [12:46:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [12:51:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [12:51:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [12:56:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [12:56:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:01:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:01:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:17:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:22:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:22:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:37:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:37:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:42:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:42:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:44:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:44:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:54:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:54:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [13:59:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [13:59:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:18:31] 06Data-Engineering, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Delete ganeti VM eventlog1003.eqiad.wmnet - https://phabricator.wikimedia.org/T383276#10448173 (10Gehel) [14:24:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:24:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:28:09] 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, 10MW-on-K8s, and 3 others: Generate a dumps-enabled mediawiki image - https://phabricator.wikimedia.org/T381473#10448251 (10Gehel) [14:28:42] 06Data-Engineering, 10Dumps-Generation: 2024-11-20 dump run appears stuck - https://phabricator.wikimedia.org/T380729#10448262 (10Gehel) [14:29:12] 10Data-Engineering (Q2 2024 October 1st - December 31th): Increase the number of partitions for the webrequest_frontend topics. - https://phabricator.wikimedia.org/T380566#10448266 (10Gehel) [14:30:07] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Request Kerberos identity for jsn.sherman - https://phabricator.wikimedia.org/T378786#10448279 (10Gehel) [14:30:33] 10Data-Engineering (Q2 2024 October 1st - December 31th): Refine jobs fail to send alert emails due to a decommissioned MX - https://phabricator.wikimedia.org/T377698#10448285 (10Gehel) [14:30:57] 06Data-Engineering, 10Dumps-Generation: dumps-exception-checker.service on dumpsdata1006 is repeatedly failing - https://phabricator.wikimedia.org/T377571#10448289 (10Gehel) [14:31:49] 06Data-Engineering, 10Dumps-Generation: Hide autoblocks from the globalblocks table database dump - https://phabricator.wikimedia.org/T376726#10448299 (10Gehel) [14:32:47] 14Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Retry package added needs the types-retry 0.9.9.4 typing stub - https://phabricator.wikimedia.org/T374396#10448309 (10Gehel) [14:33:46] 06Data-Engineering, 13Patch-For-Review: Requesting Kerberos access for ifrahkhanyaree - https://phabricator.wikimedia.org/T371894#10448324 (10Gehel) [14:33:52] 06Data-Engineering, 06DC-Ops, 10ops-eqiad, 06SRE: Degraded RAID on dumpsdata1007 - https://phabricator.wikimedia.org/T369829#10448327 (10Gehel) [14:34:00] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 06Experimentation Lab, 06Trust and Safety Product Team: Hide rows in the globalblocks table when the associated globaluser row has gu_hidden_level as not 0 - https://phabricator.wikimedia.org/T371488#10448326 (10Gehel) [14:34:20] 06Data-Engineering, 10Event-Platform: mw-page-content-change-enrich flink app is missing in k8s staging - https://phabricator.wikimedia.org/T367116#10448332 (10Gehel) [14:35:28] 06Data-Engineering: Package request: install elixir and erlang-otp to the analytics clients - https://phabricator.wikimedia.org/T362678#10448346 (10Gehel) [14:36:41] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 03Discovery-Search (Current work): Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and valida... - https://phabricator.wikimedia.org/T367405#10448370 [14:36:55] 06Data-Engineering, 07Epic: [Epic] Migrate Data Platform Engineering maintained git repos to GitLab - https://phabricator.wikimedia.org/T368927#10448372 (10Ottomata) [14:44:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:44:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:04:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:04:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:10:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:10:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:15:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:15:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:18:56] 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, 10MW-on-K8s, and 4 others: Generate a dumps-enabled mediawiki image - https://phabricator.wikimedia.org/T381473#10448513 (10Gehel) [15:22:20] 10Data-Engineering (Q2 2024 October 1st - December 31th): Increase the number of partitions for the webrequest_frontend topics. - https://phabricator.wikimedia.org/T380566#10448532 (10Gehel) [15:24:20] 06Data-Engineering, 10Dumps-Generation: 2024-11-20 dump run appears stuck - https://phabricator.wikimedia.org/T380729#10448542 (10Gehel) [15:30:05] 10Data-Engineering (Q2 2024 October 1st - December 31th): Refine jobs fail to send alert emails due to a decommissioned MX - https://phabricator.wikimedia.org/T377698#10448611 (10Gehel) [15:31:55] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Request Kerberos identity for jsn.sherman - https://phabricator.wikimedia.org/T378786#10448630 (10Gehel) [15:33:23] 06Data-Engineering, 10Dumps-Generation: Hide autoblocks from the globalblocks table database dump - https://phabricator.wikimedia.org/T376726#10448651 (10Gehel) [15:35:24] 06Data-Engineering, 10Dumps-Generation: dumps-exception-checker.service on dumpsdata1006 is repeatedly failing - https://phabricator.wikimedia.org/T377571#10448676 (10Gehel) [15:36:18] 14Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Retry package added needs the types-retry 0.9.9.4 typing stub - https://phabricator.wikimedia.org/T374396#10448682 (10Gehel) [15:36:36] 06Data-Engineering, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Create a container image for analytics/refinery to be used with Airflow tasks - https://phabricator.wikimedia.org/T383417 (10BTullis) 03NEW [15:38:59] 06Data-Engineering, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Create a container image for analytics/refinery to be used with Airflow tasks - https://phabricator.wikimedia.org/T383417#10448711 (10BTullis) p:05Triage→03High a:03BTullis [15:42:26] 06Data-Engineering, 10Event-Platform: mw-page-content-change-enrich flink app is missing in k8s staging - https://phabricator.wikimedia.org/T367116#10448749 (10Gehel) [15:43:27] 06Data-Engineering, 06DC-Ops, 10ops-eqiad, 06SRE: Degraded RAID on dumpsdata1007 - https://phabricator.wikimedia.org/T369829#10448754 (10Gehel) [15:44:28] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 06Experimentation Lab, 06Trust and Safety Product Team: Hide rows in the globalblocks table when the associated globaluser row has gu_hidden_level as not 0 - https://phabricator.wikimedia.org/T371488#10448760 (10Gehel) [15:44:32] 06Data-Engineering: Requesting Kerberos access for ifrahkhanyaree - https://phabricator.wikimedia.org/T371894#10448762 (10Gehel) [15:49:06] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-Page-deletion, 10Event-Platform, 10MediaWiki-Platform-Team (Radar): Undelete of page with same title leads to unexpected results - https://phabricator.wikimedia.org/T351411#10448801 (10Ottomata) 05Open→03Invalid We can close as invalid. A... [15:49:13] 06Data-Engineering: Package request: install elixir and erlang-otp to the analytics clients - https://phabricator.wikimedia.org/T362678#10448806 (10Gehel) [15:50:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:50:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:53:27] 06Data-Engineering, 10Scap, 07git-lfs: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472#10448826 (10Gehel) [15:54:10] 06Data-Engineering, 10Event-Platform: [SPIKE] Investigate what happens to deployed Flink clusters if the k8s operator goes down? - https://phabricator.wikimedia.org/T346231#10448828 (10Gehel) [15:54:31] 06Data-Engineering: Update the sqoop configuration for mediawiki to obtain linktarget from the production replicas, instead of wikireplicas - https://phabricator.wikimedia.org/T352879#10448829 (10Gehel) [15:55:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:55:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:57:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:57:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:59:01] 06Data-Engineering: Migrate hue.wikimedia.org to bullseye - https://phabricator.wikimedia.org/T349400#10448871 (10Gehel) [16:02:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:02:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:03:11] 06Data-Engineering, 07Epic: Remove all resources associated with the superset-(next-)k8s.wimedia.org domains - https://phabricator.wikimedia.org/T358480#10448888 (10Gehel) [16:04:55] 14Data-Engineering (Sprint 7), 13Patch-For-Review: [Data Platform] Deploy Spark History Service - https://phabricator.wikimedia.org/T330176#10448905 (10Gehel) [16:07:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:07:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:12:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:12:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:14:16] 06Data-Engineering, 10Cassandra, 10Pageviews-API: Improve user management for AQS Cassandra - https://phabricator.wikimedia.org/T142073#10448936 (10Gehel) [16:16:40] 14Data-Engineering (Q1 2024 July 1st - September 30th), 06Java-Scala-Standardization, 03Discovery-Search (Current work), 10Release-Engineering-Team (Radar): Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403#10448943 (10Gehel) [16:19:34] 06Data-Engineering: Implement periodical cleaning of Airflow databases - https://phabricator.wikimedia.org/T322036#10448958 (10Gehel) [16:20:03] 06Data-Engineering: Airflow scheduler and webserver logs should be readable by airflow instance admins - https://phabricator.wikimedia.org/T304615#10448963 (10Gehel) [16:23:55] 06Data-Engineering: Send a critical alert to data-engineering if produce_canary_events isn't running correctly - https://phabricator.wikimedia.org/T337055#10448982 (10Gehel) [16:26:25] 06Data-Engineering, 06Discovery-Search, 07Epic, 10Event-Platform: [Epic] Set up multi DC Kafka stretch cluster - https://phabricator.wikimedia.org/T340492#10449027 (10Gehel) [16:31:07] 06Data-Engineering, 06Product-Analytics: Conda analytics environments breakage - conflicting dependencies between r-base and other - https://phabricator.wikimedia.org/T343823#10449077 (10Gehel) [16:32:08] 06Data-Engineering, 06Movement-Insights, 06Product-Analytics: Reconstruct Hive & Hadoop permissions for shared database - https://phabricator.wikimedia.org/T288983#10449085 (10Gehel) [16:32:38] 06Data-Engineering, 06Infrastructure-Foundations, 10Event-Platform: > ~1 request/second to intake-logging.wikimedia.org times out at the traffic/service interface - https://phabricator.wikimedia.org/T264021#10449089 (10Gehel) [16:33:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:33:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:35:24] 06Data-Engineering: Request DataHub edit access for David Martin (dmartin) - https://phabricator.wikimedia.org/T344217#10449118 (10Gehel) [16:36:35] 06Data-Engineering, 10Event-Platform: Upgrade eventgate Docker image to Bullseye and nodejs 12 - https://phabricator.wikimedia.org/T343510#10449128 (10Gehel) [16:38:05] 06Data-Engineering, 10Data-Engineering-Jupyter: Autocomplete is very slow (unusable) in Newpyter - https://phabricator.wikimedia.org/T290008#10449143 (10Gehel) [16:38:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:38:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:39:53] 06Data-Engineering, 07Epic: Re-evaluate data cache security issues, along with per user data caching - https://phabricator.wikimedia.org/T358753#10449150 (10Gehel) [16:42:19] 06Data-Engineering, 06Release-Engineering-Team, 03Discovery-Search (Current work), 13Patch-For-Review: SonarQube build are failing with Java 11 - https://phabricator.wikimedia.org/T355122#10449176 (10Gehel) [16:47:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:47:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:56:24] 14Data-Engineering-Icebox-DEPRECATED, 10Observability-Logging, 10Wikimedia-Logstash: Evaluate storing logs from applications in yarn with the typical logging infrastructure - https://phabricator.wikimedia.org/T300937#10449256 (10Gehel) [16:56:45] 06Data-Engineering, 10Data Pipelines, 07Spike: Explore Containerization Solutions for DE Applications - https://phabricator.wikimedia.org/T288254#10449259 (10Gehel) [16:58:35] 06Data-Engineering, 06serviceops: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598#10449268 (10Gehel) [16:58:40] 06Data-Engineering, 10Data-Platform-SRE (2025.01.11 - 2025.01.31), 13Patch-For-Review: Create a container image for analytics/refinery to be used with Airflow tasks - https://phabricator.wikimedia.org/T383417#10449270 (10BTullis) 05Open→03Resolved This image is now published and usable. ` btullis@mar... [16:59:14] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10CheckUser, 06DBA, 06Trust and Safety Product Team, 07Schema-change-in-production: Remove cuc_actiontext, cuc_only_for_read_old, and cuc_private from cu_changes on WMF wikis - https://phabricator.wikimedia.org/T370903#10449275 (10Marostegui) [17:01:02] 06Data-Engineering, 06Infrastructure-Foundations, 10Puppet-Infrastructure: GeoIP2-Anonymous-IP Subscription expired - https://phabricator.wikimedia.org/T342878#10449286 (10Gehel) [17:01:24] 06Data-Engineering, 06Experimentation Lab: [blocker] Airflow unittests failing with TypeError: Pool.create_or_update_pool() - https://phabricator.wikimedia.org/T352577#10449290 (10Gehel) [17:01:34] 06Data-Engineering: [Iceberg Migration] P.O.C. on Iceberg sensor using Postgres table to keep status of updates - https://phabricator.wikimedia.org/T340466#10449291 (10Gehel) [17:04:26] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 10Event-Platform: Add Antoine_Quhen to the deployment group - https://phabricator.wikimedia.org/T347296#10449307 (10Gehel) [17:17:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:18:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:18:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:23:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:23:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:32:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:32:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:37:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:37:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:39:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:39:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:44:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:44:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:49:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:49:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:04:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:04:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:14:22] 06Data-Engineering, 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10449468 (10xcollazo) >>! In T383030#10445832, @xcollazo wrote: > IIRC, we've been here before. Perhaps we should wait until @Milimetric comes back, he had a manual way of resyncing all this, is... [18:19:47] 06Data-Engineering, 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10449478 (10xcollazo) ` $ hostname -f dumpsdata1006.eqiad.wmnet /usr/bin/rsync -va --contimeout=600 --timeout=600 --exclude=**bad/ --exclude=**save/ --exclude=**not/ --exclude=**temp/ --exclude... [18:20:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:20:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:25:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:25:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:27:15] 06Data-Engineering, 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10449493 (10xcollazo) Ran with a wildcard to hit all wikis: ` $ hostname -f dumpsdata1006.eqiad.wmnet /usr/bin/rsync -va --contimeout=600 --timeout=600 --exclude=**bad/ --exclude=**save/ --exc... [18:27:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:27:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:28:02] 06Data-Engineering, 06Product-Analytics, 10Event-Platform: Enable Event Platform instruments to opt out of collecting User-Agent data - https://phabricator.wikimedia.org/T382173#10449495 (10Ottomata) [18:32:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:32:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:34:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:34:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:36:12] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647#10449505 (10Ottomata) Q, is this resolved? I think the stuff in this comment remains to be done? https://phabricato... [18:36:22] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board): Implement Airflow Dataset class for RestExternalTaskSensor - https://phabricator.wikimedia.org/T372647#10449508 (10Ottomata) Hm, I guess I moved it to the Done column...:D [18:37:49] 06Data-Engineering, 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10449511 (10xcollazo) Documented this procedure at on [[ https://wikitech.wikimedia.org/w/index.php?diff=2258947 | Dumps/Troubleshooting ]]. [18:38:27] 06Data-Engineering, 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10449512 (10xcollazo) @ValterVB please confirm things look good from your side, and if so, please close this ticket. Thanks! [18:39:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:39:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:44:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:44:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:49:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:49:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:58:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:58:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:03:26] 06Data-Engineering, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, and 6 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10449559 (10Ottomata) [19:27:02] 06Data-Engineering, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Delete ganeti VM eventlog1003.eqiad.wmnet - https://phabricator.wikimedia.org/T383276#10449641 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d5902b17-47ea-438c-9707-48b63d58385b) set by btullis@cumin1002 for 7 days, 0:00:0... [19:28:36] 06Data-Engineering, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Delete ganeti VM eventlog1003.eqiad.wmnet - https://phabricator.wikimedia.org/T383276#10449644 (10BTullis) a:03BTullis [19:29:29] 06Data-Engineering, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Delete ganeti VM eventlog1003.eqiad.wmnet - https://phabricator.wikimedia.org/T383276#10449647 (10BTullis) I'm shutting down the machine for a little while, prior to deleting it. ` btullis@ganeti1028:~$ sudo gnt-instance shutdown eventlog1003.e... [20:38:40] 06Data-Engineering, 06Data-Engineering-Icebox: Improve pageview automated traffic detection heuristics - https://phabricator.wikimedia.org/T280565#10449800 (10Mayakp.wiki) Note than we have made 2 significant changes to the automated traffic heuristics: - included Redirect pageviews T376196 - applied the hue... [20:45:16] 06Data-Engineering, 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10449808 (10ValterVB) I downloaded the file and everything seems OK. Thank you [20:57:17] 06Data-Engineering, 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10449844 (10xcollazo) 05Open→03Resolved a:03xcollazo [21:08:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:08:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:12:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:12:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:16:35] 06Data-Engineering, 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10449882 (10odimitrijevic) Thanks @xcollazo. Will this delay impact the timeliness of any of the downstream pipelines? [21:17:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:19:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:19:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:34:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:34:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:35:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:35:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:00:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:00:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:01:06] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:01:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:21:06] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:21:06] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:27:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:27:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:32:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:32:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:37:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:37:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:42:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:42:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:52:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:52:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:57:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:57:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [23:02:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [23:02:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [23:17:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [23:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [23:18:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [23:18:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [23:23:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [23:23:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag