[01:53:09] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 05FY2024-25 KR 5.2 Simplify feature development, and 2 others: Design and document new Domain Events feature in MediaWiki core - https://phabricator.wikimedia.org/T379959#10435918 (10Pppery) [05:25:05] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10436124 (10Marostegui) [05:25:13] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10436125 (10Marostegui) 05Open→03Resolved This is done [07:39:50] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 13Patch-For-Review: Switch dumps 1.0 processes to use the analytics MariadB replicas (dbstore100[7-9]) - https://phabricator.wikimedia.org/T382947#10436234 (10Marostegui) >>! In T382947#10432801, @B... [08:45:13] 06Data-Engineering-Icebox, 10Data Pipelines, 10Wikidata: Back-fill Wikidata reliability Graphite metrics - https://phabricator.wikimedia.org/T321838#10436487 (10Michael) This can probably be closed by now, as presumably all the relevant source data is long gone? However, I'm not on the Wikidata team anymore... [08:54:39] 06Data-Engineering, 06Data-Engineering-Radar: Reduce the number of files generated by geoeditors airflor jobs - https://phabricator.wikimedia.org/T304852#10436515 (10JAllemandou) 05Open→03Resolved Resolving as airflow code embeds the change. [09:05:47] 06Data-Engineering-Icebox: Create new table for 'referer' aggregated data - https://phabricator.wikimedia.org/T112284#10436554 (10JAllemandou) 05Open→03Resolved a:03JAllemandou I think we should consider this done. Resolving. [09:07:19] 06Data-Engineering-Icebox: Update clickstream code to support more languages - https://phabricator.wikimedia.org/T292476#10436560 (10JAllemandou) 05Open→03Resolved a:03JAllemandou This is done! [09:09:10] 14Analytics, 06Data-Engineering, 10Data Pipelines: Add cawiki to clickstream dataset - https://phabricator.wikimedia.org/T327982#10436565 (10JAllemandou) 05Open→03Resolved a:03JAllemandou This is has been done in parent task. Resolving. [09:09:42] 06Data-Engineering, 06Data-Engineering-Radar, 10Data Pipelines, 06Privacy Engineering: Add cswiki to clickstream - https://phabricator.wikimedia.org/T339805#10436570 (10JAllemandou) 05Open→03Resolved a:03JAllemandou This has been done in parent task. Resolving. [09:11:18] 06Data-Engineering-Icebox: Re-examine how internal search referrals are handled by Clickstream - https://phabricator.wikimedia.org/T292435#10436574 (10JAllemandou) [09:11:22] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data Pipelines, 06Privacy Engineering, 07Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532#10436575 (10JAllemandou) [09:11:45] 06Data-Engineering, 06Research: Consider adding more namespaces to Clickstream dataset - https://phabricator.wikimedia.org/T296359#10436576 (10JAllemandou) [09:11:47] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data Pipelines, 06Privacy Engineering, 07Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532#10436577 (10JAllemandou) [09:12:01] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data Pipelines, 06Privacy Engineering, 07Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532#10436579 (10JAllemandou) 05Open→03Resolved This is done! Resolving. [11:15:19] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 13Patch-For-Review: Switch dumps 1.0 processes to use the analytics MariadB replicas (dbstore100[7-9]) - https://phabricator.wikimedia.org/T382947#10436809 (10BTullis) >>! In T382947#10436234, @Ma... [11:21:13] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 13Patch-For-Review: Switch dumps 1.0 processes to use the analytics MariadB replicas (dbstore100[7-9]) - https://phabricator.wikimedia.org/T382947#10436830 (10Marostegui) >>! In T382947#10436809, @B... [14:52:15] 10Data-Engineering (Q2 2024 October 1st - December 31th): Human pageviews potentially misclassified as automated - https://phabricator.wikimedia.org/T382713#10437375 (10JAllemandou) 05Open→03Declined I ran a query on raw data about this and found that the 647 automated queries for 2024-12-13 were run usi... [14:59:39] 06Data-Engineering, 10Data-Engineering-Wikistats: Add "Top used photos" metric - https://phabricator.wikimedia.org/T220485#10437386 (10mforns) I think there's a tag war between us and Herald... In any case, this data is available now as part of the Commons Impact Metrics dumps. See: https://wikitech.wikimedia.... [15:07:28] 06Data-Engineering, 10Data-Engineering-Wikistats: Add "Top used photos" metric - https://phabricator.wikimedia.org/T220485#10437408 (10Ottomata) There [[ https://wikimedia.slack.com/archives/C05H0JYT85V/p1736200895793169 | is a war ]], but now I know how to win. [15:12:23] 06Data-Engineering, 06Product-Analytics: webrequest dataset sets referer_class "unknown" instead of "external (search engine)" for origin-based referer values - https://phabricator.wikimedia.org/T383088#10437433 (10Ottomata) [15:20:40] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-content-history-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10437452 (10xcollazo) Today we discovered that the flink app was down, unkown for how l... [15:28:47] 06Data-Engineering: Some search entries in wmf.webrequest have their query appended to their uri_path - https://phabricator.wikimedia.org/T383135#10437479 (10AndrewTavis_WMDE) Thanks @Aklapper! 🙏 Adding #data-engineering as my best guess. Hope that this works and that others can add more specific tags for the te... [16:11:20] 06Data-Engineering, 10Dumps-Generation, 10Data-Platform-SRE (2024.11.30 - 2024.12.20): WE 5.4 KR - Hypothesis 5.4.6 - Q3 FY24/25 - Validate Dumps 1.0 compatibility with PHP 8.1 - https://phabricator.wikimedia.org/T382484#10437669 (10BTullis) [16:12:02] 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, 10MW-on-K8s, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10437672 (10BTullis) [16:26:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:26:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:11:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [17:11:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [17:11:44] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10MediaWiki-General, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#10437923 (10Ottomata) - Merged [[ http... [17:45:27] 06Data-Engineering, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, and 6 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10438043 (10Krinkle) >>! In T355837#10431981, @Michael wrote: > […] > > Thank you so much for... [18:06:35] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10MediaWiki-General, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#10438164 (10Ottomata) [[ https://gerri... [18:07:38] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10MediaWiki-General, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#10438165 (10Ottomata) [18:18:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:18:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:43:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:43:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [18:47:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:47:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:07:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:07:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:18:17] 06Data-Engineering, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, and 6 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10438413 (10Michael) >>! In T355837#10438043, @Krinkle wrote: >>>! In T355837#10431981, @Micha... [19:35:54] 06Data-Engineering, 10ActiveAbstract, 10Dumps-Generation: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10438498 (10VirginiaPoundstone) Seems like it's been quiet on the list servs so far. On February 7th, 2024 if it remains quiet, let's go ahead and stop producing them. L... [20:31:59] 06Data-Engineering, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, and 6 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10438710 (10Krinkle) >>! In T355837#10438413, @Michael wrote: > […] > If I search for `microti... [21:08:48] 06Data-Engineering, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, and 6 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10438797 (10colewhite) >>! In T355837#10438413, @Michael wrote: > Wait what? That needs to be... [21:39:12] 06Data-Engineering, 06Movement-Insights, 06Product-Analytics: webrequest dataset sets referer_class "unknown" instead of "external (search engine)" for origin-based referer values - https://phabricator.wikimedia.org/T383088#10438890 (10Mayakp.wiki) [21:57:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:57:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:02:03] 06Data-Engineering, 10ActiveAbstract, 10Dumps-Generation, 13Patch-For-Review: Undeploy and archive ActiveAbstract - https://phabricator.wikimedia.org/T382069#10438974 (10Ladsgroup) This patch would be the easiest way to stop producing the dumps, then the next step would be to remove all mentions of abstrac... [22:17:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:18:30] 06Data-Engineering: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175 (10VirginiaPoundstone) 03NEW [22:18:40] 06Data-Engineering: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10439027 (10VirginiaPoundstone) p:05Triage→03High [22:20:29] 06Data-Engineering: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10439029 (10VirginiaPoundstone) @Ahoelzl this is some support work that we need to do for WE 5.5. Deadline is end of January. Who from data engineering could pick this up? [22:22:07] 06Data-Engineering: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10439036 (10VirginiaPoundstone) [22:22:52] 06Data-Engineering: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10439040 (10VirginiaPoundstone) [22:24:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:24:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:39:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:39:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [23:06:36] FIRING: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [23:06:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag