[00:14:18] RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [00:14:18] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [00:16:03] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [00:16:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [04:16:03] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [04:16:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [08:16:03] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [08:16:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [08:31:36] 06Data-Engineering, 10MobileFrontend, 06Traffic: Add ismobile attribute to X-Analytics header - https://phabricator.wikimedia.org/T390924#10737454 (10Krinkle) [09:14:18] RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [09:14:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [09:15:48] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [09:15:48] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [09:25:48] FIRING: [3x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [09:25:49] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [09:27:56] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop afl_patrolled_by from abuse_filter_log in production - https://phabricator.wikimedia.org/T391056#10737699 (10FCeratto-WMF) [09:50:49] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [09:50:49] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [10:00:48] FIRING: [3x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [10:00:49] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [10:10:48] FIRING: [3x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [10:10:49] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [10:34:37] 06Data-Engineering, 06Data-Engineering-Radar, 06Traffic, 10MediaWiki-Platform-Team (Radar): Replicate current low-message alerting from VarnishKafka - https://phabricator.wikimedia.org/T391810 (10Fabfur) 03NEW [11:15:48] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [13:05:44] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 10EntitySchema, and 18 others: Add namespace descriptions for Special:NamespaceInfo in WMF-deployed extensions - https://phabricator.wikimedia.org/T373070#10738670 (10Nemoralis) [13:48:39] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [15:12:25] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Add metrics for monthly reconciles - https://phabricator.wikimedia.org/T388439#10739400 (10tchin) a:03tchin [15:21:28] 06Data-Engineering, 07Documentation: https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log should be on Wikitech - https://phabricator.wikimedia.org/T387878#10739455 (10Ottomata) HuH! You are very correct, they should! Have they always been on mw.org and we didn't notice? [15:21:45] 06Data-Engineering, 06Data-Platform-SRE, 07Documentation: https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log should be on Wikitech - https://phabricator.wikimedia.org/T387878#10739456 (10Ottomata) [15:30:37] 06Data-Engineering, 06Machine-Learning-Team, 06Research, 10Event-Platform: Emit revision revert risk scores as a stream and expose in EventStreams API - https://phabricator.wikimedia.org/T326179#10739547 (10Ottomata) Nice!!! https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To#EventSt... [16:32:56] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10MW-Interfaces-Team (MWI-Roadmap): DomainEvents - [Hypothesis] WE5.2.6 Event Broadcasting Discovery & Design - https://phabricator.wikimedia.org/T384874#10739968 (10Ottomata) [16:33:44] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10MW-Interfaces-Team (MWI-Roadmap): DomainEvents - [Hypothesis] WE5.2.6 Event Broadcasting Discovery & Design - https://phabricator.wikimedia.org/T384874#10739974 (10Ottomata) ===Briefly describe what was accomplished during the h... [16:33:53] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10MW-Interfaces-Team (MWI-Roadmap): DomainEvents - [Hypothesis] WE5.2.6 Event Broadcasting Discovery & Design - https://phabricator.wikimedia.org/T384874#10739978 (10Ottomata) p:05Triage→03High [16:35:20] 10Data-Engineering-Roadmap, 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10739980 (10Ottomata) I just closed out {T384874} in Asana. I also updated the task's description and added a commen... [16:42:42] (03PS1) 10Hasan Akgün (WMDE): Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) [16:43:01] (03CR) 10CI reject: [V:04-1] Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [16:48:39] RESOLVED: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [16:48:39] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [16:50:03] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [16:50:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [16:55:03] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [16:55:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [17:06:56] (03PS2) 10Hasan Akgün (WMDE): Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) [17:07:24] (03CR) 10CI reject: [V:04-1] Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [17:15:03] RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [17:15:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [17:15:38] (03PS3) 10Hasan Akgün (WMDE): Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) [17:16:01] (03CR) 10CI reject: [V:04-1] Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [17:17:50] (03PS4) 10Hasan Akgün (WMDE): Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) [17:18:05] (03CR) 10CI reject: [V:04-1] Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [17:19:17] (03PS5) 10Hasan Akgün (WMDE): Add Prometheus stats push [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) [18:10:18] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 06Infrastructure-Foundations: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10740623 (10odimitrijevic) [18:14:39] 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: [Trino] Connect to Google Sheets - https://phabricator.wikimedia.org/T385748#10740669 (10IAckerman-WMF) p:05Triage→03Medium [18:17:52] 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, and 2 others: [Trino] Develop procedure and scripting for Trino cluster maintenance. - https://phabricator.wikimedia.org/T386391#10740727 (10IAckerman-WMF) p:05Triage→03Medium [18:29:41] 06Data-Engineering, 06Discovery-Search, 06Infrastructure-Foundations, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10740803 (10Gehel) [18:29:46] 06Data-Engineering, 06Discovery-Search, 06Infrastructure-Foundations, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10740804 (10Gehel) p:05Triage→03High [18:44:07] (03CR) 10Mforns: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1129234 (https://phabricator.wikimedia.org/T386649) (owner: 10Dr0ptp4kt) [18:44:42] (03CR) 10Mforns: [V:03+2 C:03+2] Add 21 wikis for sqoop, mediawiki_history eligibility [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1129234 (https://phabricator.wikimedia.org/T386649) (owner: 10Dr0ptp4kt) [19:18:32] 06Data-Engineering, 06Discovery-Search, 06Infrastructure-Foundations, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10740954 (10bking) As a heads-up, we are migrating from Elastic->OpenSearch in T388610. Once that is... [19:32:17] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Modify table maintenance mechanism to support Iceberg's rewrite_position_delete_files() - https://phabricator.wikimedia.org/T391280#10740998 (10xcollazo) [[ https://gitlab.wikimedia.org/repos/data-engineering/air... [19:34:39] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Commons-Impact-Metrics, 13Patch-For-Review: [CIM] Skewed ranking with the top Editors monthly API - https://phabricator.wikimedia.org/T370470#10741013 (10Eevans) v1.0.12 of the Data Gateway has been deployed, [[ https://phabricator.wikimedia.org/T370470#1... [19:41:13] (03CR) 10Mforns: [V:03+2] Fix commons impact metrics all-wikis cassandra loading queries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1135077 (https://phabricator.wikimedia.org/T370470) (owner: 10Mforns) [19:46:28] !log deploying refinery for weekly train [19:46:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:07:57] !log finished deployment of refinery for regular deployment train [20:07:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:40:41] 10Data-Engineering-Jupyter: Jupyter spawner's list of Conda environments only updates when an environment is spawned - https://phabricator.wikimedia.org/T391894 (10nshahquinn-wmf) 03NEW [21:06:29] !log re-running Commons impact metrics cassandra loading for all top endpoints since the beginning of Commons Impact Metrics time. [21:06:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:45:10] 06Data-Engineering, 06Discovery-Search, 06Infrastructure-Foundations, 10Data-Platform-SRE (2025-04-12 - 2025-05-02): Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#10741570 (10Volans) @bking thanks for the summary. Do you have a timeline for the completion the pro... [23:24:21] 06Data-Engineering, 10Data-Engineering-Jupyter: Upgrade to JupyterLab ≥ 4.2 - https://phabricator.wikimedia.org/T391905 (10nshahquinn-wmf) 03NEW [23:24:33] 06Data-Engineering, 10Data-Engineering-Jupyter: Jupyter spawner's list of Conda environments only updates when an environment is spawned - https://phabricator.wikimedia.org/T391894#10741744 (10nshahquinn-wmf)