[00:50:15] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12482472 (10Dzahn) [00:54:32] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12482478 (10Dzahn) [00:54:58] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12482482 (10Dzahn) 05Open→03In progress [00:55:20] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12482484 (10Dzahn) [00:58:09] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12482489 (10Dzahn) I sent an email for the out-of-band key verification. [02:40:05] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:20:05] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:30:05] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:00:05] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:05:05] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:15:05] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:25:20] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:55:20] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [04:59:05] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:29:05] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [05:34:05] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:04:33] 06Data-Engineering, 06Data-Platform-SRE, 06Wikimedia Enterprise: Request for a dedicated Airflow instance for Commercial Partnerships (Enterprise) - https://phabricator.wikimedia.org/T440539#12482996 (10brouberol) Could you expand a bit of what type of infrastructure the instance would need to have access to... [11:13:31] 06Data-Engineering: Requesting Kerberos identity for rsilvola - https://phabricator.wikimedia.org/T442695 (10Rsilvola) 03NEW [12:13:33] FIRING: [2x] WebrequestPageviewHighKafkaConsumerLag: ... [12:13:39] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [12:14:59] RESOLVED: [2x] WebrequestPageviewHighKafkaConsumerLag: ... [12:15:04] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [12:59:02] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12484143 (10APizzata-WMF) Here are all the MRs: - [[ https://gitlab.wikimedia.org/repos/data-engineering/node-rdkafka-factory/-/... [13:01:33] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604#12484180 (10JAllemandou) I have continued the research on Gravitino + JDBC wi... [15:38:03] 06Data-Engineering, 06Data-Platform-SRE: Decide upon and deploy batch scheduling for Spark on dse-k8s - https://phabricator.wikimedia.org/T442726 (10BTullis) 03NEW [16:08:18] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12485244 (10Milimetric) [16:10:47] 06Data-Engineering: MediaFileUrlInfo.getTranscoding() never returns movie_480; all movie transcodes of 240 px and up are labelled movie_240_479 - https://phabricator.wikimedia.org/T442316#12485269 (10Milimetric) a:03dr0ptp4kt [16:10:59] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): MediaFileUrlInfo.getTranscoding() never returns movie_480; all movie transcodes of 240 px and up are labelled movie_240_479 - https://phabricator.wikimedia.org/T442316#12485272 (10Milimetric) [16:12:17] 06Data-Engineering: transcoded video url's on commons are not recognized by MediaFileUrlParser - https://phabricator.wikimedia.org/T442320#12485286 (10Milimetric) a:03dr0ptp4kt [16:12:28] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): transcoded video url's on commons are not recognized by MediaFileUrlParser - https://phabricator.wikimedia.org/T442320#12485289 (10Milimetric) [16:14:45] 06Data-Engineering, 10Event-Platform: eventlogging_EditAttemptStep sometimes misses dt field - https://phabricator.wikimedia.org/T442089#12485309 (10Milimetric) a:03amastilovic [16:14:56] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Event-Platform: eventlogging_EditAttemptStep sometimes misses dt field - https://phabricator.wikimedia.org/T442089#12485312 (10Milimetric) [16:16:36] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop CentralAuth globalnames table in wmf production - https://phabricator.wikimedia.org/T442241#12485348 (10Milimetric) [16:18:00] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485354 (10Milimetric) a:03tchin [16:18:23] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485356 (10Milimetric) [16:19:35] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10AQS2.0: Review and adapt wikistats / AQS video analytics to the introduction of MPEG-DASH - https://phabricator.wikimedia.org/T419879#12485381 (10Milimetric) [16:19:52] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Epic: Epic: Migrate uses of webrequest to webrequest_v2 - https://phabricator.wikimedia.org/T441283#12485389 (10Milimetric) [16:20:05] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Epic: Epic: Migrate uses of webrequest to webrequest_v2 - https://phabricator.wikimedia.org/T441283#12485392 (10Milimetric) a:03mforns [16:24:52] 06Data-Engineering, 10Dumps-Generation, 10MediaWiki-Core-Snapshots: Importing the same page twice creates duplicate revisions when its text contains \r\n - https://phabricator.wikimedia.org/T440235#12485421 (10Milimetric) a:03xcollazo [16:25:09] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Dumps-Generation, 10MediaWiki-Core-Snapshots: Importing the same page twice creates duplicate revisions when its text contains \r\n - https://phabricator.wikimedia.org/T440235#12485423 (10Milimetric) [16:27:33] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Engineering-Dashiki: Pingback dashboard missing values after January 2026 - https://phabricator.wikimedia.org/T440089#12485428 (10Milimetric) Following up on T440089, looks like the airflow job never ran after the change (cc @xcollazo ) [16:27:35] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Engineering-Dashiki: Pingback dashboard missing values after January 2026 - https://phabricator.wikimedia.org/T440089#12485432 (10Milimetric) [16:28:39] 06Data-Engineering: Add width bands to the mediarequests API - https://phabricator.wikimedia.org/T439420#12485448 (10Milimetric) a:03GGoncalves-WMF might be easily done with the other mediarequests changes [16:35:13] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos identity for rsilvola - https://phabricator.wikimedia.org/T442695#12485483 (10Milimetric) This is a follow-up to {T411624}, moving to our radar and tagging SRE. [16:53:28] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): User Acceptance tests in staging - https://phabricator.wikimedia.org/T437670#12485631 (10Ottomata) > ❓Cancelling a API call should return CANCELLED. It looked like LAC still computed for the full 10 seconds and reported DEADLINE_EXCEEDED afterwards.... [16:54:06] (03PS3) 10Aqu: dev-metrics: optimize git_interaction writes [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) [16:57:32] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Run Load test - https://phabricator.wikimedia.org/T441494#12485664 (10Ottomata) > lower the CPU request to about 1.25 can we make cpu requests of 1.25 and cpu limit of 2? Either way, we might as well do this 1.25 now eh? [16:59:57] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485671 (10Jdforrester-WMF) This is probably not all being done by @tchin for all ~22 outstanding... [17:01:34] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485675 (10taavi) [17:02:01] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12485683 (10Ottomata) Nice thank you! Reviewing. High level question here in phab: Do we need a `wiki_id` label for this SLI?... [17:17:52] (03PS4) 10Aqu: dev-metrics: optimize git_interaction writes [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) [17:28:56] (03CR) 10Aqu: dev-metrics: optimize git_interaction writes (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1359279 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [17:37:52] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Run Load test - https://phabricator.wikimedia.org/T441494#12485841 (10AKhatun_WMF) Wanted to see what happens after some traffic starts flowing in. Will change to 1.25 and observe instead. Same thing. [17:57:43] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485957 (10karapayneWMDE) [17:58:16] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12485983 (10APizzata-WMF) I was thinking to have for the 3 major wikis: `commonswiki, wikidatawiki, enwiki`, you can find this i... [18:24:09] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Blunderbuss: Have analytics_test artifacts go to analytics_test HDFS cluster - https://phabricator.wikimedia.org/T442190#12486106 (10atsuko) Fixed permissions on `analytics-test-hadoop`, ` sudo -u hdfs kerberos-run-command hdfs hdf... [18:26:37] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12486115 (10herron) >>! In T440968#12485683, @Ottomata wrote: > Hm, maybe 200000 too much? Let's ask o11y. Hello! Yes lets t... [18:53:31] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Dumps-Generation, 13Patch-For-Review: mediawiki-content-pipelines: Gate the File Exports on the ingest of the monthly reconcile - https://phabricator.wikimedia.org/T442415#12486207 (10xcollazo) >>! In T442415#12486205, @CodeReviewBot wrote: > xc... [19:03:15] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Essential-Work: [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604#12486240 (10Ottomata) ^ +1, all things pointing to HMS 4. I think we go fo... [19:11:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12486255 (10Ottomata) @APizzata-WMF yeah let's just drop `wiki_id` from labels for now then. If we need it we can consider addin... [20:15:47] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12486613 (10Dzahn) [x] verified key out-of-band [20:16:02] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12486616 (10Dzahn) [20:33:07] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12486678 (10Dzahn) Welcome @cfoster ! Confirming we have verified your key out of band. Almost all boxes are... [20:40:05] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos identity for rsilvola - https://phabricator.wikimedia.org/T442695#12486705 (10Dzahn) If Kerberos is needed it's a request for a an upgrade from level 1 or 2 (which was not specified in the original task) to level 3. (this is T405517). th... [20:47:03] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12486719 (10cfoster) @mpopov is out of office through this week - is there an alternate approver we could con... [20:53:29] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12486743 (10Dzahn) Unfortunately only one is listed. Usually we recommend multiple ones to avoid this kind of... [21:15:44] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting Kerberos identity for rsilvola - https://phabricator.wikimedia.org/T442695#12486830 (10Rsilvola) I now need level 3 for data lake access. [21:18:31] 06Data-Engineering, 06Data-Platform-SRE, 06Wikimedia Enterprise: Request for a dedicated Airflow instance for Commercial Partnerships (Enterprise) - https://phabricator.wikimedia.org/T440539#12486854 (10HShaikh) The infra we would need access to would be the same or similar to what the airflow-platform-eng... [21:29:44] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Include v2 dumps into mirrors rsync configuration - https://phabricator.wikimedia.org/T437463#12486929 (10xcollazo) 05In progress→03Resolved [21:51:30] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12487016 (10nshahquinn-wmf) @APizzata-WMF I was going to agree and close this ticket, but, unfortunately, someth... [21:57:17] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: mediawiki-content-pipelines: Add a Spark test harness - https://phabricator.wikimedia.org/T441886#12487038 (10xcollazo) The Spark test harness is done, and the `pytest` CI job runs it. The fixes for the issues it found are in rel... [22:14:13] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12487071 (10AKhatun_WMF) ` Halley Coplin  ... Yes, let's proceed with always returning latest for now. To properly support stable, we need to go... [22:18:05] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Debug errors in prod - https://phabricator.wikimedia.org/T442775 (10AKhatun_WMF) 03NEW [22:18:53] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Debug errors in prod - https://phabricator.wikimedia.org/T442775#12487093 (10AKhatun_WMF) [22:26:27] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12487095 (10nshahquinn-wmf) Query showing the issue: ` SELECT CASE WHEN moderator_action... [23:51:45] 06Data-Engineering, 10ChangeProp, 10EventStreams, 06MediaWiki-Engineering, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12487406 (10Jdforrester-WMF)