[08:31:14] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11762967 (10APizzata-WMF) Also `SLIS` results look like intended: ` spark.sql( """ SELECT tag... [08:47:09] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add overwrite to check_bad_parsing - https://phabricator.wikimedia.org/T421677 (10APizzata-WMF) 03NEW [09:00:51] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11763107 (10dcausse) @APizzata-WMF sorry I did not spot this earlier but the values `["970"]` is not correct,... [09:18:05] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:23:05] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:34:21] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Investigate Gobblin failures - https://phabricator.wikimedia.org/T419436#11763242 (10JAllemandou) More failures from March 27th to March 30th: Real Error: * 2x Error with file availablity in task pod. Pos... [09:51:42] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11763303 (10APizzata-WMF) Hey @dcausse I must have misunderstood and thought the `exists|` part could be remov... [09:53:49] !log Test Kitchen edge-unique experiments (poll 45906) - adds: synth-aa-test-traffic-impact-2, synth-aa-test-traffic-impact-1, synth-aa-test-traffic-impact-3; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [09:53:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:04:53] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11763369 (10dcausse) >>! In T368987#11763303, @APizzata-WMF wrote: > Hey @dcausse I must have misunderstood an... [10:17:46] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11763441 (10dcausse) I think one of the cause of this confusion is the current contract between ALIS/SLIS and... [10:27:37] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11763474 (10Raine) >>! In T419980#11736923, @JAllemandou wrote: > The sqoop jobs fleet starts at midnight UTC on the first of the mon... [10:33:14] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11763489 (10JAllemandou) > @JAllemandou We will postpone the upgrade (i.e. the table hot swap). However, the new tables are already c... [10:37:24] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11763502 (10Raine) >>! In T419980#11763489, @JAllemandou wrote: >> @JAllemandou We will postpone the upgrade (i.e. the table hot swap... [10:42:06] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Visualizing inconsistencies and reconciles via Superset - https://phabricator.wikimedia.org/T420787#11763538 (10gmodena) cc / @lerickson @trueg We have a similar problem to solve with WDQS getting out of sync. We have a streaming update pipeline that pr... [10:58:47] 06Data-Engineering, 10Observability-Alerting, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Reduce IRC/alert noise associated with monitor_refine_ systemd timers from alertmanager - https://phabricator.wikimedia.org/T337052#11763607 (10BTullis) 05Open→03Resolved a:03BTullis I propose to resolve th... [11:51:51] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11763867 (10APizzata-WMF) I have created and validated all my update commands, will shortly run them all and p... [12:02:29] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11763939 (10dcausse) >>! In T368987#11763867, @APizzata-WMF wrote: > I have created and validated all my updat... [12:39:38] 06Data-Engineering, 06Data-Engineering-Radar, 10FR-Tech-Analytics: Integrate airflow-fr-tech-ops into Bitu - https://phabricator.wikimedia.org/T421703 (10MoritzMuehlenhoff) 03NEW [12:40:17] 06Data-Engineering, 06Data-Engineering-Radar, 10Bitu, 10FR-Tech-Analytics, 06Infrastructure-Foundations: Integrate airflow-fr-tech-ops into Bitu - https://phabricator.wikimedia.org/T421703#11764138 (10MoritzMuehlenhoff) p:05Triage→03Medium a:05brouberol→03MoritzMuehlenhoff [13:39:53] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions, 13Patch-For-Review: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11765168 (10APizzata-WMF) Unfortunately, due to permission on the `delta` and `full` tab... [13:45:00] (03PS9) 10Snwachukwu: Extend mediarequest Cassandra loads with poster/plays for video-requests API [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1250005 (https://phabricator.wikimedia.org/T415202) [14:30:12] !log Test Kitchen edge-unique experiments (poll 46728) - adds: none; removes: logged-out-retention-round3; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:30:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:56:01] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Backfill datasets affected by Nov 2025 automated traffic incident - https://phabricator.wikimedia.org/T421735 (10mforns) 03NEW [14:56:57] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Backfill datasets affected by Nov 2025 automated traffic incident - https://phabricator.wikimedia.org/T421735#11765816 (10mforns) Here's the [[ https://docs.google.com/document/d/1XDgDEW8-BgqixTviyB8D4kH6-i0g9TEEcYmt8B6FLuM/edit?usp=sharing | backfilling... [14:57:27] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06Reader Growth Team, 06Wikipedia-Android-App-Backlog, and 3 others: Add page_id and namespace to X-Analytics header in Mobile App requests (2025 remake) - https://phabricator.wikimedia.org/T409358#11765817 (10Ottomata) I t... [14:57:34] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06Reader Growth Team, 06Wikipedia-Android-App-Backlog, and 3 others: Add page_id and namespace to X-Analytics header in Mobile App requests (2025 remake) - https://phabricator.wikimedia.org/T409358#11765818 (10Ottomata) [15:09:17] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0: Consider updating our heuristics for media type classification in AQS / wikistats - https://phabricator.wikimedia.org/T419882#11765890 (10Snwachukwu) AQS / wikistats classifies media files using a simple file-extension-to-type mapping defined... [15:10:46] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Investigate Gobblin failures - https://phabricator.wikimedia.org/T419436#11765923 (10Ottomata) > FailureRequest to uri https://meta.wikimedia.org/w/api.php?format=json&action=streamconfigs&all_settings=true... [15:36:55] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Task Tries and Logs for Airflow DAGs sometimes unavailable - https://phabricator.wikimedia.org/T419162#11766106 (10JAllemandou) I think I found the culprit for this. In the stack trace, the error happens at... [15:45:58] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0: Use transcoding signal to resolve ambiguous extensions - https://phabricator.wikimedia.org/T421743 (10Snwachukwu) 03NEW [16:00:47] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions, 13Patch-For-Review: Add an Image: filtering by suggestion "kind" or "confidence" - https://phabricator.wikimedia.org/T368987#11766282 (10APizzata-WMF) Tested in the dev env and the results are good: ` spark.sql("... [18:01:49] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026-03-27 - 2026-04-17): Investigate Gobblin failures - https://phabricator.wikimedia.org/T419436#11766900 (10JAllemandou) > This is a client side timeout, yes? I wonder what our client timeout is... We use Apache HTTP version 4.5... [19:31:06] !log Test Kitchen edge-unique experiments (poll 47622) - adds: logged-out-retention-round5; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [19:31:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:58:30] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Event-Platform: Fix PyFlink log levels - https://phabricator.wikimedia.org/T419997#11767448 (10AKhatun_WMF) Another log related issue is that [flink logger "extra" fields](https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/... [20:36:57] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11767627 (10Raine) @Marostegui or @Ladsgroup, when can we do the table swap? Would one of you be available for the swap (coordinated... [21:00:31] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11767761 (10Raine) [21:01:22] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11767768 (10Raine) [21:55:17] 06Data-Engineering: Requesting Kerberos access for matmarex - https://phabricator.wikimedia.org/T421783 (10matmarex) 03NEW [22:02:49] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 21): Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254#11767998 (10Sfaci) [22:24:15] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Add support for variables to DbtSkeinOperator - https://phabricator.wikimedia.org/T421789 (10amastilovic) 03NEW