[00:52:04] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Product-Analytics: dbt: Weekly scheduling dependency checks only check the first date of the week - https://phabricator.wikimedia.org/T436744#12320619 (10amastilovic) This should now be fixed - it took a little while longer because the fix touched on... [01:16:12] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653#12320638 (10Dzahn) a:05thcipriani→03None [01:17:50] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653#12320648 (10Dzahn) [07:11:00] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): User and team Hive database cleanup tracking for Sept 2026 - https://phabricator.wikimedia.org/T437733#12320941 (10APizzata-WMF) [07:40:35] 06Data-Engineering, 10Data Pipelines, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Airflow 3: remove Amazon provider warnings from CLI startup - https://phabricator.wikimedia.org/T437981 (10RKemper) 03NEW [07:43:50] 06Data-Engineering, 10Data Pipelines, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Airflow 3: remove Amazon provider warnings from CLI startup - https://phabricator.wikimedia.org/T437981#12321042 (10RKemper) [07:50:29] 06Data-Engineering, 10Data Pipelines, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Airflow 3: remove Amazon provider warnings from CLI startup - https://phabricator.wikimedia.org/T437981#12321058 (10RKemper) [08:11:14] 06Data-Engineering, 10Data Pipelines, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Airflow 3: remove Amazon provider warnings from CLI startup - https://phabricator.wikimedia.org/T437981#12321084 (10RKemper) I've uploaded a patch that should remove 80+% of the noise in the original ti... [08:37:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Persistence-Design-Review, 10Event-Platform, 13Patch-For-Review: [Data Persistence Design Review] Relative trending pages - https://phabricator.wikimedia.org/T432352#12321231 (10JMonton-WMF) @pmiazga did a review of https://gerrit.wikimedia.... [08:50:00] 06Data-Engineering, 10Dumps-Generation: Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988 (10brouberol) 03NEW p:05Triage→03High [08:50:08] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321300 (10brouberol) [08:50:09] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321301 (10brouberol) 05Open→03In progress [09:05:52] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321373 (10brouberol) I was finally able to observe something that explains the slowness, after being plagued by it for we... [09:12:27] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321386 (10brouberol) Another type of pod mounting the ceph dumps PVC are the pods cleaning up the temporary files from Ce... [09:13:20] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321389 (10brouberol) Indeed, I'm seeing the root folderi in the volume being owned by gid 900 right now: ` www-data@media... [09:20:11] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Implement brownout logic for dumps v1 on dumps web interface - https://phabricator.wikimedia.org/T437461#12321423 (10Gehel) [09:20:45] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Implement brownout logic for dumps v1 on dumps web interface - https://phabricator.wikimedia.org/T437461#12321425 (10GGoncalves-WMF) We'd like this to be done ideally still in September, but early October is reasonab... [09:23:41] 06Data-Engineering, 10Data Pipelines, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Airflow 3: remove Amazon provider warnings from CLI startup - https://phabricator.wikimedia.org/T437981#12321443 (10atsuko) Thanks for cleaning it up! We were sort-of soft-blocked yesterday with image r... [09:33:47] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Figure out the biggest HDFS consumers over time - https://phabricator.wikimedia.org/T437605#12321496 (10KCVelaga_WMF) @xcollazo ah right, this slipped through. We can reduce the data delete from 180 to 90 days (standard retention for raw data), as all th... [09:44:38] (03PS1) 10Joal: Update changelog.md to release v0.3.26 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1341715 [09:48:11] (03CR) 10JavierMonton: [C:03+1] Update changelog.md to release v0.3.26 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1341715 (owner: 10Joal) [09:49:45] (03CR) 10Joal: [C:03+2] "Merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1341715 (owner: 10Joal) [09:49:52] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321600 (10brouberol) I'm now going to clear [airflow-dumps.wikimedia.org / mediawiki_dumps_sql_xml_large_a_to_z_full / sc... [09:50:31] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321601 (10brouberol) ` brouberol@deploy1003:~$ k get pod -o json cleanup-enwiki-tempfiles-t19iwct | jq .spec.securityCont... [09:59:28] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321662 (10brouberol) It took 2 attempts to `chown -R` the whole volume: ` NAME... [10:00:59] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321673 (10brouberol) Sync pods now also use `fsGroup=33`: ` brouberol@deploy1003:~$ k get pod -o json enwiki-sql-xml-enwi... [10:02:20] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321691 (10brouberol) Same for the monitor pods ` brouberol@deploy1003:~$ k get pod -o json mediawiki-sql-xml-dumps-monito... [10:02:25] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321692 (10brouberol) I'm calling this done! [10:02:36] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321694 (10brouberol) 05In progress→03Resolved [10:03:42] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12321703 (10brouberol) ` www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-9rrdb:/$ ls -alh /mnt/dumpsdata/ total... [10:06:12] (03Merged) 10jenkins-bot: Update changelog.md to release v0.3.26 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1341715 (owner: 10Joal) [10:08:26] Starting build #73 for job analytics-refinery-maven-release [10:37:32] Project analytics-refinery-maven-release build #73: 09SUCCESS in 29 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/73/ [10:52:50] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Fundraising-Backlog, and 2 others: Find a better way for WMDE to get impression counts for their banners - https://phabricator.wikimedia.org/T243092#12322007 (10AbbanWMDE) 05Open→03Resolved [10:57:24] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users level 3 for derenrich - https://phabricator.wikimedia.org/T437668#12322034 (10hnowlan) Pinging @HSwan-WMF for manager approval [11:34:17] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Fix iceberg table location in hive metastore - https://phabricator.wikimedia.org/T408939#12322285 (10JAllemandou) 05Declined→03Open Re-opening after having done some investigations: Andrew is right, as long as we change only the location to be fully-... [12:10:35] (03PS1) 10Joal: Update iceberg table location to fully qualified [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1341854 (https://phabricator.wikimedia.org/T408939) [12:16:10] (03PS1) 10Joal: Remove hive scripts for aqs table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1341858 (https://phabricator.wikimedia.org/T437883) [12:19:20] (03PS1) 10Joal: Remove hive interlanguage and session_length [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1341859 (https://phabricator.wikimedia.org/T438021) [13:25:38] (03CR) 10Xcollazo: "This is risky." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1341854 (https://phabricator.wikimedia.org/T408939) (owner: 10Joal) [13:26:46] (03CR) 10Xcollazo: "(and by "apply it", I mean running the ALTER version of the change)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1341854 (https://phabricator.wikimedia.org/T408939) (owner: 10Joal) [13:28:13] (03CR) 10Xcollazo: [C:03+1] "+1 in principle, but given discussion in 1341854, perhaps let's hold this patchset for a bit?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1341858 (https://phabricator.wikimedia.org/T437883) (owner: 10Joal) [13:29:04] (03CR) 10Xcollazo: [C:03+1] Remove hive interlanguage and session_length [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1341859 (https://phabricator.wikimedia.org/T438021) (owner: 10Joal) [14:01:42] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12322970 (10AKhatun_WMF) > the idea was to return null when the reference/citation counts are not available in cache to prevent Attribution API from... [14:04:24] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12322999 (10brouberol) I'm re-opening this as I've observed the root volume dir owned by gid 900 again: ` www-data@medi... [14:04:36] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12323003 (10brouberol) 05Resolved→03In progress [14:07:04] (03CR) 10Joal: "That was actually my plan before merging this. Let me do the test with `aqs_hourly`, and I'll come back on the task." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1341854 (https://phabricator.wikimedia.org/T408939) (owner: 10Joal) [14:14:49] 06Data-Engineering, 06Data-Engineering-Radar, 06App Growth, 10Data-Platform, and 7 others: Image Suggestions uses AI-generated images from Commons when adding images on English Wikipedia - https://phabricator.wikimedia.org/T422513#12323096 (10Seddon) [14:18:26] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653#12323130 (10hnowlan) Access granted! [14:33:55] 06Data-Engineering, 10GitLab, 06Release-Engineering-Team: Trusted-runner access for developer-activity-pipelines (publish a container image) - https://phabricator.wikimedia.org/T438042 (10Antoine_Quhen) 03NEW [14:56:47] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12323406 (10GGoncalves-WMF) Just a suggestion: if we go in that direction (and I think we should!), it might make more sense to also not change the... [15:37:19] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Handling Flagged Revs for Reference Counts in Attribution API - https://phabricator.wikimedia.org/T435557#12323698 (10AKhatun_WMF) > it might make more sense to also not change the default behavior of LAC yet, and instead have Attribution API set a timeo... [15:39:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel - https://phabricator.wikimedia.org/T425734#12323720 (10xcollazo) >>! In T425734#12323699, @CodeReviewBot wro... [15:47:51] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12323787 (10Gehel) [15:48:36] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12323806 (10Gehel) [15:49:44] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 05acl*data-engineering, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 07Essential-Work: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12323819 (10Gehel) [15:51:29] 06Data-Engineering, 10Data-Platform, 06Data-Platform-SRE, 07Epic, 07OKR-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12323823 (10Gehel) p:05Triage→03Medium [15:54:26] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10SRE-SLO: page_change SLO windows - https://phabricator.wikimedia.org/T438054 (10APizzata-WMF) 03NEW [16:05:56] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653#12323954 (10hnowlan) 05Open→03Resolved a:03hnowlan [16:09:15] 06Data-Engineering: Build an HDFS growth dashboard based on wmf.hdfs_usage - https://phabricator.wikimedia.org/T438056 (10xcollazo) 03NEW [16:10:44] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: User and team Hive database cleanup tracking for Sept 2026 - https://phabricator.wikimedia.org/T437733#12323995 (10xcollazo) Opened {T438056} for consideration as #essential-work Such a pipeline would give us good visibility on rap... [16:10:57] 06Data-Engineering, 07Essential-Work: Build an HDFS growth dashboard based on wmf.hdfs_usage - https://phabricator.wikimedia.org/T438056#12324000 (10xcollazo) [17:06:47] 06Data-Engineering, 06Data-Engineering-Radar, 06Research: mwparserfromhtml errors - https://phabricator.wikimedia.org/T437669#12324348 (10AKhatun_WMF) - Infobox: I don't have opinions, anything that seems like the right usage should be implemented. I thought I should raise what I thought was a bug. Your solu... [18:01:16] (03PS2) 10Milimetric: sqoop-mediawiki-tables - add a shard override [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1335154 (https://phabricator.wikimedia.org/T431476) (owner: 10Ottomata) [19:01:26] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Calculate active UWERs on a weekly cadence - https://phabricator.wikimedia.org/T437961#12324837 (10nshahquinn-wmf) [19:02:04] 14Data-Engineering (Q4 FS25/26 April 1st - June 30st), 05Metrics-Sprint-2026-2027: Heartbeat - Active UWERs - https://phabricator.wikimedia.org/T426420#12324838 (10nshahquinn-wmf) a:05Snwachukwu→03None [19:47:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:52:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [19:55:33] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12325019 (10brouberol) ` www-data@mediawiki-dumps-legacy-dump-toolbox-59cc97c9bf-9rrdb:/$ ls -alh /mnt/dumpsdata/ total 4.0... [19:55:40] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Dumps pods sometimes take more than 5 minutes to start - https://phabricator.wikimedia.org/T437988#12325021 (10brouberol) 05In progress→03Resolved [20:16:23] PROBLEM - statsv Varnishkafka log producer on cp1114 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [20:17:23] RECOVERY - statsv Varnishkafka log producer on cp1114 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [20:28:45] 06Data-Engineering, 06Data-Engineering-Radar, 06Research, 13Patch-For-Review: mwparserfromhtml errors - https://phabricator.wikimedia.org/T437669#12325123 (10Isaac) Excellent - if you wouldn't mind taking a look: https://gitlab.wikimedia.org/repos/research/html-dumps/-/merge_requests/44. If you want me to... [21:16:37] 06Data-Engineering, 06Data-Engineering-Radar, 06Research, 13Patch-For-Review: mwparserfromhtml errors - https://phabricator.wikimedia.org/T437669#12325233 (10AKhatun_WMF) TY! And yes, a version bump would be great, [21:38:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:43:54] 06Data-Engineering, 06Data-Engineering-Radar, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: REST API vNext: Create dedicated REST shared-schema repository - https://phabricator.wikimedia.org/T432799#12325326 (10AGhirelli-WMF) 05In progress→03Resolved [22:08:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly