[00:17:54] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence: Data Persistence Design Review: Year in Review (YiR) - https://phabricator.wikimedia.org/T401260#11127018 (10Ottomata) a:05mforns→03Ottomata [06:23:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [06:28:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:52:59] 06Data-Engineering, 06Data-Engineering-Radar, 10AbuseFilter, 06DBA, 07Schema-change-in-production: Add default value for afl_ip and remove default value for afl_ip_hex in abuse_filter_log table - https://phabricator.wikimedia.org/T401906#11127822 (10FCeratto-WMF) [10:37:13] (03CR) 10Joal: "Questions about the pageview_actor integration, the rest looks good :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1163417 (owner: 10Mforns) [10:50:49] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159 (10AndrewTavis_WMDE) 03NEW [11:00:21] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11127971 (10AndrewTavis_WMDE) [11:08:33] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11127983 (10AndrewTavis_WMDE) [11:22:51] (03CR) 10Mforns: Improve the automated traffic detection pipeline (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1163417 (owner: 10Mforns) [11:30:49] (03PS6) 10Mforns: Improve the automated traffic detection pipeline [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1163417 [12:09:28] (03CR) 10Joal: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1163417 (owner: 10Mforns) [12:32:02] 06Data-Engineering, 06Data-Engineering-Radar, 10ConfirmEdit (CAPTCHA extension), 10MediaWiki-extensions-Campaigns, and 3 others: Send hCaptcha API response data to event platform - https://phabricator.wikimedia.org/T379179#11128264 (10kostajh) 05Stalled→03In progress [12:44:53] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11128285 (10Zabe) 05Stalled→03Open [12:52:45] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10service-utils, 10Event-Platform: Migrate and re-deploy eventgate-wikimedia using new service-utils - https://phabricator.wikimedia.org/T403169 (10tchin) 03NEW [12:59:02] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform: Add user-agent to http calls from eventgate-wikimedia - https://phabricator.wikimedia.org/T403171 (10tchin) 03NEW [13:02:25] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform: Add user-agent to http calls from eventgate-wikimedia - https://phabricator.wikimedia.org/T403171#11128362 (10tchin) [14:07:36] (03PS7) 10Mforns: Improve the automated traffic detection pipeline [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1163417 [14:10:50] (03PS8) 10Mforns: Improve the automated traffic detection pipeline [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1163417 [14:11:38] (03CR) 10Joal: [C:03+2] Improve the automated traffic detection pipeline (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1163417 (owner: 10Mforns) [14:19:01] (03CR) 10Mforns: [V:03+2] "Tested!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1163417 (owner: 10Mforns) [14:21:53] !log deploying refinery for bot-detection heuristic change [14:21:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:37:08] !log Deployed refinery onto HDFS [14:37:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:37:24] !lof Restart webrequest_actor jobs with new heuristic [14:37:28] !log Restart webrequest_actor jobs with new heuristic [14:37:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:43:51] (03PS1) 10Vgutierrez: Add ja3n X-Analytics sub-field [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1182855 (https://phabricator.wikimedia.org/T400270) [14:49:08] 06Data-Engineering, 06Data-Engineering-Radar, 10ConfirmEdit (CAPTCHA extension), 10MediaWiki-extensions-Campaigns, and 4 others: Send hCaptcha API response data to event platform - https://phabricator.wikimedia.org/T379179#11128903 (10kostajh) [16:06:36] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Movement-Insights, 06Traffic: NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#11129327 (10mforns) In the last couple hours, we've deployed the improvements to the automated traffic detection.... [16:08:26] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11129336 (10Ladsgroup) [16:15:14] (03PS1) 10Aleksandar Mastilovic: Set cl_to and cl_collation columns in categorylinks to NULL [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1182877 (https://phabricator.wikimedia.org/T397923) [16:17:26] (03CR) 10Ottomata: "LGTM! Thank you." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1182877 (https://phabricator.wikimedia.org/T397923) (owner: 10Aleksandar Mastilovic) [16:24:54] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] Set cl_to and cl_collation columns in categorylinks to NULL [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1182877 (https://phabricator.wikimedia.org/T397923) (owner: 10Aleksandar Mastilovic) [17:54:13] 06Data-Engineering, 06Data-Engineering-Icebox: MW REST API Historical Data Endpoint Needs - https://phabricator.wikimedia.org/T240387#11129722 (10Ottomata) 05Open→03Declined Being bold and declining this old task. [18:35:06] 06Data-Engineering, 10Research-engineering: Add analytics-research user to stat boxes - https://phabricator.wikimedia.org/T403207 (10fkaelin) 03NEW [18:40:50] 06Data-Engineering, 06Data-Engineering-Radar, 10ConfirmEdit (CAPTCHA extension), 10MediaWiki-extensions-Campaigns, and 4 others: Send hCaptcha API response data to event platform - https://phabricator.wikimedia.org/T379179#11129831 (10kostajh) [18:40:52] 06Data-Engineering, 06Data-Engineering-Radar, 10ConfirmEdit (CAPTCHA extension), 10MediaWiki-extensions-Campaigns, and 4 others: Send hCaptcha API response data to event platform - https://phabricator.wikimedia.org/T379179#11129832 (10kostajh) 05In progress→03Resolved [18:46:04] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Hack unique_devices_per_domain recreating `.m` subdomain use `x_analytics` `ismobile` value - https://phabricator.wikimedia.org/T401666#11129843 (10Mayakp.wiki) > Are you ok with us taking advantage of this breaking change to remove both the druid dataso... [19:00:05] 06Data-Engineering-Radar, 10HaproxyKafka, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Shutdown varnishkafka webrequest instances - https://phabricator.wikimedia.org/T393772#11129887 (10ssingh) @Fabfur: I think this all done and the alerts have been removed as well. Confirming: can we close th... [19:04:03] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 13Patch-For-Review: Adapt Sqoop for categorylinks schema change - https://phabricator.wikimedia.org/T397923#11129897 (10amastilovic) 05In progress→03Resolved [19:19:30] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Hack unique_devices_per_domain recreating `.m` subdomain use `x_analytics` `ismobile` value - https://phabricator.wikimedia.org/T401666#11129941 (10JAllemandou) >>! In T401666#11129843, @Mayakp.wiki wrote: >> Are you ok with us taking advantage of this b... [19:21:49] 06Data-Engineering, 10Multi-Content-Revisions, 07Schema-change: Replace page.page_content_model with id reference to content_models table - https://phabricator.wikimedia.org/T403211 (10Umherirrender) 03NEW [19:51:55] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Optimize metrics computation for the MW Content Pipeline - https://phabricator.wikimedia.org/T401010#11129984 (10xcollazo) I looked at implementing vanilla SQL equivalents to the metrics we do, and found out the following: For equivalent to: ` #... [20:46:56] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Hack unique_devices_per_domain recreating `.m` subdomain use `x_analytics` `ismobile` value - https://phabricator.wikimedia.org/T401666#11130079 (10Mayakp.wiki) hey @JAllemandou , unless there is a bigger Foundation wide initiative to get rid of Turnilo... [22:06:25] 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Movement-Insights: mediawiki_history - account for temp accounts in mediawiki_user_history_check_error - https://phabricator.wikimedia.org/T401325#11130278 (10amastilovic) It seems to me that this growth is organic and should be taken into account wh...