[05:56:40] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Cite, and 5 others: Remove Reference Previews tracking metrics - https://phabricator.wikimedia.org/T242127#10837815 (10awight) a:03awight [06:38:21] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 10Cite, and 6 others: Remove Reference Previews tracking metrics - https://phabricator.wikimedia.org/T242127#10837855 (10awight) a:05awight→03None [07:27:06] 06Data-Engineering-Radar, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Shutdown varnishkafka webrequest instances - https://phabricator.wikimedia.org/T393772#10837899 (10Fabfur) [07:39:41] 06Data-Engineering-Radar, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Shutdown varnishkafka webrequest instances - https://phabricator.wikimedia.org/T393772#10837925 (10Fabfur) [07:40:13] 06Data-Engineering-Radar, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Shutdown varnishkafka webrequest instances - https://phabricator.wikimedia.org/T393772#10837931 (10Fabfur) 05Open→03In progress varnishkafka webrequest has been shut down on all cache hosts [10:31:21] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE, and 2 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10838343 (10fnegri) > you first need to stop mariadb Sorry my message was not clear enough, I was implying these steps were //afte... [10:32:36] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE, and 2 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10838345 (10fnegri) > run-puppet-agent (will this automatically install 1011? do I need a puppet patch?) Ok, I didn't see your not... [10:36:53] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10838374 (10fnegri) > Where is that in Puppet? Found it: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148297 [10:59:21] !log deploying https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1147737 to eventgate-analytics-external in support of xLab [10:59:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:06:12] 06Data-Engineering, 06Data-Engineering-Radar, 10HaproxyKafka, 06Traffic, and 2 others: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10838476 (10Fabfur) [11:07:42] 14Data-Engineering (Q3 2025 January 1st - March 31th), 10HaproxyKafka, 10DPE HAProxy Migration, 13Patch-For-Review: [HAProxy migration] HAProxy and VarnishKafka should produce compatible datasets - https://phabricator.wikimedia.org/T382571#10838483 (10Fabfur) [11:07:54] 14Data-Engineering (Q3 2025 January 1st - March 31th), 10HaproxyKafka: [HAProxy migration] Take actions on HAProxy `uri_host` and `accept_language` differences with VarnishKafka - https://phabricator.wikimedia.org/T386354#10838485 (10Fabfur) [11:08:03] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10HaproxyKafka, 06Traffic, 10DPE HAProxy Migration: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10838486 (10Fabfur) [11:08:26] 14Data-Engineering (Q3 2025 January 1st - March 31th), 10HaproxyKafka, 06Traffic, 10DPE HAProxy Migration: Make webrequest_frontend being ingested using the in-data `dt` field - https://phabricator.wikimedia.org/T388397#10838498 (10Fabfur) [11:09:46] 14Data-Engineering (Q3 2025 January 1st - March 31th), 10HaproxyKafka, 06Traffic, 10DPE HAProxy Migration: Update HAProxyKafka kafka-timestamp type - https://phabricator.wikimedia.org/T389521#10838502 (10Fabfur) [11:09:58] 06Data-Engineering-Radar, 10HaproxyKafka, 06Traffic, 10Data-Platform-SRE (2025.05.02 - 2025.05.23), and 2 others: Replicate current low-message alerting from VarnishKafka - https://phabricator.wikimedia.org/T391810#10838503 (10Fabfur) [11:10:22] 06Data-Engineering-Radar, 10HaproxyKafka, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Shutdown varnishkafka webrequest instances - https://phabricator.wikimedia.org/T393772#10838508 (10Fabfur) [11:42:15] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10838639 (10Marostegui) >>! In T394372#10838374, @fnegri wrote: >> Where is that in Puppet? > > Found it: https://gerrit.wikimedia... [11:58:25] (03PS1) 10Aqu: Add EvolveAndRefineToHiveTable job to refine [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1148317 (https://phabricator.wikimedia.org/T369845) [12:01:34] (03CR) 10Joal: [C:03+1] "Ok for this patch - not sure of how easy it'll be to re-release v0.2.64." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1147771 (https://phabricator.wikimedia.org/T367405) (owner: 10Peter Fischer) [12:02:26] (03CR) 10Joal: [C:03+1] "Actually I read the original comment wrong: NO need to re-release, all good!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1147771 (https://phabricator.wikimedia.org/T367405) (owner: 10Peter Fischer) [12:03:35] fabfur: this chart makes me joyful: https://grafana.wikimedia.org/goto/RQqo1w-HR?orgId=1 [12:18:00] 06Data-Engineering, 06cloud-services-team, 10Data-Services: Create existencelinks table in production - https://phabricator.wikimedia.org/T394617#10838837 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup I just created the tables in all wikis. >>! In T394617#10833232, @Bugreporter wrote: > This tab... [12:27:29] joal fabfur: woohoo [12:41:53] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics (Kanban): Wikidata editor numbers for stats.wikimedia and Grafana are dramatically different - https://phabricator.wikimedia.org/T394770 (10AndrewTavis_WMDE) 03NEW [12:43:56] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics (Kanban): Wikidata editor numbers for stats.wikimedia and Grafana are dramatically different - https://phabricator.wikimedia.org/T394770#10838933 (10AndrewTavis_WMDE) [12:44:39] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics (Kanban): Wikidata editor numbers for stats.wikimedia and Grafana are dramatically different - https://phabricator.wikimedia.org/T394770#10838944 (10AndrewTavis_WMDE) [12:44:52] joal: the end of an era! [12:45:45] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics (Kanban): Wikidata editor numbers for stats.wikimedia and Grafana are dramatically different - https://phabricator.wikimedia.org/T394770#10838957 (10AndrewTavis_WMDE) [12:59:42] brouberol: https://media1.tenor.com/m/q9FDykY6oHkAAAAC/hooray.gif [13:00:22] that reminds me that I have `zoidberg` symlinked to `rg` on a server somewhere [13:05:11] ok but why? [13:05:13] :D [13:06:53] because I'm an idiot [13:15:17] 10Data-Engineering (Q4 2025 April 1st - June 30th): Facilitate automatic artifact cache warming for airflow-dags artifacts - https://phabricator.wikimedia.org/T392244#10839136 (10BTullis) >>! In T392244#10834848, @xcollazo wrote: >>I think that would be enough to allow us to stop using scap for artifact sync and... [15:33:29] 06Data-Engineering: Improve spider detection in the webrequest refinery pipeline - https://phabricator.wikimedia.org/T394794 (10Joe) 03NEW [15:39:04] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics (Kanban): Wikidata editor numbers for stats.wikimedia and Grafana are dramatically different - https://phabricator.wikimedia.org/T394770#10839894 (10AndrewTavis_WMDE) [15:39:41] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics (Kanban): Wikidata editor numbers for stats.wikimedia and Grafana are dramatically different - https://phabricator.wikimedia.org/T394770#10839896 (10AndrewTavis_WMDE) [15:52:04] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Enable Spark data lineage for all Airflow instances - https://phabricator.wikimedia.org/T386862#10839953 (10mforns) @brouberol I've written some code that I think might work. But I think that we would need to define another connection in `... [16:11:23] (03CR) 10Joal: [C:03+1] "I'm thorn between a +1 and asking to just have 2 chained spark jobs launched by Airflow. I think this is easier to understand and maintain" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1148317 (https://phabricator.wikimedia.org/T369845) (owner: 10Aqu) [17:58:00] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics (Kanban): Wikidata editor numbers for stats.wikimedia and Grafana are dramatically different - https://phabricator.wikimedia.org/T394770#10840658 (10Ahoelzl) @gehel can you take a look? [18:00:13] 06Data-Engineering, 06MW-Interfaces-Team, 10observability: [Needs grooming] Turnilo: include authentication status in request data cube - https://phabricator.wikimedia.org/T332864#10840678 (10Ahoelzl) Moved to Next Up (in consideration for Q1) [18:11:53] 06Data-Engineering: Improve spider detection in the webrequest refinery pipeline - https://phabricator.wikimedia.org/T394794#10840700 (10Ahoelzl) @nshahquinn-wmf can you assess the suggested change? [18:14:35] 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10840707 (10Ahoelzl) @JAllemandou any implications for the Data Platform / sqooping? [18:20:23] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Cassandra, 13Patch-For-Review: Audit and update AQS Cassandra roles & grants - https://phabricator.wikimedia.org/T313877#10840733 (10Eevans) 05Open→03Resolved a:03Eevans This is complete now; Please re-open and ping me right away if any of these... [18:29:35] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics (Kanban): Wikidata editor numbers for stats.wikimedia and Grafana are dramatically different - https://phabricator.wikimedia.org/T394770#10840809 (10Addshore) [18:30:52] 06Data-Engineering: Improve spider detection in the webrequest refinery pipeline - https://phabricator.wikimedia.org/T394794#10840834 (10JAllemandou) I don't know how familiar people reading this task are about our traffic-flagging, so I'll take from the beginning :) We categorize traffic in 3: * `spider` - comp... [18:42:48] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics (Kanban): Wikidata editor numbers for stats.wikimedia and Grafana are dramatically different - https://phabricator.wikimedia.org/T394770#10840881 (10Addshore) https://stats.wikimedia.org/#/wikidata.org/contributing/editors/normal%7Cline%7C2-year%7C~total%7... [18:50:18] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all depen... - https://phabricator.wikimedia.org/T367405#10840923 [19:14:40] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Investigate first set of DQ alerts - https://phabricator.wikimedia.org/T394310#10841129 (10xcollazo) >>! In T394310#10834427, @gmodena wrote: > ... > Are all revision you posted supposed to be suppressions? I'm surprised to see `revi... [20:25:48] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 07Security: Investigate first set of DQ alerts - https://phabricator.wikimedia.org/T394310#10841429 (10xcollazo) [20:27:17] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 07Security: Investigate first set of DQ alerts - https://phabricator.wikimedia.org/T394310#10841445 (10xcollazo) (Seems like this is evolving to be a security related issue, thus marked ticket as so, and removed subscribers to just... [21:59:33] 10Quarry, 06cloud-services-team, 10Data-Services: Quarry WMCloud (ruwiki_p, section s6) experiencing sustained replication lag (~16 h) - https://phabricator.wikimedia.org/T394859#10841931 (10Reedy)