[03:23:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [04:41:40] 10Quarry, 06cloud-services-team, 10Data-Services: Quarry WMCloud (ruwiki_p, section s6) experiencing sustained replication lag (~16 h) - https://phabricator.wikimedia.org/T394859#10846437 (10Marostegui) >>! In T394859#10845998, @Voyagerim wrote: > @Marostegui , expectations regarding replication latency thre... [04:42:10] 10Quarry, 06cloud-services-team, 06Data-Persistence, 10Data-Services: Quarry WMCloud (ruwiki_p, section s6) experiencing sustained replication lag (~16 h) - https://phabricator.wikimedia.org/T394859#10846438 (10Marostegui) 05Open→03Resolved a:03Marostegui Lag is back to 0 [06:13:38] 06Data-Engineering: Improve spider detection in the webrequest refinery pipeline - https://phabricator.wikimedia.org/T394794#10846696 (10Joe) Here are the changes to regex.yaml between 1.5.3 and 1.6.1; the differences are not huge so I don't expect huge shifts in spider traffic numbers - also because we smartly... [07:23:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:57:31] !log installing linux-image-6.12.22+bpo-amd64 on the remaining dse-k8s-worker nodes for T394897 [08:57:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:57:35] T394897: [Cephfs] Clients occasionally fail to release caps, resulting in blocked requests and Airflow service disruption - https://phabricator.wikimedia.org/T394897 [09:25:02] (03PS1) 10Giuseppe Lavagetto: Update dependency on ua-parser to 1.6.1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149333 (https://phabricator.wikimedia.org/T394794) [09:25:04] (03PS1) 10Giuseppe Lavagetto: Allow loading an external regex.yaml file for ua-parser [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149334 (https://phabricator.wikimedia.org/T394794) [09:27:12] 06Data-Engineering, 13Patch-For-Review: Improve spider detection in the webrequest refinery pipeline - https://phabricator.wikimedia.org/T394794#10847277 (10Joe) I've created two patches. My java knowledge is only matched by my knowledge of html/CSS, so please take my code with copious amounts of salt and tell... [09:27:26] (03CR) 10CI reject: [V:04-1] Allow loading an external regex.yaml file for ua-parser [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149334 (https://phabricator.wikimedia.org/T394794) (owner: 10Giuseppe Lavagetto) [09:27:48] (03CR) 10CI reject: [V:04-1] Update dependency on ua-parser to 1.6.1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149333 (https://phabricator.wikimedia.org/T394794) (owner: 10Giuseppe Lavagetto) [10:08:19] (03CR) 10Hnowlan: Allow loading an external regex.yaml file for ua-parser (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149334 (https://phabricator.wikimedia.org/T394794) (owner: 10Giuseppe Lavagetto) [10:43:20] (03CR) 10Giuseppe Lavagetto: [C:04-1] "The new version of the jar depends on snakeyaml 2.x, whereas we still depend on snakeyaml 1.x; I'd rather not embark in the adventure of u" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149333 (https://phabricator.wikimedia.org/T394794) (owner: 10Giuseppe Lavagetto) [10:43:36] (03PS2) 10Giuseppe Lavagetto: Allow loading an external regex.yaml file for ua-parser [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149334 (https://phabricator.wikimedia.org/T394794) [11:23:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:42:10] (03CR) 10Joal: "jackason-YAML depends on snakeyaml, and the jackson version we use depends on snakeyaml 1.26, not 2.x. Agreed to not change it if not need" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149333 (https://phabricator.wikimedia.org/T394794) (owner: 10Giuseppe Lavagetto) [12:49:16] (03CR) 10Joal: "One nit and one question" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149334 (https://phabricator.wikimedia.org/T394794) (owner: 10Giuseppe Lavagetto) [13:03:34] (03PS1) 10Aqu: Add documentation to ProduceCanaryEvents [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149397 (https://phabricator.wikimedia.org/T369845) [13:17:17] (03CR) 10Joal: "I don't think the comments really add value: the routes defined in the default HTTP config work within k8s!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149397 (https://phabricator.wikimedia.org/T369845) (owner: 10Aqu) [13:29:37] (03CR) 10Giuseppe Lavagetto: Allow loading an external regex.yaml file for ua-parser (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149334 (https://phabricator.wikimedia.org/T394794) (owner: 10Giuseppe Lavagetto) [13:29:44] (03PS3) 10Giuseppe Lavagetto: Allow loading an external regex.yaml file for ua-parser [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149334 (https://phabricator.wikimedia.org/T394794) [13:52:35] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop afl_patrolled_by from abuse_filter_log in production - https://phabricator.wikimedia.org/T391056#10848270 (10Marostegui) @FCeratto-WMF can you continue with the schema change deployment? [13:53:50] 06Data-Engineering, 06Data-Engineering-Radar, 06Java-Scala-Standardization: Migrate WMF Maven projects to new parent pom - https://phabricator.wikimedia.org/T360219#10848287 (10Gehel) 05Open→03Resolved Mostly done, the last project will be addressed as part of T367315 [14:04:01] (03CR) 10Joal: Allow loading an external regex.yaml file for ua-parser (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1149334 (https://phabricator.wikimedia.org/T394794) (owner: 10Giuseppe Lavagetto) [15:14:19] 06Data-Engineering: Create a view for existencelinks table - https://phabricator.wikimedia.org/T394898#10848867 (10fnegri) Per [this page](https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas#Who_admins_what) #data-engineering is responsible for view creation and updates. [15:19:24] 06Data-Engineering, 10Data-Services: Create a view for existencelinks table - https://phabricator.wikimedia.org/T394898#10848921 (10taavi) [15:23:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [16:35:51] 06Data-Engineering, 06Movement-Insights: event.editattemptstep is not logging some revisions that appear in mediawiki_history - https://phabricator.wikimedia.org/T394961#10849343 (10Hghani) [17:00:27] 06Data-Engineering, 10Data-Platform-SRE (2025.05.02 - 2025.05.23): datahub.wikimedia.org is down - https://phabricator.wikimedia.org/T395057 (10BTullis) 03NEW [17:00:32] 06Data-Engineering, 10Data-Platform-SRE (2025.05.02 - 2025.05.23): datahub.wikimedia.org is down - https://phabricator.wikimedia.org/T395057#10849462 (10BTullis) p:05Triage→03High [17:58:51] 06Data-Engineering, 10Data-Platform-SRE (2025.05.02 - 2025.05.23): datahub.wikimedia.org is down - https://phabricator.wikimedia.org/T395057#10849839 (10BTullis) The mysql setup job does not seem to be able to reach `an-mariadb1001:3306` If I disable the mysql setup job, the next init task is the `datahub-pro... [18:02:11] (03CR) 10Mforns: [V:03+2 C:03+2] Prepare sqoop and CIM queries for mediawiki image table migration [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1139117 (owner: 10Mforns) [18:08:28] !log starting refinery deploy for CIM adjustments and mediawiki image table [18:08:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:20:34] 10Quarry: Add "wikishared" database to Quarry and to wiki replicas - https://phabricator.wikimedia.org/T395072 (10Novem_Linguae) 03NEW [18:24:16] !log finished refinery deploy for CIM adjustments and mediawiki image table [18:24:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:36:11] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Enable merge-on-read for wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T393012#10850129 (10xcollazo) >>! In T393012#10827607, @xcollazo wrote: > Still to do here: > > [x] Run daily maintenance for a cou... [18:38:23] 10Quarry, 06cloud-services-team, 10Data-Services: Add "wikishared" database to wiki replicas - https://phabricator.wikimedia.org/T395072#10850145 (10JJMC89) [18:42:28] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Add data quality metrics to mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T392494#10850157 (10xcollazo) 05In progress→03Resolved [18:42:49] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Document wmf_content.mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T392948#10850159 (10xcollazo) 05In progress→03Resolved [18:43:11] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Enable merge-on-read for wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T393012#10850161 (10xcollazo) 05In progress→03Resolved [19:23:32] 06Data-Engineering, 10ContentTranslation, 10Metrics Platform: Update WMF-deployed extensions to use mw.config checks instead of manual m-dot URL hacks - https://phabricator.wikimedia.org/T390923#10850364 (10Izno) I have filed T395079 about one of these recommendations. [19:23:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:38:25] 06Data-Engineering, 10ChangeProp, 10Observability-Tracing, 13Patch-For-Review: Implement tracing across changeprop-jobqueue - https://phabricator.wikimedia.org/T395038#10850403 (10dr0ptp4kt) [19:38:41] 06Data-Engineering, 10ChangeProp, 10Observability-Tracing, 13Patch-For-Review: Implement tracing across changeprop-jobqueue - https://phabricator.wikimedia.org/T395038#10850404 (10dr0ptp4kt) Added a tag and a couple subscribers here for visibility. [20:48:16] Hey folks, FR Analytics was alerted that the fundraising team was seeing lower than expected impression volumes from kafka. The timing appears to line up with the switch from the old webrequest_text topic to webrequest_frontend_text. I've opened a phab here- can someone please tell me the approriate tags and if i should add anyone to the ticket? [20:48:16] https://phabricator.wikimedia.org/T395089 [21:34:52] 06Data-Engineering, 10FR-Tech-Analytics, 10FR-tech-data-integrity, 10fundraising-tech-ops: Low volume in new webrequest feed - https://phabricator.wikimedia.org/T395089#10850795 (10Jgreen) [21:46:53] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for Neslihan_Turan_WMDE - https://phabricator.wikimedia.org/T394395#10850819 (10Dzahn) 05In progress→03Stalled [21:47:39] 06Data-Engineering, 10FR-Tech-Analytics, 10FR-tech-data-integrity, 10fundraising-tech-ops: Low volume in new webrequest feed - https://phabricator.wikimedia.org/T395089#10850820 (10Jgreen) [23:23:32] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem