[02:58:30] <jinxer-wm>	 FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[06:58:30] <jinxer-wm>	 FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[06:58:56] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11 - https://phabricator.wikimedia.org/T403838#11156135 (10dcausse)
[08:07:19] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 13Patch-For-Review: Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11 - https://phabricator.wikimedia.org/T403838#11156346 (10brouberol) ` ~ ❯ docker pull docker-registry.discovery.wmnet/flink-kubernetes-operator:1.12.1-wmf0 1.12.1-wm...
[08:14:20] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 13Patch-For-Review: Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11 - https://phabricator.wikimedia.org/T403838#11156361 (10brouberol) a:03brouberol
[08:14:59] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10DPE-Mediawiki-Content: When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it - https://phabricator.wikimedia.org/T388793#11156364 (10Antoine_Quhen) Here is a test environment to replicate the problem: https://gitlab.wikimedia.org/...
[08:53:14] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 13Patch-For-Review: Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11 - https://phabricator.wikimedia.org/T403838#11156620 (10brouberol) 05Open→03In progress
[10:12:13] <wikibugs>	 06Data-Engineering, 06Traffic: Reduce noise from duplicate sequence-gap alerts on HaProxy-webrequests - https://phabricator.wikimedia.org/T401383#11156883 (10Vgutierrez) local tests show that HAProxy issued `46410639` but it never reached the kafka cluster, probably because haproxykafka failed to parse it for...
[10:24:00] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11156943 (10Ladsgroup)
[10:58:30] <jinxer-wm>	 FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[11:00:55] <wikibugs>	 06Data-Engineering, 06Traffic: Reduce noise from duplicate sequence-gap alerts on HaProxy-webrequests - https://phabricator.wikimedia.org/T401383#11157028 (10Vgutierrez) probably unrelated but I've found what it could be a HAProxy bug related to `%rt` being increased twice per request: https://github.com/hapro...
[11:08:57] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 13Patch-For-Review: Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11 - https://phabricator.wikimedia.org/T403838#11157041 (10brouberol) All operators have been upgraded.
[12:07:15] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06Trust and Safety Product Team, and 3 others: Drop index cul_target_type from cu_log - https://phabricator.wikimedia.org/T401884#11157142 (10mszwarc) 05Open→03In progress a:03mszwarc
[12:16:51] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11 - https://phabricator.wikimedia.org/T403838#11157156 (10dcausse) Tested one job in codfw and it worked well, thanks!
[12:21:48] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11157164 (10Ladsgroup) @jcrespo this might cause some reduction in backup sizes.
[12:37:16] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11157230 (10Ladsgroup)
[14:04:12] <wikibugs>	 06Data-Engineering, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Upgrade flink-k8s-operator to 1.12.1 or downgrade to 1.11 - https://phabricator.wikimedia.org/T403838#11157702 (10brouberol) 05In progress→03Resolved
[14:28:32] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: EventBus - Add central user id to MediaWiki events - https://phabricator.wikimedia.org/T403664#11157824 (10Ottomata) Def an important choice and I am on the fence!  Let's get some more minds.  Asking in slack in...
[14:58:30] <jinxer-wm>	 FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[15:07:21] <wikibugs>	 (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] Fix wmf_raw.categorylinks column types [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1185169 (https://phabricator.wikimedia.org/T397923) (owner: 10Aleksandar Mastilovic)
[15:08:13] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: Global Editor Metrics for YiR, Apps Activity Tab, and Growth Impact Module - https://phabricator.wikimedia.org/T403660#11158125 (10Ottomata) a:03Ottomata
[15:17:17] <wikibugs>	 06Data-Engineering, 10Data Pipelines: Add user Central ID to mediawiki_history table in Hive - https://phabricator.wikimedia.org/T365648#11158194 (10Ottomata) > add this information to MediaWiki EventBus emitted events {T403664}
[15:27:13] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: EventBus - Add central user id to MediaWiki events - https://phabricator.wikimedia.org/T403664#11158297 (10taavi) Both are honestly fine.  But given the explicit bikeshedding invite, I !vote for * `user_central_i...
[16:14:53] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06Discovery-Search, 06Infrastructure-Foundations, and 2 others: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860#11158658 (10elukey) @RKemper Hi! What is the status of the code review? Ping me if you need some help/revie...
[16:19:29] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: EventBus - Add central user id to MediaWiki events - https://phabricator.wikimedia.org/T403664#11158673 (10Tgr) >>! In T403664#11154141, @nshahquinn-wmf wrote: > it's used by the only existing MediaWiki central I...
[16:50:27] <wikibugs>	 06Data-Engineering, 10Product-Analytics (Kanban): Follow-up analysis to understand usage of dumps to inform v2 rollout - https://phabricator.wikimedia.org/T402963#11158883 (10KCVelaga_WMF) a:03KCVelaga_WMF
[17:19:25] <wikibugs>	 (03PS2) 10Ottomata: Sqoop CentralAuth's globaluser and localuser tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1184925 (https://phabricator.wikimedia.org/T389666)
[17:19:37] <wikibugs>	 (03CR) 10Ottomata: "Thanks!  Not sure if I did that right." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1184925 (https://phabricator.wikimedia.org/T389666) (owner: 10Ottomata)
[17:26:48] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 06Growth-Team, 06Machine-Learning-Team, and 2 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11159027 (10Ottomata) Quick note about event based solu...
[17:28:35] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Data-Persistence, 06Growth-Team, 06Machine-Learning-Team, and 2 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11159042 (10Ottomata) @achou FYI, I am prioritizing my...
[18:03:38] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Use `x_analytics` `ismobile` instead of url `.m` subdomain for webrequest mobile-web `access_method` - https://phabricator.wikimedia.org/T401665#11159192 (10Krinkle) Thanks @JAllemandou. I checked pageviews for wikitech.wikimedia.org. This makes for a gr...
[18:45:39] <wikibugs>	 (03CR) 10Ottomata: Sqoop CentralAuth's globaluser and localuser tables (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1184925 (https://phabricator.wikimedia.org/T389666) (owner: 10Ottomata)
[18:58:30] <jinxer-wm>	 FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent
[19:06:37] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: Global Editor Metrics for YiR, Apps Activity Tab, and Growth Impact Module - https://phabricator.wikimedia.org/T403660#11159431 (10Ottomata) Another product requirement qu...
[19:29:19] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Use `x_analytics` `ismobile` instead of url `.m` subdomain for webrequest mobile-web `access_method` - https://phabricator.wikimedia.org/T401665#11159539 (10Mayakp.wiki)  >> In the coming week, we'll rollout to test.wikipedia.org (T401595). >> It is fine...
[19:58:16] <wikibugs>	 (03PS1) 10Santiago Faci: Added `agent.ua_string` as a possible source when parsing user agent [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1186049 (https://phabricator.wikimedia.org/T385180)
[20:16:07] <wikibugs>	 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): QA features on the new mobile URLs - https://phabricator.wikimedia.org/T403638#11159914 (10SLong-WMF) Hey folks! I was made aware of this on Friday and can help keep things organized and reported here (since I manage the QS team). I'...
[20:24:40] <wikibugs>	 (03PS1) 10Xcollazo: Remove 'partial' flag from MediawikiDumper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1186058 (https://phabricator.wikimedia.org/T384381)
[20:25:02] <wikibugs>	 (03CR) 10Bearloga: "I think it would be good to add some tests containing `agent.ua_string` and also a test that has both `http.request_headers['user-agent']`" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1186049 (https://phabricator.wikimedia.org/T385180) (owner: 10Santiago Faci)
[20:31:42] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11159991 (10Krinkle) @JAllemandou Let me know if/when the rollout may continue (from the POV of the unique devices dataset, that is). It looks like...
[20:34:24] <wikibugs>	 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): QA features on the new mobile URLs - https://phabricator.wikimedia.org/T403638#11160014 (10SLong-WMF)
[21:00:15] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Growth-Team, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: Global Editor Metrics for YiR, Apps Activity Tab, and Growth Impact Module - https://phabricator.wikimedia.org/T403660#11160081 (10KStoller-WMF) >>! In T403660#11159431, @...
[21:09:11] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11160108 (10Ladsgroup)
[21:17:37] <wikibugs>	 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): QA features on the new mobile URLs - https://phabricator.wikimedia.org/T403638#11160189 (10SLong-WMF) From Native Apps: They're researching the impact this may have on Native Apps currently. @ABorbaWMF will be producing a test plan i...
[21:57:50] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11160373 (10JAllemandou) >>! In T401666#11133036, @Hghani wrote: > @JAllemandou Thanks for generating the test data. The domain data looks okay to m...
[21:59:29] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11160376 (10JAllemandou) >>! In T401666#11134260, @nshahquinn-wmf wrote: > @JAllemandou You probably already have them in mind, but just in case, he...
[22:02:20] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11160405 (10JAllemandou) >>! In T401666#11159991, @Krinkle wrote: > @JAllemandou Let me know if/when the rollout may continue (from the POV of the u...
[22:05:35] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11160408 (10Ladsgroup)
[22:06:53] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Use `x_analytics` `ismobile` instead of url `.m` subdomain for webrequest mobile-web `access_method` - https://phabricator.wikimedia.org/T401665#11160423 (10JAllemandou) > Hey @jallemandou I believe the impact of this is to unique devices domain table an...
[22:10:49] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11160436 (10Ladsgroup) Because I do not have enough fun in my life, "db1225:3312" and "db1239:3312" are missing the index we...
[22:16:25] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11160445 (10JAllemandou)
[22:18:41] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th): Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11160448 (10Krinkle) Ah I see. I wasn't sure how frequently the dataset is published/used. To confirm- you're saying rolling out further tomorrow, f...
[22:24:24] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11160476 (10Ladsgroup)
[22:28:44] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763#11160487 (10Ladsgroup)
[22:32:07] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop cl_to and cl_collation from categorylinks in wmf production - https://phabricator.wikimedia.org/T402925#11160495 (10Ladsgroup)
[23:15:43] <wikibugs>	 (03PS1) 10Joal: Add access_method to unique-devices per-domain [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1186087 (https://phabricator.wikimedia.org/T401666)
[23:34:08] <wikibugs>	 10Data-Engineering (Q1 FY25/26 July 1st - September 30th), 13Patch-For-Review: Update unique_devices tables to add the `access_method` field - https://phabricator.wikimedia.org/T401666#11160652 (10nshahquinn-wmf) >>! In T401666#11160376, @JAllemandou wrote: >>>! In T401666#11134260, @nshahquinn-wmf wrote: >> -...