[02:27:04] 10Analytics-Canonical-Data, 06Movement-Insights: Update the canonical wiki dataset - https://phabricator.wikimedia.org/T363287 (10nshahquinn-wmf) 03NEW [07:07:13] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1022545 (https://phabricator.wikimedia.org/T363270) (owner: 10Gerrit maintenance bot) [07:07:40] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1022543 (https://phabricator.wikimedia.org/T363263) (owner: 10Gerrit maintenance bot) [07:08:00] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1022540 (https://phabricator.wikimedia.org/T363256) (owner: 10Gerrit maintenance bot) [07:08:22] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1022537 (https://phabricator.wikimedia.org/T363243) (owner: 10Gerrit maintenance bot) [08:11:14] 06Data-Engineering, 10EventStreams, 10stewardbots, 10Toolforge, and 2 others: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher - https://phabricator.wikimedia.org/T329327#9739324 (10aborrero) >>! In T329327#9738585, @bd808 wrot... [08:37:22] (03PS11) 10Aqu: Add CLI to create or update Iceberg tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [08:54:33] 06Data-Engineering, 10Cassandra, 06Data-Persistence, 10Data-Platform-SRE (2024.04.15 - 2024.05.05), 13Patch-For-Review: Encrypt Airflow connections to AQS Cassandra - https://phabricator.wikimedia.org/T362181#9739504 (10BTullis) a:03BTullis [09:08:08] !log run 'kill `pgrep -u dbad2021`' on all stat nodes to unblock puppet [09:08:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:21:56] (03PS12) 10Aqu: Add CLI to create or update Iceberg tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [09:26:09] (03CR) 10CI reject: [V:04-1] Add CLI to create or update Iceberg tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) (owner: 10Aqu) [09:31:50] (03PS13) 10Aqu: Add CLI to create or update Iceberg tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [11:01:29] (03PS1) 10Joal: [WIP] Update ClickstreamBuilder [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1023828 (https://phabricator.wikimedia.org/T355588) [11:07:21] (03CR) 10CI reject: [V:04-1] [WIP] Update ClickstreamBuilder [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1023828 (https://phabricator.wikimedia.org/T355588) (owner: 10Joal) [11:37:36] (03PS2) 10Joal: [WIP] Update ClickstreamBuilder [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1023828 (https://phabricator.wikimedia.org/T355588) [11:42:53] (03CR) 10Ottomata: development: webrequest: add haproxy_pid field (031 comment) [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1021902 (https://phabricator.wikimedia.org/T351117) (owner: 10Gmodena) [11:59:23] 06Data-Engineering, 10EventStreams, 10stewardbots, 10Toolforge, and 2 others: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher - https://phabricator.wikimedia.org/T329327#9740030 (10Ottomata) This is probably not helpful, but E... [12:24:59] (03PS3) 10Joal: [WIP] Update ClickstreamBuilder [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1023828 (https://phabricator.wikimedia.org/T355588) [12:27:42] !log shutting down stat1010 to allow the GPU power cable to be fitted for: T336040 [12:27:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:27:45] T336040: Bring stat1010 into service with GPU from stat1005 - https://phabricator.wikimedia.org/T336040 [13:19:26] (03PS14) 10Aqu: Add CLI to create or update Iceberg tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [14:16:49] (03PS4) 10Joal: [WIP] Update ClickstreamBuilder [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1023828 (https://phabricator.wikimedia.org/T355588) [15:00:29] !log starting refinery deployment [15:00:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:54:49] !log Deployed refinery using scap, then deployed onto hdfs. [15:55:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:11:16] 06Data-Engineering, 10EventStreams, 10stewardbots, 10Toolforge, and 2 others: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher - https://phabricator.wikimedia.org/T329327#9741202 (10bd808) >>! In T329327#9739317, @aborrero wrot... [16:14:59] btullis: <3 [16:20:18] Thx. :-) [16:30:05] (KafkaReplicationFactorTooLow) firing: ... [16:30:05] Kafka topic codfw.mediawiki.job.securePollArchiveElection replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://grafana.wikimedia.org/d/000000234/kafka-by-topic?var-kafka_cluster=jumbo-eqiad&var-kafka_broker=All&var-topic=codfw.mediawiki.job.securePollArchiveElection&viewPanel=40 - ... [16:30:05] https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [16:34:04] 06Data-Engineering, 06Data Products, 06Tech-Docs-Team: Review and deprecate Analytics/Geoeditors page on Wikitech - https://phabricator.wikimedia.org/T356538#9741330 (10TBurmeister) [16:35:05] (KafkaReplicationFactorTooLow) resolved: ... [16:35:05] Kafka topic codfw.mediawiki.job.securePollArchiveElection replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://grafana.wikimedia.org/d/000000234/kafka-by-topic?var-kafka_cluster=jumbo-eqiad&var-kafka_broker=All&var-topic=codfw.mediawiki.job.securePollArchiveElection&viewPanel=40 - ... [16:35:05] https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [16:37:34] 06Data-Engineering, 06Data Products, 07Documentation: Review and deprecate Analytics/Geoeditors page on Wikitech - https://phabricator.wikimedia.org/T356538#9741335 (10TBurmeister) [16:48:45] elukey: That GPU on stat1010 should be working now. Feel free to test at your convenience. [16:53:56] btullis: thanksss [19:12:46] (03PS2) 10Mforns: Modify Commons Impact Metrics queries to ignore ancestor categories [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1023491 (https://phabricator.wikimedia.org/T358699) [19:19:25] (03PS3) 10Mforns: Modify Commons Impact Metrics queries to ignore ancestor categories [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1023491 (https://phabricator.wikimedia.org/T358699) [19:27:43] (03CR) 10Xcollazo: [C:03+2] "Minor commit message edit suggested." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1023491 (https://phabricator.wikimedia.org/T358699) (owner: 10Mforns) [19:28:06] (03Abandoned) 10Xcollazo: Calculate deep counts for primary categories only. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1023494 (https://phabricator.wikimedia.org/T358681) (owner: 10Xcollazo) [20:38:22] 10Analytics-Canonical-Data, 06Movement-Insights: Update the canonical wiki dataset - https://phabricator.wikimedia.org/T363287#9742718 (10nshahquinn-wmf) p:05Triage→03Medium a:03nshahquinn-wmf