[08:27:12] 06Data-Engineering, 06serviceops-radar, 10Event-Platform: Improve eventgates health check/readiness probe - https://phabricator.wikimedia.org/T373192#10090668 (10brouberol) To add to that, I understand @Ottomata's sentiment to have a readiness probe that reflects whether it is working correctly end-to-end. H... [08:56:18] 06Data-Engineering, 10Data Pipelines, 13Patch-For-Review: Fix generation of _IMPORTED flags by Gobblin - https://phabricator.wikimedia.org/T365223#10090834 (10Antoine_Quhen) 05Open→03Resolved a:03Antoine_Quhen [09:22:13] 06Data-Engineering, 13Patch-For-Review: Timeout hive-metastore locks - https://phabricator.wikimedia.org/T365563#10091043 (10Antoine_Quhen) [09:23:40] 06Data-Engineering, 13Patch-For-Review: Timeout hive-metastore locks - https://phabricator.wikimedia.org/T365563#10091044 (10Antoine_Quhen) [11:23:11] 06Data-Engineering, 06serviceops-radar, 10Event-Platform, 13Patch-For-Review: Improve eventgates health check/readiness probe - https://phabricator.wikimedia.org/T373192#10091426 (10JMeybohm) To unblock the kafka hardware replacements, I've added an alternative readiness probe to the chart which will be us... [11:25:34] 06Data-Engineering, 06Data-Platform, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Change page.page_links_updated to fixed-length timestamp in wmf wikis - https://phabricator.wikimedia.org/T371742#10091435 (10Ladsgroup) [11:27:32] 06Data-Engineering, 10CheckUser, 06Data Products, 06DBA, and 2 others: Remove cuc_actiontext, cuc_only_for_read_old, and cuc_private from cu_changes on WMF wikis - https://phabricator.wikimedia.org/T370903#10091437 (10Ladsgroup) [11:59:26] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): MediaWiki Reconciliation API - https://phabricator.wikimedia.org/T368782#10091562 (10daniel) Tagging #mw-interfaces-team for awareness. [12:01:12] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Dumps 2.0 (Kanban Board): MediaWiki Reconciliation API - https://phabricator.wikimedia.org/T368782#10091578 (10daniel) >>! In T368782#9939636, @Ottomata wrote: > Hm, a potential problem: > > We really don't want this API to be publicly accessible. I'm... [12:19:23] 10Quarry: Update cluster to 1.26 - https://phabricator.wikimedia.org/T373093#10091623 (10rook) Doesn't appear to be fully deploying. Cluster deploys, but kube-system pods seem to have some issues. Main issue is maybe k8s-keystone-auth which is giving: ` Warning Failed 11m (x4 over 13m) kubelet... [12:40:41] 10Quarry: Update cluster to 1.26 - https://phabricator.wikimedia.org/T373093#10091722 (10rook) Looks like the same kinds of things are happening in tf-infra-test ` NAME READY STATUS RESTARTS AGE pod/coredns-745687fb66-8jw96 1/1 R... [13:39:41] Hello folks! What is the status of an-worker1165 and an-worker1127? There are some alerts pending since a long time (if they are not a concern, let's silence) [16:40:05] btullis: o/ any news for the statXXXX reboots? https://phabricator.wikimedia.org/T366555 [16:42:44] elukey: I am out of office today. Back tomorrow. I will work on those stat server and snapshot server reboots. [16:45:42] ahhhhh ok sorryyyyy [16:45:44] didn't know it! [16:45:50] please don't answer to me when you are afk [16:46:02] :) [16:50:42] 10Quarry: Update cluster to 1.26 - https://phabricator.wikimedia.org/T373093#10093066 (10rook) Looks like 1.26 isn't working anymore as some of the image tags that 1.26 wants have been removed. Upgrading to 1.27 [16:55:32] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Product-Analytics: Publishing conda environments with WMF Data Workflow Utils is broken - https://phabricator.wikimedia.org/T367848#10093093 (10Ahoelzl) [16:56:16] 10Data-Engineering (Q1 2024 July 1st - September 30th): Remove stale GDI Equity Landscape jobs. - https://phabricator.wikimedia.org/T369649#10093095 (10Ahoelzl) [16:56:59] 06Data-Engineering: [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition - https://phabricator.wikimedia.org/T354694#10093126 (10Ahoelzl) [17:14:50] 10Quarry: Update cluster to 1.26 - https://phabricator.wikimedia.org/T373093#10093234 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/67 [17:15:35] 10Quarry: Remove quarry-124 cluster - https://phabricator.wikimedia.org/T373375 (10rook) 03NEW [17:17:09] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Metrics Platform: Make jsonschema-tools merge values of enums when merging allOf - https://phabricator.wikimedia.org/T345317#10093265 (10Ahoelzl) [17:17:10] 10Quarry: Remove quarry-124 cluster - https://phabricator.wikimedia.org/T373375#10093268 (10rook) [17:17:11] 10Quarry: Update cluster to 1.26 - https://phabricator.wikimedia.org/T373093#10093269 (10rook) [17:17:24] 10Quarry: Update cluster to 1.26 - https://phabricator.wikimedia.org/T373093#10093272 (10rook) 05Open→03Resolved [17:18:23] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization, 07Epic: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#10093275 (10Ahoelzl) [18:16:15] 10Quarry: Update cluster to 1.26 - https://phabricator.wikimedia.org/T373093#10093518 (10bd808) [19:19:00] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Rollback haproxy feed automated ingestion - https://phabricator.wikimedia.org/T372456#10093810 (10gmodena) [20:55:59] (03PS1) 10Clare Ming: Update Metrics Platform common fragment to include instrument name identifier. [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1066880 (https://phabricator.wikimedia.org/T366802) [21:07:13] (03PS1) 10Clare Ming: Update Metrics Platform web base schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1066883 (https://phabricator.wikimedia.org/T366802) [21:10:46] (03PS1) 10Clare Ming: Update Metrics Platform app base schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1066889 (https://phabricator.wikimedia.org/T366802)