[03:59:52] GENGHIS KHAN! [04:00:21] ChanFucker! [04:01:00] WMF Fucker! [07:49:17] 10serviceops, 10SRE, 10Abstract Wikipedia team (Phase λ – Launch), 10Patch-For-Review, 10Service-deployment-requests: New Service Request: function-orchestrator and function-evaluator (for Wikifunctions launch) - https://phabricator.wikimedia.org/T297314 (10JMeybohm) AIUI the only thing talking to the ev... [08:44:07] 10serviceops: sextant dependency management - https://phabricator.wikimedia.org/T341967 (10JMeybohm) [08:48:28] the parsoid get/200 average latency alert has been firing for ~7h now, known ? [08:53:11] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [08:54:06] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [10:30:26] hi folks! [10:30:45] I have a quick review to increase an envoy tls-proxy timeout for lift wing - https://gerrit.wikimedia.org/r/c/operations/puppet/+/938815/ [11:22:26] elukey: done [12:24:32] 10serviceops, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984 (10JMeybohm) p:05Triage→03Medium [12:26:41] 10serviceops, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984 (10JMeybohm) [12:26:43] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes: etcd cluster reimage strategies to use with the K8s upgrade cookbook - https://phabricator.wikimedia.org/T330060 (10JMeybohm) [12:26:45] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Scrape controller-manager and scheduler metrics - https://phabricator.wikimedia.org/T324959 (10JMeybohm) [12:26:47] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Reserve resources for system daemons on kubernetes nodes - https://phabricator.wikimedia.org/T277876 (10JMeybohm) [12:26:50] 10serviceops, 10Prod-Kubernetes: PodSecurityPolicies will be deprecated with Kubernetes 1.21 - https://phabricator.wikimedia.org/T273507 (10JMeybohm) [12:26:52] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE-Sprint-Week-Sustainability-March2023, and 2 others: Write a cookbook to set a k8s cluster in maintenance mode - https://phabricator.wikimedia.org/T277677 (10JMeybohm) [12:26:58] 10serviceops: [EPIC] Docker deprecation as a container runtime enginer for kubernetes. - https://phabricator.wikimedia.org/T269684 (10JMeybohm) [12:27:03] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Agree strategy for Kubernetes BGP peering to top-of-rack switches - https://phabricator.wikimedia.org/T306649 (10JMeybohm) [12:31:26] 10serviceops, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984 (10JMeybohm) [12:33:47] 10serviceops, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984 (10JMeybohm) [12:33:56] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Fix naming confusion around main/wikikube kubernetes clusters - https://phabricator.wikimedia.org/T336861 (10JMeybohm) [12:45:37] jayme: <3 [12:46:00] that being for the +1 or the task mess? :-p [12:46:21] the review! [13:16:25] 10serviceops, 10SRE, 10Abstract Wikipedia team (Phase λ – Launch), 10Patch-For-Review, 10Service-deployment-requests: New Service Request: function-orchestrator and function-evaluator (for Wikifunctions launch) - https://phabricator.wikimedia.org/T297314 (10Jdforrester-WMF) >>! In T297314#9018519, @JMeyb... [13:25:30] folks if you are ok I'll start some kafka partitions move in main-codfw [13:27:58] I am reusing https://gitlab.wikimedia.org/elukey/kafka_main_rebalance/-/tree/main/main-codfw from the last time, I'll commit all the moves that I do [13:28:03] together with rollback etc.. [13:29:00] and the plan is https://gitlab.wikimedia.org/elukey/kafka_main_rebalance/-/tree/main/main-codfw/topicmappr-json [13:29:35] in theory these ones shouldn't require any restart of changeprop, but who knows [13:30:47] 10serviceops: Rebalance kafka partitions in main-{eqiad,codfw} clusters - 2023 edition - https://phabricator.wikimedia.org/T341558 (10elukey) As happened the last time I created https://gitlab.wikimedia.org/elukey/kafka_main_rebalance/-/tree/main/main-codfw. I'll create and commit "completed" and "rollback" comm... [13:42:03] elukey: 👍 [13:44:05] 10serviceops, 10SRE, 10Abstract Wikipedia team (Phase λ – Launch), 10Patch-For-Review, 10Service-deployment-requests: New Service Request: function-orchestrator and function-evaluator (for Wikifunctions launch) - https://phabricator.wikimedia.org/T297314 (10JMeybohm) >>! In T297314#9019527, @Jdforrester-... [14:10:30] started! [14:14:29] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, and 2 others: Future of Thumbor's memcached backend - https://phabricator.wikimedia.org/T318695 (10akosiaris) [14:16:12] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, and 2 others: Future of Thumbor's memcached backend - https://phabricator.wikimedia.org/T318695 (10akosiaris) @hnowlan @jijiki. nutcracker removal merged and deployed. I am gonna let you have the pleasure of resolving this task :-) [15:14:30] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search: Requesting permission to use kafka-main cluster to transport CirrusSearch updates - https://phabricator.wikimedia.org/T341625 (10akosiaris) [15:26:09] very nice consequence of the first moves - https://grafana.wikimedia.org/d/000000027/kafka?forceLogin&from=now-3h&orgId=1&to=now&var-cluster=kafka_main&var-datasource=thanos&var-disk_device=All&var-kafka_broker=All&var-kafka_cluster=main-codfw&viewPanel=38 [15:27:28] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Requesting permission to use kafka-main cluster to transport CirrusSearch updates - https://phabricator.wikimedia.org/T341625 (10Gehel) [15:29:27] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Requesting permission to use kafka-main cluster to transport CirrusSearch updates - https://phabricator.wikimedia.org/T341625 (10Gehel) p:05Triage→03High [15:29:32] 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Requesting permission to use kafka-main cluster to transport CirrusSearch updates - https://phabricator.wikimedia.org/T341625 (10Gehel) [15:34:03] <_joe_> elukey: NICE! [15:35:41] with partition leaders we are less lucky, it takes more time to even them out