[09:15:15] 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Requesting permission to enable kafka log compaction for page_rerender on kafka-main - https://phabricator.wikimedia.org/T354794 (10Joe) It generally seems ok, but a few considerations: * kafka-main is much sma... [09:15:41] <_joe_> inflatador: replied on the task, sorry for the delay [09:19:22] 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Requesting permission to enable kafka log compaction for page_rerender on kafka-main - https://phabricator.wikimedia.org/T354794 (10brouberol) > I also want to note that this doesn't solve the long-standing iss... [10:24:26] Hey folks, just following up regarding out network maintenance in codfw rack b5 later today [10:24:39] kubernetes2032, kubernetes2031 and kubernetes2023 are in that rack [10:24:53] and also parse2006 and parse2007 [10:25:57] claime: I think you updated the sheet to detail the actions needed, are you able to take care of those beforehand? [10:26:05] task is T355549 btw [10:26:10] topranks: yep, that was my plan [10:26:14] sorry I wasn't more explicit [10:26:23] no probs at all that's great :) [10:26:38] let me know if I can help with anything [10:28:41] topranks: I put an alert down at 1530UTC for me to drain/cordon the kube nodes and depool the parse ones, I'll downtime them as well, don't think there's anything else that needs done on our end [10:29:42] yep sounds good. I'm gonna run a downtime for the whole rack before we begin so I can handle that side of it unless you need to do it before you drain [10:31:09] I don't, so I'll remove that from the todo :) Thanks [10:44:16] in advance of the same, taking maps2006 out of the tegola configs: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/992887 [10:45:00] tbh it'd probably be a small bump in errors but worth being careful. I'll depool kartotherian closer to the time [12:41:22] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2357.codfw.wmnet with OS bullseye [12:41:39] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2395.codfw.wmnet with OS bullseye [12:41:54] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2267.codfw.wmnet with OS bullseye [13:21:23] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2395.codfw.wmnet with OS bullseye completed: - mw2395 (**PASS**) - Downtimed on... [13:24:10] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2267.codfw.wmnet with OS bullseye completed: - mw2267 (**PASS**) - Downtimed on... [13:28:22] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw2357.codfw.wmnet with OS bullseye completed: - mw2357 (**PASS**) - Downtimed on... [14:20:41] hey folks. I'm looking for a review on https://gerrit.wikimedia.org/r/c/operations/puppet/+/976735 for the etcd config [14:38:24] topranks: all serviceops nodes ready for the migration [14:38:41] claime: great, thanks for that! [14:54:07] sobanski: moritzm: mutante: for the Zuul / python2.7 meeting, nothing has happened on that front since last week. I have been busy with Gerrit upgrade and running the MediaWiki train. [14:54:23] so i propose to cancel the meeting which is scheduled in half an hour? [14:56:17] Works for me, i'll move it to next week at the same time [14:56:31] +1 :) [15:01:16] ack, +1 [15:11:18] danke schon! [15:21:50] 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [15:38:19] 10serviceops, 10Maps: Repool maps primaries in Kartotherian - https://phabricator.wikimedia.org/T355892 (10hnowlan) [15:39:05] 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [16:56:01] 10serviceops, 10SRE, 10SecTeam-Processed, 10Security, 10Vuln-Misconfiguration: Helm Chart misconfigurations - https://phabricator.wikimedia.org/T355167 (10sbassett) 05In progress→03Resolved p:05Triage→03Low [16:56:30] 10serviceops, 10SRE, 10SecTeam-Processed, 10Security, 10Vuln-Misconfiguration: Helm Chart misconfigurations - https://phabricator.wikimedia.org/T355167 (10sbassett) 05Resolved→03In progress Whoops, I'll leave it in progress until the patches are actually merged/deployed.