[07:12:07] hi folks morning! [07:12:34] I am planning to deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/954593 later on, to add another Rate Limit class for the api gateway [07:12:38] lemme know if you are ok [07:51:34] 10serviceops, 10SRE, 10ops-codfw: mw2448 is unreachable again - https://phabricator.wikimedia.org/T345597 (10MoritzMuehlenhoff) [08:55:40] hi there, it looks like mw1356.eqiad.wmnet is unreachable at the moment [08:56:12] jnuche: I'm rebooting appservers [08:56:42] Well parsoid servers at the moment but same difference [08:57:10] Is it causing an issue ? [08:58:24] * jnuche I rerun the train presync and [08:58:30] oops [08:58:42] I rerun the train prsync and the host was failing to connect [08:59:32] It's supposed to be rebooting, but it's not coming back up yet [09:01:28] ack, not really an issue, but I wanted to mention it somewhere in case the host needed some attention [09:01:55] It's taking a while to come back up though, thanks for mentioning it [10:44:08] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) - https://phabricator.wikimedia.org/T329826 (10JMeybohm) I put together a small go tool to validate some/all tokens with provided... [11:37:14] 10serviceops, 10Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10akosiaris) Adding a data point that just crossed my mind, just to rule it out. A mysqldump of the recommendation API database right now sits at 810MB. A bz... [12:10:32] 10serviceops, 10Data-Persistence, 10Performance-Team, 10SRE, 10Datacenter-Switchover: September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345263 (10LSobanski) [12:10:52] 10serviceops, 10Data-Persistence, 10Performance-Team, 10SRE, 10Datacenter-Switchover: September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345263 (10LSobanski) [12:11:22] 10serviceops, 10Data-Persistence, 10Performance-Team, 10SRE, 10Datacenter-Switchover: September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345263 (10LSobanski) [12:11:44] 10serviceops, 10Data-Persistence, 10Performance-Team, 10SRE, 10Datacenter-Switchover: September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345263 (10LSobanski) [12:35:26] 10serviceops, 10Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10elukey) @SCherukuwada Hi! Shall we restart the conversation about recommendation-api? [13:05:43] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2025.codfw.wmnet with OS bullseye [13:06:32] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2026.codfw.wmnet with OS bullseye [13:08:05] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2027.codfw.wmnet with OS bullseye [13:10:36] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) @JMeybohm @jbond @Volans thanks. [13:12:15] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2028.codfw.wmnet with OS bullseye [13:20:42] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2025.codfw.wmnet with OS bullseye completed: - kubernetes2025 (**PASS*... [13:21:19] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2029.codfw.wmnet with OS bullseye [13:21:56] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [13:22:34] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2027.codfw.wmnet with OS bullseye completed: - kubernetes2027 (**PASS*... [13:23:33] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [13:24:47] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2030.codfw.wmnet with OS bullseye [13:25:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2028.codfw.wmnet with OS bullseye completed: - kubernetes2028 (**PASS*... [13:25:44] 10serviceops, 10SRE, 10ops-codfw: mw2448 is unreachable again - https://phabricator.wikimedia.org/T345597 (10Jhancock.wm) a:03Jhancock.wm [13:26:07] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [13:36:52] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2026.codfw.wmnet with OS bullseye completed: - kubernetes2026 (**PASS*... [13:37:46] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2031.codfw.wmnet with OS bullseye [13:45:18] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2032.codfw.wmnet with OS bullseye [14:01:29] 10serviceops, 10SRE, 10Datacenter-Switchover: Switchover cookbooks live test - https://phabricator.wikimedia.org/T345588 (10ops-monitoring-bot) kamila@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in codfw: Datacenter Switchover Live test - T345588 started. [14:02:51] 10serviceops, 10SRE, 10ops-codfw: mw2448 is unreachable again - https://phabricator.wikimedia.org/T345597 (10Jhancock.wm) @MoritzMuehlenhoff I got into the idrac gui and there's nothing of interest to report. There's no BIOS or CPU errors this time. I do see something in the system lifecycle logs about seve... [14:13:37] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2029.codfw.wmnet with OS bullseye completed: - kubernetes2029 (**PASS*... [14:17:50] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2030.codfw.wmnet with OS bullseye executed with errors: - kubernetes20... [14:18:06] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [14:22:52] 10serviceops, 10SRE, 10ops-codfw: mw2448 is unreachable again - https://phabricator.wikimedia.org/T345597 (10MoritzMuehlenhoff) Thanks, let's retry with latest firmware, if anything happens again we can still open a case with Dell. [14:25:53] 10serviceops, 10SRE, 10Datacenter-Switchover: Switchover cookbooks live test - https://phabricator.wikimedia.org/T345588 (10ops-monitoring-bot) kamila@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in codfw: Datacenter Switchover Live test - T345588 completed. [14:26:00] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2031.codfw.wmnet with OS bullseye executed with errors: - kubernetes20... [14:26:15] 10serviceops, 10SRE, 10Datacenter-Switchover: Switchover cookbooks live test - https://phabricator.wikimedia.org/T345588 (10ops-monitoring-bot) kamila@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in codfw: Datacenter Switchover Live test - T345588 started. [14:33:15] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2032.codfw.wmnet with OS bullseye executed with errors: - kubernetes20... [14:38:20] 10serviceops, 10SRE, 10ops-codfw: mw2448 is unreachable again - https://phabricator.wikimedia.org/T345597 (10Jhancock.wm) 05Open→03Resolved [14:48:34] 10serviceops, 10SRE, 10ops-eqiad: Move eqiad thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343993 (10Jclark-ctr) [14:48:55] 10serviceops, 10SRE, 10decommission-hardware, 10ops-eqiad: Decommission thumbor100[12] - https://phabricator.wikimedia.org/T344598 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr [14:50:15] 10serviceops, 10SRE, 10Datacenter-Switchover: Switchover cookbooks live test - https://phabricator.wikimedia.org/T345588 (10ops-monitoring-bot) kamila@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in codfw: Datacenter Switchover Live test - T345588 completed. [14:51:22] 10serviceops, 10Parsoid (Tracking): Upgrade nodejs on testreduce1001 - https://phabricator.wikimedia.org/T345220 (10MoritzMuehlenhoff) >>! In T345220#9140286, @MoritzMuehlenhoff wrote: > I can do that, but I'll first reimage the server after https://gerrit.wikimedia.org/r/c/operations/puppet/+/954682/ is appli... [15:18:17] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2030.codfw.wmnet with OS bullseye [15:18:59] 10serviceops, 10SRE, 10ops-codfw: mw2448 is unreachable again - https://phabricator.wikimedia.org/T345597 (10MoritzMuehlenhoff) p:05Triage→03Medium a:05Jhancock.wm→03Clement_Goubert [15:19:31] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2031.codfw.wmnet with OS bullseye [15:21:40] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2032.codfw.wmnet with OS bullseye [15:57:31] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2033.codfw.wmnet with OS bullseye [16:01:05] 10serviceops, 10SRE, 10Datacenter-Switchover: Switchover cookbooks live test - https://phabricator.wikimedia.org/T345588 (10kamila) 05Open→03Resolved [16:07:06] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2032.codfw.wmnet with OS bullseye completed: - kubernetes2032 (**PASS*... [16:07:18] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2031.codfw.wmnet with OS bullseye completed: - kubernetes2031 (**WARN*... [16:07:49] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [16:08:19] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2034.codfw.wmnet with OS bullseye [16:08:50] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2035.codfw.wmnet with OS bullseye [16:28:53] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2030.codfw.wmnet with OS bullseye completed: - kubernetes2030 (**PASS*... [16:29:55] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2036.codfw.wmnet with OS bullseye [16:30:31] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [16:34:40] 10serviceops, 10Parsoid (Tracking): Upgrade nodejs on testreduce1001 - https://phabricator.wikimedia.org/T345220 (10ssastry) /srv/data has the db content from 1001. I would ideally like that copied over. And yes, need db grants (similar to that on testreduce1001). I also need to be able to run npm install on t... [16:38:56] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2033.codfw.wmnet with OS bullseye completed: - kubernetes2033 (**PASS*... [16:39:20] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2037.codfw.wmnet with OS bullseye [17:02:51] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2034.codfw.wmnet with OS bullseye completed: - kubernetes2034 (**PASS*... [17:03:09] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2035.codfw.wmnet with OS bullseye completed: - kubernetes2035 (**WARN*... [17:03:44] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [17:04:14] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye [17:04:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye executed with errors: - kubernetes20... [17:04:55] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye [17:05:03] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye executed with errors: - kubernetes20... [17:08:15] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye [17:08:26] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2036.codfw.wmnet with OS bullseye completed: - kubernetes2036 (**PASS*... [17:08:42] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye [17:09:10] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [17:18:05] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-53] - https://phabricator.wikimedia.org/T342534 (10Papaul) [17:21:23] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-53] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2037.codfw.wmnet with OS bullseye completed: - kubernetes2037 (**PASS*... [17:34:03] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Jul-Sep-2023), 10Datacenter-Switchover: CommRel support for September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345265 (10Trizek-WMF) [17:55:40] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-53] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye completed: - kubernetes2038 (**PASS*... [17:57:05] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-53] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye completed: - kubernetes2039 (**WARN*... [17:57:25] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-53] - https://phabricator.wikimedia.org/T342534 (10Papaul) [17:58:54] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-53] - https://phabricator.wikimedia.org/T342534 (10Papaul) [18:11:22] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-53] - https://phabricator.wikimedia.org/T342534 (10Papaul) This is all done but i have to fix DNS mgmt issue with 2029 and 2030 during the provision of those servers the serial number were swapped. Whne you login to 2029 mgmt...