[00:32:16] 10serviceops, 10SRE, 10Patch-For-Review: Remove mediawiki::packages::fonts from non thumbor servers - https://phabricator.wikimedia.org/T294378 (10Dzahn) 05Open→03Resolved [01:32:42] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10Papaul) [07:37:25] 10serviceops, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10elukey) The next step is to add ml-serve2005 (new node) to the `ml-serve-codfw` cluster with Buster + overlay settings + new partm... [10:00:43] 10serviceops, 10Prod-Kubernetes: kubernetes1014 unreachable - https://phabricator.wikimedia.org/T301099 (10JMeybohm) [10:02:22] 10serviceops, 10Prod-Kubernetes: kubernetes1014 unresponsive - https://phabricator.wikimedia.org/T301099 (10JMeybohm) [10:09:56] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.3.0 - https://phabricator.wikimedia.org/T300804 (10JMeybohm) 05Open→03Resolved [13:35:43] first node with overlay fs is running! [13:35:48] ml-serve2005 [13:35:54] (with buster, still keeping things separate) [15:03:16] 10serviceops, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10elukey) Question about naming - we have two apt components, `component/kubernetes-future` and `component/calico-future`. Do we wan... [16:34:48] 10serviceops, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Move Docker settings for kubernetes workers to overlay fs - https://phabricator.wikimedia.org/T300744 (10JMeybohm) >>! In T300744#7693351, @elukey wrote: > Question about naming - we have two apt components, `component/kubernetes-futur... [16:35:54] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2039.codfw.wmnet with OS buster [16:50:02] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.3.0 - https://phabricator.wikimedia.org/T300804 (10dancy) Thanks @JMeybohm! [16:54:12] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2040.codfw.wmnet with OS buster [17:06:47] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2039.codfw.wmnet with OS buster completed: - mc20... [17:08:39] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2041.codfw.wmnet with OS buster [17:17:28] 10serviceops, 10SRE, 10GitLab (Infrastructure), 10Patch-For-Review: gitlab: enable IPv6 for https - https://phabricator.wikimedia.org/T300816 (10Jelto) 05Open→03In progress p:05Triage→03Medium a:03Jelto [17:23:57] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2040.codfw.wmnet with OS buster completed: - mc2040 (**PASS**) - Remo... [17:29:12] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2042.codfw.wmnet with OS buster [17:39:05] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2041.codfw.wmnet with OS buster completed: - mc2041 (**PASS**) - Remo... [17:40:17] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2043.codfw.wmnet with OS buster [17:59:37] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2042.codfw.wmnet with OS buster completed: - mc2042 (**PASS**) - Remo... [18:11:22] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2043.codfw.wmnet with OS buster completed: - mc2043 (**PASS**) - Remo... [18:20:41] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2044.codfw.wmnet with OS buster [18:21:07] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2045.codfw.wmnet with OS buster [18:25:11] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10RobH) [18:51:30] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2044.codfw.wmnet with OS buster completed: - mc2044 (**PASS**) - Remo... [18:52:20] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2045.codfw.wmnet with OS buster completed: - mc2045 (**PASS**) - Remo... [18:53:12] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2046.codfw.wmnet with OS buster [18:54:34] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2047.codfw.wmnet with OS buster [19:23:18] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2046.codfw.wmnet with OS buster completed: - mc2046 (**PASS**) - Remo... [19:26:05] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2048.codfw.wmnet with OS buster [19:26:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2047.codfw.wmnet with OS buster completed: - mc2047 (**PASS**) - Remo... [19:33:01] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2049.codfw.wmnet with OS buster [19:55:25] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2048.codfw.wmnet with OS buster completed: - mc2048 (**PASS**) - Remo... [19:58:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2050.codfw.wmnet with OS buster [20:02:03] helloo! [20:02:48] I was wondering if I could get some eyes on https://phabricator.wikimedia.org/T296051 ? [20:04:09] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2049.codfw.wmnet with OS buster completed: - mc2049 (**PASS**) - Remo... [20:14:15] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2051.codfw.wmnet with OS buster [20:28:40] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2050.codfw.wmnet with OS buster completed: - mc2050 (**PASS**) - Remo... [20:44:05] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2051.codfw.wmnet with OS buster completed: - mc2051 (**PASS**) - Remo... [21:02:19] 10serviceops: move micro sites from ganeti to kubernetes - https://phabricator.wikimedia.org/T300171 (10Dzahn) [21:48:15] 10serviceops, 10SRE, 10Wikimedia-Etherpad, 10vm-requests: create bullseye VM for Etherpad upgrade (and upgrade it:) - https://phabricator.wikimedia.org/T300568 (10Dzahn) This is scheduled for Thursday, Feb 10 2022, 9 to 10.30 UTC. Added to SRE calendar "vendor maintenance", mailed ops-l and reached out to... [22:41:51] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2052.codfw.wmnet with OS buster [22:50:15] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2053.codfw.wmnet with OS buster [23:01:29] 10serviceops, 10Parsoid, 10SRE: Move testreduce to nodejs 12 - https://phabricator.wikimedia.org/T301303 (10Arlolra) [23:12:13] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2052.codfw.wmnet with OS buster completed: - mc2052 (**PASS**) - Remo... [23:17:30] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2054.codfw.wmnet with OS buster [23:20:52] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2053.codfw.wmnet with OS buster completed: - mc2053 (**PASS**) - Remo... [23:21:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mc2055.codfw.wmnet with OS buster [23:48:31] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2054.codfw.wmnet with OS buster completed: - mc2054 (**PASS**) - Remo... [23:53:04] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mc2055.codfw.wmnet with OS buster completed: - mc2055 (**PASS**) - Remo... [23:54:35] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10Papaul) [23:55:18] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10Papaul) 05Open→03Resolved This is complete