[07:41:28] 10serviceops, 10Content-Transform-Team-WIP, 10Maintenance-Worktype, 10Wikimedia-Incident: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) - https://phabricator.wikimedia.org/T344324 (10JMeybohm) Could we just circumvent the problem by enabling the thanos-swift listener in tegolas servi... [07:48:05] 10serviceops, 10Content-Transform-Team-WIP, 10Maintenance-Worktype, 10Wikimedia-Incident: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) - https://phabricator.wikimedia.org/T344324 (10Joe) >>! In T344324#9112384, @JMeybohm wrote: > Could we just circumvent the problem by enabling the... [09:27:18] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: admin_ng private data no longer in /etc/helmfile-defaults/private/admin/ - https://phabricator.wikimedia.org/T297417 (10JMeybohm) This just bit me as it is a tad confusing - I decided to put admin secrets back in place as it is more clean (... [10:04:34] 10serviceops, 10AQS2.0, 10Cassandra, 10SRE, 10Service-deployment-requests: AQS 2.0 differentially private pageviews deploy API - https://phabricator.wikimedia.org/T343855 (10JAllemandou) >>! In T343855#9111286, @Htriedman wrote: > 1. Ensure that (a) historical data is loaded into cassandra (currently thi... [10:35:01] 10serviceops, 10Beta-Cluster-Infrastructure, 10Thumbor, 10Beta-Cluster-reproducible: deployment-prep needs a Thumbor instance - https://phabricator.wikimedia.org/T344605 (10hnowlan) [10:44:09] 10serviceops, 10Content-Transform-Team-WIP, 10Maintenance-Worktype, 10Wikimedia-Incident: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) - https://phabricator.wikimedia.org/T344324 (10jijiki) [11:01:56] 10serviceops, 10MW-on-K8s, 10Wikidata, 10Wikidata-Termbox, and 2 others: Migrate termbox to mw-api-int - https://phabricator.wikimedia.org/T334064 (10Joe) Just deployed the change to termbox-test, and I still see my test url `http://termbox-test.staging.svc.eqiad.wmnet:3031/termbox?entity=Q229877&revision... [11:39:39] 10serviceops, 10MW-on-K8s: mw-on-k8s app container CPU throttling at low average load - https://phabricator.wikimedia.org/T342748 (10Clement_Goubert) 05In progress→03Resolved [11:39:46] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [11:46:55] 10serviceops, 10MW-on-K8s, 10SRE: mw-on-k8s tls-proxy container CPU throttling at low average load - https://phabricator.wikimedia.org/T344814 (10Clement_Goubert) [11:48:06] 10serviceops, 10MW-on-K8s: mw-on-k8s tls-proxy container CPU throttling at low average load - https://phabricator.wikimedia.org/T344814 (10Clement_Goubert) [11:48:23] 10serviceops, 10MW-on-K8s: mw-on-k8s tls-proxy container CPU throttling at low average load - https://phabricator.wikimedia.org/T344814 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High [11:48:33] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [12:21:52] 10serviceops, 10Maps, 10Regression, 10Russian-Sites: Vandal attack on OpenStreetMap affected Wikimedia Maps - https://phabricator.wikimedia.org/T344753 (10Jgiannelos) [12:28:28] Just a heads up there is a kafka connectivity issue related to this UBN issue T344753 . I noticed the failure while waiting to verify tiles are updated. [12:40:44] 10serviceops, 10MW-on-K8s: mw-on-k8s tls-proxy container CPU throttling at low average load - https://phabricator.wikimedia.org/T344814 (10Clement_Goubert) Dumping the envoy configuration in one of our containers as well as there being no CLI flag set for it means envoy is setting it's number of threads to the... [12:50:55] nemo-yiannis: AFAICT that only happened once (on one tegola replica) in the last two days - does not seem to be a generic issue to me [12:51:36] hey jayme: I need to deploy a new version of the eventstreams service on k8s [12:52:04] i saw you did some changes to the helm chart, that have not been deployed yet. Is it safe to proceed or shall I wait? [12:52:35] jayme: even if its a transient issue someone needs to trigger the cronjobs again (unless they retry somehow) so we don't wait until tomorrow for vandalism to be fixed [12:54:54] 10serviceops, 10Maps, 10Regression, 10Russian-Sites: Vandal attack on OpenStreetMap affected Wikimedia Maps - https://phabricator.wikimedia.org/T344753 (10JMeybohm) >>! In T344753#9113224, @Jgiannelos wrote: > That means the pregeneration failed and tiles are stuck in the vandalized version. > For more log... [12:56:34] gmodena: safe to deploy [12:57:21] jayme awesome thanks. Just to be sure: we are deploying a configuration change, this should not require depooling. Did I understand correctly? [12:59:00] gmodena: for my part in the diff no depooling is required. rolling update will rake care of that [12:59:43] jayme thanks for clarifying. [13:01:58] nemo-yiannis: I do have a lack of understanding of maps and tegola stuff, sorry. So you're saying the tegola-vector-tiles-main-pregeneration cronjob in eqiad needs to be re-run? [13:02:31] yes, the kafka connectivity error caused them to exit without doing the tiles pregeneration [13:12:52] ah, that I can do :) [13:15:53] ofc that leads to the same error again... [13:17:46] I don't get why there is only one error. the cronjob spawns 6 times by default - shouldn't there be 6? [13:18:30] i dont know :/ [13:18:44] but yeah makes sense what you are saying [13:20:43] On a different note we are using directly one node and not some sort of discovery URL. Is the actual kafka endpoint reachable? kafka-main1001.eqiad.wmnet [13:24:52] <_joe_> nemo-yiannis: unless someone turned it off yesterday, yes [13:25:57] also if i remember correctly we are waiting for envoy to be up and then we start consuming events and it looks like the curl to the envoy healthcheck endpoint is also failing [13:26:01] <_joe_> nemo-yiannis: I'm not sure why you'd use a single broker and not the full list [13:27:30] effie: do you know more maybe ^^ [13:27:59] the kafka broker will advertise the others brokers btw [13:28:10] jayme: reading [13:28:35] the more i look to the logs the more i think there is something wrong with the connection to envoy and not actually reaching kafka broker. [13:28:47] the kafka protocol is a bit weird, but IIRC upon connection, the broker informs the client of the entire list of brokers and the client library does the rest [13:29:18] so while it might be a misconfiguration, it "should" be using all brokers [13:29:18] this is my understanding too [13:29:39] s/miconfiguration/not fully correct configuration/ [13:58:02] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [14:20:36] <_joe_> akosiaris: oh yes that wasn't the point, I think we have functions to get the brokers lists [14:24:20] 10serviceops, 10Maps, 10Regression, 10Russian-Sites: Vandal attack on OpenStreetMap affected Wikimedia Maps - https://phabricator.wikimedia.org/T344753 (10Jgiannelos) It looks like kicking the cronjobs manually did the trick cc @jijiki [14:27:15] for the time being, the tegola pregeneration job run manually on eqiad, we are monitoring [14:27:28] 10serviceops, 10Maps, 10Regression, 10Russian-Sites: Vandal attack on OpenStreetMap affected Wikimedia Maps - https://phabricator.wikimedia.org/T344753 (10IKhitron) Can't you just freeze the map as it was three days ago, while the task is in progress? [14:30:14] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Wikikube CPU capacity issue - https://phabricator.wikimedia.org/T343306 (10Jclark-ctr) [15:04:44] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes10[27-56] - https://phabricator.wikimedia.org/T342533 (10Jclark-ctr) [15:30:29] 10serviceops, 10Maps, 10Regression, 10Russian-Sites: Vandal attack on OpenStreetMap affected Wikimedia Maps - https://phabricator.wikimedia.org/T344753 (10Jgiannelos) Unfortunately reverting the map tiles to a specific state from scratch is going to take days. [15:50:58] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: admin_ng private data no longer in /etc/helmfile-defaults/private/admin/ - https://phabricator.wikimedia.org/T297417 (10JMeybohm) 05Open→03Resolved a:03JMeybohm I think this is done. `profile::kubernetes::deployment_server_secrets::admin_services:` is bac... [16:06:48] https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red?orgId=1&viewPanel=65&from=now-6h&to=now&refresh=1m [16:07:09] ^ any ideas about this or already looked into? reqrate bump ~14:05, seems to align with a deploy ending? [16:11:36] 10serviceops, 10SRE, 10decommission-hardware, 10ops-eqiad: Decommission thumbor100[12] - https://phabricator.wikimedia.org/T344598 (10RobH) [16:14:07] It was about clienthints, I don't know much about this [16:14:14] (and I'm about to head out) [16:15:47] <_joe_> kostajh: ^^ maybe we need to rollback clienthints [17:03:04] 10serviceops, 10Maps, 10Regression, 10Russian-Sites: Vandal attack on OpenStreetMap affected Wikimedia Maps - https://phabricator.wikimedia.org/T344753 (10TheDJ) Ideally, we’d have an api, where trusted ppl can enter a node id or something, and trigger a purge and regenerate for the item and its derivative... [17:05:11] 10serviceops, 10SRE, 10decommission-hardware, 10ops-eqiad: Decommission thumbor100[12] - https://phabricator.wikimedia.org/T344598 (10ayounsi) I'm going to hijack those 2 hosts before they get decommissioned for some tests. I'll rename them ganeti-test1001/1002. [17:18:05] _joe_: I don't see how that is related to the patch that was just reverted (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/951855) [17:18:38] <_joe_> kostajh: when did we extend clienthints to all users? [17:18:57] <_joe_> kostajh: me neither [17:19:01] _joe_: yesterday [17:19:13] in https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20230822T2000 [17:19:51] <_joe_> still i see a drop in request https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red?orgId=1&viewPanel=65&from=now-6h&to=now&refresh=1m [17:19:57] the patch that was just rolled back fixed a narrow case where someone (probably manually tampering with the collection endpoint) could trigger a MySQL error with duplicate key entry. [17:20:27] <_joe_> just before the deployment though [17:20:51] <_joe_> yeah sorry, probably was correlation not causation [17:20:51] right, it's not synced yet afaict [17:21:06] <_joe_> effie: it looks like it was a traffic pattern actually [17:21:14] kostajh: yeah and the observed rps increase seems to have self-resolved after almost exactly 3h anyways [17:21:26] still not sure what exactly it was, but I haven't dug into it either [17:22:17] yeah I'm looking for other causes, I don't see anything in webrequest-sampled yet [17:23:22] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2053.codfw.wmnet with OS bullseye [17:24:30] after the rollout to group2 yesterday (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20230822T2000) I would expect to see a jump in POST requests (made by clients to the CheckUser REST endpoint) [17:26:37] all (anon?) clients are sending a new POST? how often? [17:26:38] but that would be probably < 1 million in total in the last 24 hours, across group2 wikis, looking at cu_useragent_clienthints_map on a couple of wikis [17:27:07] ok, i've abandoned the backport. [17:27:17] after saving an edit, when the postEdit JS hook fires, for browsers that implement client hints JS API (Chrome and Edge). For anon and logged-in users. [17:27:29] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [17:27:40] ok [17:28:55] https://www.mediawiki.org/wiki/Extension:CheckUser/Client_Hints#When_is_the_data_collected? [17:31:00] since this hasn't come back, and doesn't seem to have actually been deployment-related, I'm inclined to just treat it as a Steve's Maxim issue and leave it alone unless anyone objects [17:31:12] ("never look at a dashboard unless you're looking for something, because you'll always find something") [17:32:27] 10serviceops, 10Data Products, 10RESTbase Sunsetting, 10Code-Health-Objective, 10Patch-For-Review: Route to new AQS Knowledge Gaps endpoint - https://phabricator.wikimedia.org/T342213 (10VirginiaPoundstone) @Milimetric & @fkaelin: is it trivial to add these headers to Knowledge Gaps in AQS? If it is non-... [17:32:48] yeah :) [17:33:40] I think of it more like "never look at metrics without a logical hypothesis for what you expect to find and confirm" [18:10:14] <_joe_> rzl: we need to re-revert I guess if no one did [18:11:12] I think it was, there was a re-revert commit [18:11:58] err no, it was an abandonment of the original, which was never synced? [18:12:01] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/951855/ [18:12:29] (the original revert, to be slightly-less-unclear) [18:12:45] yeah the revert was never completed, all good [18:38:56] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2053.codfw.wmnet with OS bullseye executed with errors: - kubernetes20... [18:45:25] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2053.codfw.wmnet with OS bullseye [19:31:32] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2053.codfw.wmnet with OS bullseye completed: - kubernetes2053 (**PASS*... [19:47:47] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2052.codfw.wmnet with OS bullseye [20:11:16] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2051.codfw.wmnet with OS bullseye [20:15:48] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2050.codfw.wmnet with OS bullseye [20:15:59] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2049.codfw.wmnet with OS bullseye [20:32:53] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2052.codfw.wmnet with OS bullseye completed: - kubernetes2052 (**PASS*... [20:46:51] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2048.codfw.wmnet with OS bullseye [20:54:33] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2051.codfw.wmnet with OS bullseye completed: - kubernetes2051 (**PASS*... [20:58:38] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2047.codfw.wmnet with OS bullseye [21:02:24] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2050.codfw.wmnet with OS bullseye completed: - kubernetes2050 (**PASS*... [21:02:42] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2049.codfw.wmnet with OS bullseye completed: - kubernetes2049 (**WARN*... [21:44:45] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2047.codfw.wmnet with OS bullseye completed: - kubernetes2047 (**PASS*... [21:49:52] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2046.codfw.wmnet with OS bullseye [22:05:33] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2045.codfw.wmnet with OS bullseye [22:07:04] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2048.codfw.wmnet with OS bullseye executed with errors: - kubernetes20... [22:09:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2044.codfw.wmnet with OS bullseye [22:17:20] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) @Jhancock.wm hey looks like i have no link on kubernetes2048. Thanks ` papaul@asw-d-codfw> show interfaces descriptions ge-5/0/28 Interface Admin Link Descript... [22:26:44] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2043.codfw.wmnet with OS bullseye [22:31:58] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2046.codfw.wmnet with OS bullseye completed: - kubernetes2046 (**PASS*... [22:49:44] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2042.codfw.wmnet with OS bullseye [22:52:14] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2044.codfw.wmnet with OS bullseye completed: - kubernetes2044 (**PASS*... [22:59:49] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2041.codfw.wmnet with OS bullseye [23:00:28] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [23:01:49] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Papaul) [23:09:46] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2043.codfw.wmnet with OS bullseye completed: - kubernetes2043 (**PASS*... [23:30:27] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2040.codfw.wmnet with OS bullseye [23:34:19] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2042.codfw.wmnet with OS bullseye completed: - kubernetes2042 (**PASS*... [23:45:42] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2041.codfw.wmnet with OS bullseye completed: - kubernetes2041 (**PASS*...