[00:01:54] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3054.esams.wmnet with OS bullseye completed: - cp3054 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [00:10:01] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [00:37:56] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3055.esams.wmnet with OS bullseye completed: - cp3055 (**PASS**) - Removed from Puppet and PuppetDB if present -... [00:41:30] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [02:32:32] 10Traffic, 10MediaWiki-Core-HTTP-Cache, 10MediaWiki-REST-API, 10RESTbase Sunsetting, and 4 others: Determine http cache control and active purging for REST endpoints serving parsoid output - https://phabricator.wikimedia.org/T308424 (10VirginiaPoundstone) [08:17:12] 10netops, 10Infrastructure-Foundations, 10SRE: eqiad/codfw virtual-chassis upgrades - https://phabricator.wikimedia.org/T327248 (10ayounsi) [09:21:18] 10netops, 10Infrastructure-Foundations, 10serviceops: Optimize k8s same row traffic flows - https://phabricator.wikimedia.org/T328523 (10ayounsi) [13:46:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: Optimize k8s same row traffic flows - https://phabricator.wikimedia.org/T328523 (10ayounsi) [14:56:58] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: Calico and BFD - https://phabricator.wikimedia.org/T328338 (10ayounsi) As a datapoint, I pushed this change https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/885814 and it showed up in the Bird config without a drop in the BGP sessi... [15:31:16] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5019.eqsin.wmnet with OS bullseye [15:38:49] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: Optimize k8s same row traffic flows - https://phabricator.wikimedia.org/T328523 (10ayounsi) With the above patch, plus the following test router config: `lang=diff [edit policy-options] + policy-statement kubestage_test_out { + term stage... [15:51:45] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5019.eqsin.wmnet with OS bullseye executed with errors: - cp5019 (**FAIL**) - Downtimed on Icinga/Alertmanager -... [15:54:01] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5019.eqsin.wmnet with OS bullseye [16:16:30] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin: eqsin hosts are not rebooting when running sre.hosts.reimage cookbook - https://phabricator.wikimedia.org/T327812 (10ssingh) @Volans and I were discussing this on IRC today, some more observations with cp5019, that failed the first attempt but worked on the second.... [16:25:32] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5030.eqsin.wmnet with OS bullseye [16:31:43] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5030.eqsin.wmnet with OS bullseye executed with errors: - cp5030 (**FAIL**) - Downtimed on Icinga/Alertmanager -... [16:33:10] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp3056.esams.wmnet with OS bullseye [16:34:01] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5030.eqsin.wmnet with OS bullseye [16:42:40] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5030.eqsin.wmnet with OS bullseye executed with errors: - cp5030 (**FAIL**) - Removed from Puppet and PuppetDB if p... [16:42:56] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5030.eqsin.wmnet with OS bullseye [16:45:35] (PurgedHighEventLag) firing: (2) High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [16:50:35] (PurgedHighEventLag) firing: (9) High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [16:54:37] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp3057.esams.wmnet with OS bullseye [16:55:35] (PurgedHighEventLag) firing: (17) High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [16:57:37] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5030.eqsin.wmnet with OS bullseye executed with errors: - cp5030 (**FAIL**) - Removed from Puppet and PuppetDB if p... [16:58:01] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5030.eqsin.wmnet with OS bullseye [17:00:35] (PurgedHighEventLag) firing: (23) High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [17:05:35] (PurgedHighEventLag) firing: (7) High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [17:10:35] (PurgedHighEventLag) resolved: (5) High event process lag with purged on cp5017:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [17:17:16] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5019.eqsin.wmnet with OS bullseye completed: - cp5019 (**WARN**) - Removed from Puppet and PuppetDB if present -... [17:22:31] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3056.esams.wmnet with OS bullseye completed: - cp3056 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [17:23:41] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [17:23:56] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp3058.esams.wmnet with OS bullseye [17:40:39] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3057.esams.wmnet with OS bullseye completed: - cp3057 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [17:41:27] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [17:41:52] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp3059.esams.wmnet with OS bullseye [18:11:02] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5030.eqsin.wmnet with OS bullseye completed: - cp5030 (**PASS**) - Removed from Puppet and PuppetDB if present -... [18:12:56] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3058.esams.wmnet with OS bullseye completed: - cp3058 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [18:13:36] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [18:13:58] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp3060.esams.wmnet with OS bullseye [18:19:46] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp1075:9331 is unreachable - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [18:20:07] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) [18:20:11] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) [18:22:46] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5021.eqsin.wmnet with OS bullseye [18:29:17] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5021.eqsin.wmnet with OS bullseye executed with errors: - cp5021 (**FAIL**) - Downtimed on Icinga/Alertmanager -... [18:29:31] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5021.eqsin.wmnet with OS bullseye [18:31:21] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3059.esams.wmnet with OS bullseye completed: - cp3059 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [18:31:51] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [18:32:13] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp3061.esams.wmnet with OS bullseye [18:37:16] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5031.eqsin.wmnet with OS bullseye [18:46:53] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5031.eqsin.wmnet with OS bullseye executed with errors: - cp5031 (**FAIL**) - Downtimed on Icinga/Alertmanager -... [18:47:14] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5031.eqsin.wmnet with OS bullseye [18:55:33] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5031.eqsin.wmnet with OS bullseye executed with errors: - cp5031 (**FAIL**) - Removed from Puppet and PuppetDB if p... [18:55:47] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5031.eqsin.wmnet with OS bullseye [19:02:26] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3060.esams.wmnet with OS bullseye completed: - cp3060 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [19:03:15] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [19:03:29] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp3062.esams.wmnet with OS bullseye [19:14:26] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10jbond) @ssingh i have finished with cp1075, i have upgraded it to the most recent network, bios and idrac version. in relation to other servers that you may have issues with i have noticed that any machine with... [19:17:47] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3061.esams.wmnet with OS bullseye completed: - cp3061 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [19:20:24] 10Wikimedia-Apache-configuration: remove pk.wikimedia.org -> wikimediapakistan.org redirect - https://phabricator.wikimedia.org/T328596 (10Nintendofan885) [19:24:56] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [19:25:05] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp3063.esams.wmnet with OS bullseye [19:27:52] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) >>! In T321309#8579719, @jbond wrote: > @ssingh i have finished with cp1075, i have upgraded it to the most recent network, bios and idrac version. in relation to other servers that you may have issues w... [19:37:57] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5021.eqsin.wmnet with OS bullseye completed: - cp5021 (**PASS**) - Removed from Puppet and PuppetDB if present -... [19:42:24] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) [19:48:53] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3062.esams.wmnet with OS bullseye completed: - cp3062 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [19:49:52] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [19:49:58] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp3064.esams.wmnet with OS bullseye [20:00:20] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5031.eqsin.wmnet with OS bullseye completed: - cp5031 (**PASS**) - Removed from Puppet and PuppetDB if present -... [20:03:18] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) [20:09:53] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3063.esams.wmnet with OS bullseye completed: - cp3063 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [20:13:22] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10jbond) >>! In T321309#8579793, @ssingh wrote: >>>! In T321309#8579719, @jbond wrote: >> @ssingh i have finished with cp1075, i have upgraded it to the most recent network, bios and idrac version. in relation to... [20:13:44] sukhe: updated task, but have copied an image to cumin2002:/srv/firmware/poweredge-r440/IDRAC/iDRAC-with-Lifecycle-Controller_Firmware_G6W0W_WN64_3.30.30.30_A00.EXE [20:13:48] thanks! [20:13:56] np [20:14:34] I found 3.21 which didn't work [20:14:51] but yeah 3.30 matches what we saw in a random Dell forum [20:15:05] yes has to be 3.30.30.30 (i think the web interface says that, but may have been in some redfish log) [20:15:13] ok thanks [20:15:26] so the idea is to update iDRAC manually to 3.30 and then do a straight upgrade to 6.x? [20:15:34] then the NIC can be done via the cookbook [20:16:01] oh actully if you try to upgrade to 6.* you dont get a good error message but if you try to upgrade from < 3.30 to 4.* or 5.* you get a more helpfull error message in the ui [20:16:11] ha [20:16:29] but anyway yes the plan woul be: [20:16:32] but what was your update path? 3.15 to 3.30 via the HTTP mgmt interface? [20:16:35] and then to 6.x from the cookbook? [20:16:44] 1. manually upgared to 3.30.30.30 [20:17:26] then upgrade idrac and nic (which you can do in one cook book run with `-c nic -c idrac (after the aformentioned patch is merged)` [20:17:46] so [20:17:51] 1. upgrade to 3.30.30.30 [20:18:05] 2. upgrade to latest idrac 6.10.0.0 [20:18:08] cool thanks! [20:18:13] 2. upgrade to nic [20:18:17] 3. reimage [20:18:44] the two 2's can be done wiht one command [20:19:05] cookbook sre.hardware.upgrade-firmware -c nic -c idrac $fqdn [20:19:23] yeah, that's what we have been doing [20:19:28] ok cool [20:19:38] I guess for finding out which hosts are on 3.15, time to play with the API instead of doing it one by one :) [20:19:42] and ill chase vola.ns to get the updates merged tomorrow [20:19:52] because in !eqiad, it was fairly simple to assume that everything needs to be upgraded [20:19:54] actully that is in puppetdb [20:19:57] oh really? [20:20:12] can I query for a list of fqds? [20:20:15] *fqdns [20:20:35] https://puppetboard.wikimedia.org/node/cp1076.eqiad.wmnet [20:20:36] we have this in https://puppetboard.wikimedia.org/fact/firmware_idrac [20:20:38] oh I see it here [20:20:54] or try the snippet in https://phabricator.wikimedia.org/T328593 [20:21:17] :D [20:21:19] thank you! [20:21:34] exactly what we needed, since https://puppetboard.wikimedia.org/query is disabled anyway [20:21:57] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [20:22:04] you need to run that python code from the puppetdb srveres, you should be able to copy the code i have in puppetdb1002.eqiad.wmnet:/home/jbond/pql/facts.py [20:22:11] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp3065.esams.wmnet with OS bullseye [20:22:23] t shold be fairly easy to update for your use case [20:22:36] can also check https://wikitech.wikimedia.org/wiki/Puppet/PQL for other pql examples [20:22:36] yeah, this is a great start [20:22:46] I was going to look at your /home/jbond/spiecrack_shell.py [20:23:03] and as always ping if you get stuck [20:23:07] thanks for the help [20:23:16] the spicerack shell is just some minor helpers its tiny tbh [20:23:19] np [20:24:16] and of course mine is a copy /home/vola.ns/spicerack_shell.py :) [20:34:01] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3064.esams.wmnet with OS bullseye completed: - cp3064 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [20:40:24] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [21:10:22] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp3065.esams.wmnet with OS bullseye completed: - cp3065 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [21:17:51] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [21:43:36] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp5022.eqsin.wmnet with OS bullseye [22:40:35] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp5022.eqsin.wmnet with OS bullseye executed with errors: - cp5022 (**FAIL**) - Downtimed on Icinga/Alertmanager -... [22:54:15] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp5022.eqsin.wmnet with OS bullseye