[00:03:43] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704894 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp4043.ulsfo.wmnet with OS trixie executed with errors: - cp4043 (**FAIL**) - Downtimed on Icinga/Alertmanager - D... [00:10:56] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704903 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp4040.ulsfo.wmnet with OS trixie completed: - cp4040 (**PASS**) - Removed from Puppet and PuppetDB if present and d... [00:11:50] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704904 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp4043.ulsfo.wmnet with OS trixie [00:41:50] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704951 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp4042.ulsfo.wmnet with OS trixie completed: - cp4042 (**WARN**) - Removed from Puppet and PuppetDB if present and d... [00:43:23] RESOLVED: ErrorBudgetBurn: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn [00:46:00] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704954 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin2002 for host cp4044.ulsfo.wmnet with OS trixie [00:46:01] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704955 (10CDobbins) [01:05:12] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp4043.ulsfo.wmnet with OS trixie completed: - cp4043 (**PASS**) - Removed from Puppet and PuppetDB if present and d... [01:10:01] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704963 (10CDobbins) [01:23:02] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704965 (10BCornwall) [01:41:42] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11704979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin2002 for host cp4044.ulsfo.wmnet with OS trixie completed: - cp4044 (**WARN**) - Downtimed on Icinga/Alertmanager - Disabled Pup... [01:43:26] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#11704980 (10RobH) Tech is running late, their dispatcher called me to let me know. They were set to be onsite at 7AM, but it will now be closer to 10:30AM / 19:30 Pacific [02:51:55] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#11705018 (10RobH) Tech is onsite and performing the hw power distro board swap on cp5022 [03:39:53] FIRING: ErrorBudgetBurn: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DErrorBudgetBurn