[08:34:48] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10ayounsi) Staging the new version on the switches: `asw-a-codfw> request system software add force-host set [ /var/tmp/jinstall-ex-4300-21.4R3-S1.5... [09:50:54] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10akosiaris) 05Open→03Resolved I 've added wiktionary too to the list per the request above. Thi... [09:58:43] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): Remove outdated invert-redis-sessions cookbook - https://phabricator.wikimedia.org/T329020 (10Clement_Goubert) [10:01:08] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): March 2023 Datacenter Switchover Blockers - https://phabricator.wikimedia.org/T328770 (10Clement_Goubert) [10:12:11] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: Remove outdated invert-redis-sessions cookbook - https://phabricator.wikimedia.org/T329020 (10Clement_Goubert) 05In progress→03Resolved Wikitech page updated https://wikitech.wikimedia.org/w/index.php?title=Switch_Datacente... [10:12:21] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): March 2023 Datacenter Switchover Blockers - https://phabricator.wikimedia.org/T328770 (10Clement_Goubert) [10:25:51] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 3 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10jijiki) [10:26:52] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 3 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10taavi) [11:21:59] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2044.codfw.wmnet with OS bullseye [11:29:54] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1041.eqiad.wmnet with OS bullseye [11:31:43] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10jbond) [11:56:51] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2044.codfw.wmnet with OS bullseye completed: - mc2044 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [12:00:11] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1041.eqiad.wmnet with OS bullseye completed: - mc1041 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [12:17:58] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10herron) [12:26:36] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10ssingh) [12:29:05] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): March 2023 SRE-side Communication - https://phabricator.wikimedia.org/T329042 (10Clement_Goubert) [12:29:31] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [12:29:47] 10serviceops, 10SRE, 10Datacenter-Switchover: March 2023 SRE-side Communication - https://phabricator.wikimedia.org/T329042 (10Clement_Goubert) [12:33:06] 10serviceops, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover SRE-side Communication - https://phabricator.wikimedia.org/T329042 (10Clement_Goubert) [12:39:20] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10Vgutierrez) [12:41:06] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10Joe) To depool all services in codfw we will just need to run: ` sudo cookbook sre.discovery.datacenter-route --reason 'T327925' depool codfw `... [12:41:45] 10serviceops, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10Marostegui) @Joe @akosiaris I assume we'll depool codfw for this one too? [12:46:13] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10Joe) Please note: this won't depool `docker-registry`, which will still be active in codfw for the duration of the maintenance. [12:48:13] 10serviceops, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover SRE-side Communication - https://phabricator.wikimedia.org/T329042 (10Clement_Goubert) [12:55:28] 10serviceops, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover SRE-side Communication - https://phabricator.wikimedia.org/T329042 (10Clement_Goubert) [13:26:52] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10jcrespo) [13:33:55] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10ayounsi) For the record, full row hosts downtime done with: `sudo cookbook sre.hosts.downtime --hours 2 -r "codfw row A upgrade" -t T327925 'P{P:n... [13:34:22] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=295bf4d5-8856-488b-9ca9-06a0ff06db18) set by ayounsi@cumin1001 for 2:00:00 on 199... [13:54:24] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) @Jhancock.wm can you please check the switch port for mw2423, it looks like i have already a server connected to port 41. Thanks [14:15:18] 10serviceops, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10akosiaris) >>! In T327991#8593396, @Marostegui wrote: > @Joe @akosiaris I assume we'll depool codfw for this one too? Yeah, as a team we are simi... [15:13:44] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2045.codfw.wmnet with OS bullseye [15:13:57] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1043.eqiad.wmnet with OS bullseye [15:28:57] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10Clement_Goubert) [15:29:53] 10serviceops, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10Clement_Goubert) [15:30:18] 10serviceops, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10Clement_Goubert) [15:38:16] 10serviceops, 10MW-on-K8s, 10Datacenter-Switchover: Prepare mw-on-k8s for March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327924 (10Clement_Goubert) [15:38:20] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): March 2023 Datacenter Switchover Blockers - https://phabricator.wikimedia.org/T328770 (10Clement_Goubert) [15:39:31] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10ayounsi) 05Open→03Resolved a:03ayounsi The upgrade was smooth, ~15min hard downtime. No user impact, all the depools did their job. There wa... [15:42:14] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): Ensure sre.switchdc.mediawiki live test multi-DC compatibility - https://phabricator.wikimedia.org/T329065 (10Clement_Goubert) [15:42:36] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): Ensure sre.switchdc.mediawiki live test multi-DC compatibility - https://phabricator.wikimedia.org/T329065 (10Clement_Goubert) p:05Triage→03High [15:43:11] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1043.eqiad.wmnet with OS bullseye completed: - mc1043 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [15:48:36] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2045.codfw.wmnet with OS bullseye completed: - mc2045 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [15:50:03] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10colewhite) [16:07:14] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [16:14:57] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1045.eqiad.wmnet with OS bullseye [16:15:16] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2046.codfw.wmnet with OS bullseye [16:19:54] 10serviceops, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 7 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ayounsi) p:05Triage→03Medium [16:21:58] 10serviceops, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 7 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ayounsi) [16:24:08] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 3 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10akosiaris) We 've discussed this internally within the team. **We realize that it's not possible to exclude wikitech from the s... [16:25:25] 10serviceops, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 7 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ayounsi) [16:30:07] 10serviceops, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 8 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10colewhite) [16:31:54] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 3 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10akosiaris) @nskaggs, @bd808 (feel free to add others), let me know what you think. [16:38:38] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 3 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10bd808) >>! In T328768#8594394, @akosiaris wrote: > @nskaggs, @bd808 (feel free to add others), let me know what you think. Opt... [16:38:52] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 3 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10jcrespo) Please apologies if I am wrong, which I am probably am, but... > Wikitech read requests will flow to eqiad, and write... [16:46:52] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1045.eqiad.wmnet with OS bullseye completed: - mc1045 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [16:50:07] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2046.codfw.wmnet with OS bullseye completed: - mc2046 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [16:59:44] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 3 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10akosiaris) > How is this possible, if there are no codfw app servers serving wikitech? As I understand (and hopefully I am the... [17:03:21] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 3 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10jcrespo) >>! In T328768#8594700, @akosiaris wrote: >> How is this possible, if there are no codfw app servers serving wikitech?... [17:21:38] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2047.codfw.wmnet with OS bullseye [17:22:16] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1046.eqiad.wmnet with OS bullseye [17:22:50] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 3 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10jcrespo) No blocker on my side, then. Supporting path 5 (security worried me more than performance). [17:23:48] 10serviceops, 10Service-deployment-requests: New Service Request 'security-api' - https://phabricator.wikimedia.org/T325147 (10akosiaris) >>! In T325147#8590684, @STran wrote: > Yes I'm so sorry! This slipped my mind. I wrote up some internal documentation for the team that might be useful in this case: https:... [17:38:59] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Brycehughes) @akosiaris How painful would a full cache purge be? It would be great to have the pro... [17:52:01] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1046.eqiad.wmnet with OS bullseye completed: - mc1046 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [17:53:32] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2047.codfw.wmnet with OS bullseye completed: - mc2047 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [18:17:28] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1047.eqiad.wmnet with OS bullseye [18:18:09] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2048.codfw.wmnet with OS bullseye [18:47:09] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1047.eqiad.wmnet with OS bullseye completed: - mc1047 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [18:53:39] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2048.codfw.wmnet with OS bullseye completed: - mc2048 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [19:03:58] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2049.codfw.wmnet with OS bullseye [19:04:19] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1049.eqiad.wmnet with OS bullseye [19:33:46] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1049.eqiad.wmnet with OS bullseye completed: - mc1049 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [19:39:07] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2049.codfw.wmnet with OS bullseye completed: - mc2049 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [20:13:41] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [20:21:47] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1051.eqiad.wmnet with OS bullseye [20:21:58] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2050.codfw.wmnet with OS bullseye [20:50:05] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1051.eqiad.wmnet with OS bullseye completed: - mc1051 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [20:57:24] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2050.codfw.wmnet with OS bullseye completed: - mc2050 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [20:58:19] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2051.codfw.wmnet with OS bullseye [21:01:05] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1053.eqiad.wmnet with OS bullseye [21:17:29] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Jaifroid) Thank you, @akosiaris. It's good to be cautious and watch out for regressions. If there... [21:29:09] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1053.eqiad.wmnet with OS bullseye completed: - mc1053 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [21:34:34] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2051.codfw.wmnet with OS bullseye completed: - mc2051 (**PASS**) - Downtimed on Icinga/Alertmanager - Disa... [22:04:43] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [22:33:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [22:42:15] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [23:13:10] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul)