[00:17:30] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw, 10ops-eqiad: Update iDRAC and NIC firmware on mc-gp* hosts - https://phabricator.wikimedia.org/T329323 (10Papaul) [00:22:47] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2439.codfw.wmnet with OS buster completed: - mw2439 (**PASS**) - Removed from Pupp... [00:22:51] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2438.codfw.wmnet with OS buster completed: - mw2438 (**PASS**) - Removed from Pupp... [00:39:11] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster [00:40:58] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2441.codfw.wmnet with OS buster [00:43:46] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2442.codfw.wmnet with OS buster [00:46:46] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2443.codfw.wmnet with OS buster [01:35:29] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster executed with errors: - mw2440 (**FAIL**) - Remove... [01:37:17] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2441.codfw.wmnet with OS buster executed with errors: - mw2441 (**FAIL**) - Remove... [01:40:01] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2442.codfw.wmnet with OS buster executed with errors: - mw2442 (**FAIL**) - Remove... [01:43:00] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2443.codfw.wmnet with OS buster executed with errors: - mw2443 (**FAIL**) - Remove... [01:52:26] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster [02:01:38] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster executed with errors: - mw2440 (**FAIL**) - Remove... [02:04:56] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) @Jhancock.wm can you please take a look at mw244[0-3], it looks like you connected the network cable to NIC 2 and not NIC 1. Thank you [02:07:27] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [02:12:12] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2444.codfw.wmnet with OS buster [02:18:35] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2445.codfw.wmnet with OS buster [02:19:25] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2446.codfw.wmnet with OS buster [02:44:12] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2447.codfw.wmnet with OS buster [02:56:04] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2444.codfw.wmnet with OS buster completed: - mw2444 (**PASS**) - Removed from Pupp... [03:04:35] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2446.codfw.wmnet with OS buster completed: - mw2446 (**PASS**) - Removed from Pupp... [03:04:40] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2445.codfw.wmnet with OS buster completed: - mw2445 (**PASS**) - Removed from Pupp... [03:29:11] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2447.codfw.wmnet with OS buster completed: - mw2447 (**PASS**) - Removed from Pupp... [06:48:55] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: spicerack.mysql_legacy errors on get_core_masters_heartbeats when checking x2 - https://phabricator.wikimedia.org/T329533 (10Marostegui) They should probably be skipped as we have two masters being written (one per DC) and they need to remai... [08:32:24] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: K8s etcd on bullseye show TLS errors in logs - https://phabricator.wikimedia.org/T329556 (10elukey) The interesting bit is that the aux cluster (on bullseye) doesn't show the problem. I am won... [08:38:53] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, 10Kubernetes: K8s etcd on bullseye show TLS errors in logs - https://phabricator.wikimedia.org/T329556 (10elukey) Ok mistery explained: https://etcd.io/docs/v3.3/op-guide/clustering/#dns-discovery > If etc... [09:11:19] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: spicerack.mysql_legacy errors on get_core_masters_heartbeats when checking x2 - https://phabricator.wikimedia.org/T329533 (10Volans) @Marostegui Just to clarify and avoid confusion, are you suggesting to remove `x2` entirely from the spicer... [09:21:10] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10akosiaris) >>! In T226931#8607179, @Brycehughes wrote: > @akosiaris Aaaaand now it seems to be be... [09:32:12] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: spicerack.mysql_legacy errors on get_core_masters_heartbeats when checking x2 - https://phabricator.wikimedia.org/T329533 (10Marostegui) @Volans I am unsure. How do we treat parsercache at the moment? x2 is special in the sense that it does... [09:35:28] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: spicerack.mysql_legacy errors on get_core_masters_heartbeats when checking x2 - https://phabricator.wikimedia.org/T329533 (10Volans) >>! In T329533#8613691, @Marostegui wrote: > @Volans I am unsure. How do we treat parsercache at the moment?... [09:40:24] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: spicerack.mysql_legacy errors on get_core_masters_heartbeats when checking x2 - https://phabricator.wikimedia.org/T329533 (10Marostegui) >>! In T329533#8613695, @Volans wrote: >>>! In T329533#8613691, @Marostegui wrote: >> @Volans I am unsur... [10:16:07] 10serviceops, 10Thumbor: Incorrect thumbnail being returned by drmrs, eqiad and esams - https://phabricator.wikimedia.org/T328875 (10akosiaris) >>! In T328875#8605050, @akosiaris wrote: > Thanks for this > >>>! In T328875#8598964, @Jonesey95 wrote: >> https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(tech... [10:27:35] 10serviceops, 10Thumbor: Incorrect thumbnail being returned by drmrs, eqiad and esams - https://phabricator.wikimedia.org/T328875 (10Jhawkinson) I'm out of my depth here, so I'm speculating, but these items from the `swift list` look peculiar: ` 5/5d/EPA_Deputy_Admin_Bob_Perciasepe.jpg/200px-EPA_Deputy_Admin_... [11:39:34] 10serviceops: Rewrite sre.mediawiki.restart-appservers using spicerack's class API - https://phabricator.wikimedia.org/T329616 (10Clement_Goubert) [11:39:43] 10serviceops: Rewrite sre.mediawiki.restart-appservers using spicerack's class API - https://phabricator.wikimedia.org/T329616 (10Clement_Goubert) p:05Triage→03Low [11:44:02] 10serviceops, 10Thumbor: Incorrect thumbnail being returned by drmrs, eqiad and esams - https://phabricator.wikimedia.org/T328875 (10MatthewVernon) We don't currently ever expire thumbnails, but are moving gradually towards doing so. [11:58:45] 10serviceops, 10Thumbor: Incorrect thumbnail being returned by drmrs, eqiad and esams - https://phabricator.wikimedia.org/T328875 (10TheDJ) > Do we understand what leads to the creation of the last 4 of those? 201px and 202px are probably just ppl (like me), retrieving a direct url of the 200px version, and a... [14:59:47] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2448.codfw.wmnet with OS buster [15:01:57] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2449.codfw.wmnet with OS buster [15:04:25] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2450.codfw.wmnet with OS buster [15:07:31] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2451.codfw.wmnet with OS buster [15:08:38] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [15:22:50] 10serviceops, 10Maps: Re-import full planet data into eqiad and codfw - https://phabricator.wikimedia.org/T314472 (10TheDJ) Should this be closed ? [15:26:38] 10serviceops, 10Maps: Re-import full planet data into eqiad and codfw - https://phabricator.wikimedia.org/T314472 (10Jgiannelos) We still have some docs pending in order to close this ticket. [16:04:31] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2449.codfw.wmnet with OS buster completed: - mw2449 (**PASS**) - Removed from Pupp... [16:04:34] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2450.codfw.wmnet with OS buster completed: - mw2450 (**PASS**) - Removed from Pupp... [16:04:37] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2448.codfw.wmnet with OS buster completed: - mw2448 (**PASS**) - Removed from Pupp... [16:04:40] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2451.codfw.wmnet with OS buster completed: - mw2451 (**PASS**) - Removed from Pupp... [16:45:05] 10serviceops, 10Thumbor: Incorrect thumbnail being returned by drmrs, eqiad and esams - https://phabricator.wikimedia.org/T328875 (10akosiaris) >>! In T328875#8614019, @TheDJ wrote: >> Do we understand what leads to the creation of the last 4 of those? > > 201px and 202px are probably just ppl (like me), retr... [16:46:30] hey folks. several instances in the 'appservers' cloud vps project are failing to run puppet, please fix. https://prometheus-alerts.wmcloud.org/?q=%40state%3Dactive&q=project%3Dappservers [16:56:57] claime: ^^ I think some of them are your's, right? I'll take care of the k8s-* ones [17:00:07] jayme: Possibly, but I had passed my stack to effie and haven't used it in months [17:00:33] ok, that should have forwarded the ping :-) [17:00:39] :) [17:50:01] thanks taavi! I fixed my stuff. e.ffie will probably not get to it until tomorrow earliest [17:53:42] jayme: awesome, thank you! [17:55:11] taavi: if e.ffie can't get to it tomorrow I'll take a look, I just don't want to break things even more if she was working on sth [18:06:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw, 10ops-eqiad: Update iDRAC and NIC firmware on mc-gp* hosts - https://phabricator.wikimedia.org/T329323 (10Papaul) [18:17:14] I have not touched it I am afraid [18:31:09] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: Migrate sre.switchdc.mediawiki to spicerack class API - https://phabricator.wikimedia.org/T328908 (10Krinkle) [18:31:20] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover eqiad pooling schedule - https://phabricator.wikimedia.org/T328903 (10Krinkle) [18:31:29] effie: all right, I'll handle it tomorrow then. It's the pontoon stuff. [18:31:30] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: Ensure sre.switchdc.mediawiki live test multi-DC compatibility - https://phabricator.wikimedia.org/T329065 (10Krinkle) [18:31:41] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover Excluded services - https://phabricator.wikimedia.org/T329193 (10Krinkle) [18:31:58] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: Post March 2023 Datacenter Switchover Tasks - https://phabricator.wikimedia.org/T328907 (10Krinkle) [18:32:04] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update wikikube codfw to k8s 1.23 - https://phabricator.wikimedia.org/T329664 (10JMeybohm) [18:32:08] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover Blockers - https://phabricator.wikimedia.org/T328770 (10Krinkle) [18:32:19] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 2 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10Krinkle) [18:34:15] 10serviceops, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10JMeybohm) [19:38:56] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw, 10ops-eqiad: Update iDRAC and NIC firmware on mc-gp* hosts - https://phabricator.wikimedia.org/T329323 (10Papaul) [19:39:13] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10Papaul) [19:39:15] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw, 10ops-eqiad: Update iDRAC and NIC firmware on mc-gp* hosts - https://phabricator.wikimedia.org/T329323 (10Papaul) 05Open→03Resolved a:03Papaul @jijiki complete [19:55:00] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: Toolhub does not have a working Kubernetes deployment outside of eqiad - https://phabricator.wikimedia.org/T329319 (10Gehel) >>! In T329319#8602419, @bd808 wrote: > @Gehel is there planned maintenance for the search-chi-eqiad cluster dur... [20:17:18] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: Toolhub does not have a working Kubernetes deployment outside of eqiad - https://phabricator.wikimedia.org/T329319 (10Ladsgroup) Is there any blocker from making toolhub in eqiad use codfw db instead of eqiad db (during the switchover)?... [20:19:06] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 2 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10Ladsgroup) Option 5 sounds good too, I think we can also reuse this solution in toolhub too (T329319) [20:43:42] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: Toolhub does not have a working Kubernetes deployment outside of eqiad - https://phabricator.wikimedia.org/T329319 (10Marostegui) >>! In T329319#8615985, @Ladsgroup wrote: > Is there any blocker from making toolhub in eqiad use codfw db... [21:19:33] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster [21:21:02] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2441.codfw.wmnet with OS buster [21:27:02] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: Toolhub does not have a working Kubernetes deployment outside of eqiad - https://phabricator.wikimedia.org/T329319 (10Ladsgroup) I think there is a misunderstanding, I'm not suggesting to make eqiad RW in m5 (there won't be anything touc... [22:25:27] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster completed: - mw2440 (**PASS**) - Removed from Pupp... [22:25:39] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2441.codfw.wmnet with OS buster completed: - mw2441 (**PASS**) - Removed from Pupp... [22:26:06] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2442.codfw.wmnet with OS buster [22:26:20] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2443.codfw.wmnet with OS buster [23:09:14] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2443.codfw.wmnet with OS buster completed: - mw2443 (**PASS**) - Removed from Pupp... [23:11:31] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2442.codfw.wmnet with OS buster completed: - mw2442 (**PASS**) - Removed from Pupp... [23:37:06] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: Toolhub does not have a working Kubernetes deployment outside of eqiad - https://phabricator.wikimedia.org/T329319 (10bd808) >>! In T329319#8604689, @Marostegui wrote: > Right now this is what lives in m5: > > ` > > +-----------------... [23:38:36] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: What should happen to Toolhub during the 2023 DC switch? - https://phabricator.wikimedia.org/T329319 (10bd808) [23:59:08] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul)