[08:25:45] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9883340 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-ctrl1002.eq... [09:06:24] 06serviceops, 06Machine-Learning-Team, 13Patch-For-Review: Rename the envoy's uses_ingress option to sets_sni - https://phabricator.wikimedia.org/T346638#9883418 (10JMeybohm) 05Open→03Resolved [09:07:07] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9883434 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-ctrl1002.eqiad.... [09:36:27] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 9 others: Upgrade push notifications to node18 - https://phabricator.wikimedia.org/T367272 (10Jgiannelos) 03NEW [09:37:05] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 9 others: Upgrade push notifications to node18 - https://phabricator.wikimedia.org/T367272#9883510 (10Jgiannelos) a:03Jgiannelos [09:49:13] 06serviceops, 06Content-Transform-Team-WIP, 10Push-Notification-Service, 07Essential-Work: Upgrade push notifications to node18 - https://phabricator.wikimedia.org/T367272#9883533 (10MSantos) [09:59:31] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Moving 1G servers out of rack D4 in prep of switch migration - https://phabricator.wikimedia.org/T361856#9883573 (10Clement_Goubert) So sorry I didn't answer earlier. Apart from `mw2282` which has been migrated to k8s, we will decom these hosts so you don't have... [10:04:02] 06serviceops, 10decommission-hardware: decommission mw2281.codfw mw22[83-90].codfw.wmnet - https://phabricator.wikimedia.org/T367275 (10Clement_Goubert) 03NEW [10:08:44] 06serviceops, 10decommission-hardware: decommission mw2281.codfw mw22[83-90].codfw.wmnet - https://phabricator.wikimedia.org/T367275#9883597 (10Clement_Goubert) a:03Clement_Goubert [10:10:39] 06serviceops, 10decommission-hardware: decommission mw2281.codfw.wmnet mw22[83-90].codfw.wmnet - https://phabricator.wikimedia.org/T367275#9883598 (10Clement_Goubert) [10:14:31] 06serviceops, 10Wikimedia-Apache-configuration: 2030.wikimedia.org is a double redirect - https://phabricator.wikimedia.org/T367013#9883619 (10akosiaris) 05Open→03Declined Given the above, I am gonna close this as `Declined` in the interest of not having lingering tasks, but feel free to reopen. [10:15:49] 06serviceops, 10decommission-hardware: decommission mw2281.codfw.wmnet mw22[83-90].codfw.wmnet - https://phabricator.wikimedia.org/T367275#9883625 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=f19f15a1-4b9d-48b8-b081-d514fc9fab58) set by cgoubert@cumin1002 for 14 days, 0:00:00 on 9 host(s... [10:15:59] 06serviceops, 10decommission-hardware: decommission mw2281.codfw.wmnet mw22[83-90].codfw.wmnet - https://phabricator.wikimedia.org/T367275#9883626 (10Clement_Goubert) [10:16:02] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Moving 1G servers out of rack D4 in prep of switch migration - https://phabricator.wikimedia.org/T361856#9883627 (10Clement_Goubert) [10:22:26] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw2281.codfw.wmnet mw22[83-90].codfw.wmnet - https://phabricator.wikimedia.org/T367275#9883635 (10Clement_Goubert) [10:40:30] 06serviceops, 10MW-on-K8s, 10Observability-Logging: Some apache access logs are invalid json - https://phabricator.wikimedia.org/T340935#9883677 (10kamila) 05Resolved→03Open I'm seeing more bad messages that look like an encoding problem: ` { "host": { "name": "parse2020.codfw.wmnet" }, "orche... [10:41:10] ^ I am very sorry (and this does not count for owing anyone a beer :D) [10:51:44] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, 07Kubernetes: Relabel eqiad wikikube worker nodes - https://phabricator.wikimedia.org/T367285 (10Clement_Goubert) 03NEW [10:54:53] ugh encodings [10:55:05] yep! [11:03:37] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9883835 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by kamila@cumin1002 for hosts: `wikikube-ctrl1003.eqiad.... [11:23:31] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, 07Kubernetes: Relabel codfw wikikube worker nodes - https://phabricator.wikimedia.org/T367286 (10Clement_Goubert) 03NEW [11:52:24] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9883984 (10Clement_Goubert) 05In progress→03Resolved a:03Clement_Goubert [13:21:22] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate charts to Calico Network Policies - https://phabricator.wikimedia.org/T359423#9884261 (10JMeybohm) [14:17:22] 06serviceops, 10Prod-Kubernetes, 13Patch-For-Review: Update all helm modules and charts to be compatible with the restricted PSS - https://phabricator.wikimedia.org/T362978#9884588 (10JMeybohm) [15:12:16] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, and 2 others: eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9885068 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-ctrl1003.eqiad.wmnet with OS bullseye exe... [15:18:57] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#9885107 (10JMeybohm) p:05Triage→03High [15:42:23] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9885253 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-ctrl1003.eq... [15:53:29] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9885312 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-ctrl1003.eqiad.... [16:00:28] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9885334 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-ctrl1003.eq... [16:08:01] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Moving 1G servers out of rack D4 in prep of switch migration - https://phabricator.wikimedia.org/T361856#9885353 (10Papaul) @Clement_Goubert next week monday 17th at 10:00am CT [16:19:23] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE-OnFire, 10Sustainability (Incident Followup): eqiad:(3) wikikube-ctrl NIC upgrade to 10G - https://phabricator.wikimedia.org/T366204#9885446 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-ctrl1003.eqiad.... [16:23:46] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Moving 1G servers out of rack D4 in prep of switch migration - https://phabricator.wikimedia.org/T361856#9885472 (10Clement_Goubert) All right, I'll do the draining monday beginning of the UTC afternoon so it's all set for you. [16:54:59] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw wikikube worker nodes - https://phabricator.wikimedia.org/T367286#9885733 (10VRiley-WMF) [19:08:35] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: Relabel eqiad wikikube worker nodes - https://phabricator.wikimedia.org/T367285#9886282 (10VRiley-WMF) [19:10:23] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: Relabel eqiad wikikube worker nodes - https://phabricator.wikimedia.org/T367285#9886284 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF These have been tagged as requested. This is resolved.