[00:31:45] 10serviceops, 10MW-on-K8s, 10SRE, 10Scap, and 2 others: Scap should check errors coming from mw-on-k8s canaries during deployments - https://phabricator.wikimedia.org/T357402#9565937 (10CodeReviewBot) thcipriani merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/219 Check bare metal an... [08:45:19] 10serviceops, 10Shellbox: Code in Shellbox specific to WMF production - https://phabricator.wikimedia.org/T357949#9566576 (10Joe) @tstarling just removing the .pipelinelib directory isn't an option, if we want to keep using shellbox in production. I don't see a good solution for than other than maintaining sep... [09:04:42] 10serviceops, 10Shellbox: Code in Shellbox specific to WMF production - https://phabricator.wikimedia.org/T357949#9566627 (10Joe) >>! In T357949#9565748, @tstarling wrote: >>>! In T357949#9565402, @bd808 wrote: >> The .pipeline files are used to create container images for running in Wikimedia production or an... [10:30:20] 10serviceops, 10Content-Transform-Team-WIP, 10RESTBase, 10RESTBase Sunsetting, and 6 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365#9566873 (10CodeReviewBot) jgiannelos merged https://gitlab.wikimedia.org/repos/content-transform/nodejs-cass... [10:37:51] 10serviceops, 10DC-Ops, 10Data-Persistence, 10Infrastructure-Foundations, and 5 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9566904 (10Marostegui) What is the idea? Will codfw remain depooled for a week or two? For DBAs this would be good so we can perfo... [10:43:06] 10serviceops, 10DC-Ops, 10Data-Persistence, 10Infrastructure-Foundations, and 6 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9566921 (10Marostegui) [10:47:40] 10serviceops, 10MW-on-K8s, 10SRE, 10Scap, 10Release-Engineering-Team (Now this 🫠): Find a way to address canary releases directly - https://phabricator.wikimedia.org/T358117#9566949 (10Clement_Goubert) We've talked this over, and while doing swagger checks made sense when there were just a few canaries o... [10:50:05] 10serviceops, 10DC-Ops, 10Data-Persistence, 10Infrastructure-Foundations, and 6 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9566974 (10Marostegui) [12:47:46] 10serviceops, 10DC-Ops, 10Data-Persistence, 10Infrastructure-Foundations, and 6 others: Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9567401 (10Marostegui) I'd love if it can be a bit longer than 7 days as we can do lots of operational maintenance and save a bunc... [13:06:37] 10serviceops, 10MW-on-K8s, 10RESTBase, 10SRE: Migrate restbase from mwapi-async to mw-api-int - https://phabricator.wikimedia.org/T358213#9567469 (10Clement_Goubert) [13:07:51] 10serviceops, 10MW-on-K8s, 10RESTBase, 10SRE: Migrate restbase from mwapi-async to mw-api-int - https://phabricator.wikimedia.org/T358213#9567482 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Medium [13:08:02] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9567484 (10Clement_Goubert) [13:13:04] 10serviceops, 10MW-on-K8s, 10SRE: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9567499 (10hnowlan) [14:57:33] 10serviceops: Have internal MediaWiki to MediaWiki HTTP requests use an envoyproxy on appservers - https://phabricator.wikimedia.org/T298265#9567967 (10Clement_Goubert) Appserver clusters are starting to use the envoy listener {F42043724} [15:05:40] 10serviceops, 10DC-Ops, 10Data-Persistence, 10Infrastructure-Foundations, and 6 others: ☂️ Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9568015 (10jijiki) [15:11:09] 10serviceops, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9568026 (10akosiaris) [15:34:35] 10serviceops, 10CommRel-Specialists-Support: CommRel support for Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T358233#9568155 (10jijiki) [15:42:40] 10serviceops, 10CommRel-Specialists-Support: CommRel support for Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T358233#9568202 (10jijiki) [15:42:42] 10serviceops, 10DC-Ops, 10Data-Persistence, 10Infrastructure-Foundations, and 6 others: ☂️ Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9568203 (10jijiki) [15:43:30] 10serviceops, 10DC-Ops, 10Data-Persistence, 10Infrastructure-Foundations, and 6 others: ☂️ Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9543213 (10jijiki) [16:56:44] 10serviceops, 10MW-on-K8s, 10SRE, 10Scap, 10Release-Engineering-Team (Now this 🫠): Scap should check errors coming from mw-on-k8s canaries during deployments - https://phabricator.wikimedia.org/T357402#9568746 (10dancy) 05Open→03Resolved [16:56:52] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9568747 (10dancy) [17:39:37] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9569076 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1458.eqiad.wmnet with OS bullseye [17:39:40] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9569077 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1467.eqiad.wmnet with OS bullseye [17:41:22] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9569088 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1468.eqiad.wmnet with OS bullseye [17:41:34] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9569090 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1483.eqiad.wmnet with OS bullseye [17:41:42] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9569091 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1484.eqiad.wmnet with OS bullseye [17:41:56] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9569092 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1485.eqiad.wmnet with OS bullseye [17:42:02] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9569096 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1494.eqiad.wmnet with OS bullseye [17:52:12] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9569195 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw2384.codfw.wmnet with OS bullseye [18:03:50] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9569252 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw2384.codfw.wmnet with OS bullseye [18:12:41] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9569294 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1458.eqiad.wmnet with OS bullseye completed: - mw1458 (**PASS**) - Downt... [18:15:02] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9569314 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1483.eqiad.wmnet with OS bullseye completed: - mw1483 (**PASS**)... [18:17:13] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9569317 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1468.eqiad.wmnet with OS bullseye completed: - mw1468 (**PASS**)... [18:22:30] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9569350 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1484.eqiad.wmnet with OS bullseye completed: - mw1484 (**PASS**)... [18:24:56] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9569356 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1494.eqiad.wmnet with OS bullseye completed: - mw1494 (**WARN**)... [18:25:26] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9569357 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1467.eqiad.wmnet with OS bullseye completed: - mw1467 (**PASS**) - Downt... [18:28:58] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#9569364 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1485.eqiad.wmnet with OS bullseye completed: - mw1485 (**PASS**)... [18:31:55] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9569379 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw2385.codfw.wmnet with OS bullseye [18:45:04] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9569419 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw2384.codfw.wmnet with OS bullseye completed: - mw2384 (**WARN**) - Remov... [19:15:03] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9569546 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw2385.codfw.wmnet with OS bullseye completed: - mw2385 (**WARN**) - Remov... [19:19:07] 10serviceops, 10SRE, 10ops-codfw: Issues reimaging servers in codfw - https://phabricator.wikimedia.org/T358001#9569554 (10hnowlan) 05Open→03Resolved a:03hnowlan >>! In T358001#9563665, @Jhancock.wm wrote: > @hnowlan I've replaced the network cable on both of these. These are both connected to a 1G swi... [20:34:25] 10serviceops, 10Datacenter-Switchover: SRE comms for Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T358286#9569949 (10jijiki) [20:54:40] 10serviceops, 10MW-on-K8s, 10SRE, 10Scap, 10Release-Engineering-Team (Now this 🫠): Find a way to address canary releases directly - https://phabricator.wikimedia.org/T358117#9570020 (10thcipriani) >>! In T358117#9566949, @Clement_Goubert wrote: > We've talked this over, and while doing swagger checks mad... [21:16:11] 10serviceops, 10Datacenter-Switchover: SRE comms for Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T358286#9570113 (10jijiki)