[07:12:09] (LVSHighRX) firing: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [07:17:09] (LVSHighRX) resolved: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [09:55:45] 06Traffic, 06DC-Ops, 10SRE-swift-storage, 10ops-codfw, 10ops-eqiad: Reimage cookbook on new eqiad hosts stuck at PXE booting - https://phabricator.wikimedia.org/T350179#9586432 (10ayounsi) One possible path forward is to work with Dell's support to solve {T304483} Another one is to narrow down the chara... [10:59:49] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 2 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752 (10akosiaris) [12:10:20] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871#9586779 (10cmooney) 05Open→03Resolved a:03cmooney Closing task - thanks all for the co-operation! [12:10:30] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544#9586782 (10cmooney) [12:14:41] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872#9586789 (10ops-monitoring-bot) Draining ganeti2032.codfw.wmnet of running VMs [12:57:34] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Decom asw-a-codfw switch stack - https://phabricator.wikimedia.org/T358244#9586848 (10cmooney) After a quick discussion on irc I think we can't wipe the config for every unit in the VC over ssh to the master. So probably easiest to do that via se... [14:37:31] 06Traffic, 06DC-Ops, 06Data-Persistence, 06collaboration-services, and 4 others: ☂️ Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9587221 (10Trizek-WMF) [14:50:07] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Decom asw-a-codfw switch stack - https://phabricator.wikimedia.org/T358244#9587275 (10Jhancock.wm) @cmooney they're on the old asw switches. Let me know when you want to move them back to the new lsw. [16:01:50] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872#9587603 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ab1a9b14-3187-4d52-a4f6-be3c445a8081... [16:02:44] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872#9587615 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=12cd3c2a-9d8e-4ba6-a42e-1faa167de80d... [16:10:47] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872#9587654 (10cmooney) All hosts moved sucessfully. Showing up on switch, macs learnt and all responding to ping a... [16:20:50] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872#9587682 (10MatthewVernon) thanos and ms swift clusters OK post-move, thank you! [17:37:45] 06Traffic, 06DC-Ops, 10SRE-swift-storage, 10ops-codfw, 10ops-eqiad: Reimage cookbook on new eqiad hosts stuck at PXE booting - https://phabricator.wikimedia.org/T350179#9588119 (10ssingh) Thanks for sharing @ayounsi! [17:55:07] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588268 (10cmooney) Ok I abandoned my previous change as my git skills weren't up to it. Prepping a new patc... [18:13:23] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588323 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=2521164e-4d59-47cb-8d79-8dd925725... [18:40:44] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588429 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=1baa9b0f-d917-4da6-83db-5cb28b50c... [18:56:34] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588508 (10cmooney) [18:57:33] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588511 (10cmooney) [19:07:11] 06Traffic, 13Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054#9588550 (10ssingh) Update: We have merged the service depooling change on `dns6001`. This means that service depooling -- recdns, ntp, authdns... [19:10:05] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588555 (10cmooney) [19:14:45] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588573 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host... [19:22:45] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588615 (10cmooney) [20:09:02] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920#9588735 (10cmooney) [20:31:19] 10Wikimedia-Apache-configuration, 06serviceops, 13Patch-For-Review: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595#9588786 (10RLazarus) a:03RLazarus [20:47:34] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588904 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host lvs2012.codfw.wmnet with... [21:00:25] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588964 (10cmooney) 05Open→03Resolved Moved to new vlan and BGP established between server and switch now. ` cmooney@lvs2012:/e... [21:00:33] 10netops, 06Infrastructure-Foundations, 06SRE: Re-IP hosts on codfw row A and B to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869#9588966 (10cmooney)