[06:01:01] 10netops, 10Infrastructure-Foundations: db2137 doesn't get an IP via PXE boot - https://phabricator.wikimedia.org/T357951#9556745 (10Marostegui) [06:01:35] 10netops, 10Infrastructure-Foundations: db2137 doesn't get an IP via PXE boot - https://phabricator.wikimedia.org/T357951#9556756 (10Marostegui) The host was being reimaged into bookworm. Please feel free to start the reimage yourself anytime. [06:23:08] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870#9556783 (10Marostegui) [06:25:11] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870#9556787 (10Marostegui) es2021 is no longer a master and it just need normal depooling cc @ABran-WMF [06:33:00] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866#9556801 (10Marostegui) @cmooney is there anything pending here or can this be closed? [07:37:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867#9557370 (10ops-monitoring-bot) Draining ganeti2028.codfw.wmnet of running VMs [12:01:52] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, and 2 others: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9558616 (10aborrero) [12:49:23] fabfur: hey sorry to bug you, we have to make an emergency IP change on lvs2014 unfortunately [12:50:01] any cange you could review this patch? [12:50:01] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1005061 [13:00:00] hey sorry was at lunch, check it now [13:00:59] honestly never did such changes [13:01:26] if it's something you already did in the past and there's no side effect I think we're ok [13:01:40] anything to change on the DNS/Pybal side? [13:06:33] fabfur: we will do it in an hour or so, talked to topranks [13:06:45] ah ok, thanks [13:06:58] sukhe: thanks! [13:07:37] fabfur: fwiw no nothing to change on the pybal side, the back-end vlans it just needs to be able to arp [13:07:51] ack! [13:07:56] and we don't have dns in place for those IPs either so it's a fiarly straightforward change [13:08:41] I disabled puppet on lvs2014 for the time being fyi. [14:18:33] 10Traffic, 10Automoderator, 10Data Products, 10Product-Analytics, and 3 others: Add revision ID to X-Analytics header - https://phabricator.wikimedia.org/T346350#9559176 (10Samwalton9-WMF) Great! I think we can mark this as resolved then? [14:21:55] topranks: looking now, sorry [14:31:29] topranks: looks good! we are here if you want us to roll it out :) [14:31:47] sukhe: thanks! no I think it's all good I can take care of it [14:32:05] sorry for the mix-up! [14:32:15] np, sorry for missing it during the review :) [14:51:31] topranks: cp2029 and cp2030 are ready for maintenance. Thanks for your work! [14:53:21] 10netops, 10Infrastructure-Foundations, 10SRE: BGP peering from LSW to K8s hosts using loopback IP not IRB - https://phabricator.wikimedia.org/T357619#9559391 (10cmooney) [14:54:12] 10netops, 10Infrastructure-Foundations, 10SRE: Update K8S BGP groups eqiad row e-f - https://phabricator.wikimedia.org/T357924#9559389 (10cmooney) 05Open→03Resolved All done, used statics to avoid any disruption to forwarding. [14:54:45] brett: hey thanks for the confirmation appreciate it :) [15:30:26] 10Traffic, 10Automoderator, 10Data Products, 10Product-Analytics, and 3 others: Add revision ID to X-Analytics header - https://phabricator.wikimedia.org/T346350#9559622 (10jsn.sherman) 05Open→03Resolved >>! In T346350#9559176, @Samwalton9-WMF wrote: > Great! I think we can mark this as resolved then?... [15:31:05] 10netops, 10Infrastructure-Foundations, 10SRE: FPC1 Failure on cr1-esams - https://phabricator.wikimedia.org/T351304#9559649 (10RobH) [15:51:50] 10netops, 10Infrastructure-Foundations, 10SRE: BGP peering from LSW to K8s hosts using loopback IP not IRB - https://phabricator.wikimedia.org/T357619#9559790 (10cmooney) 05Open→03Resolved Config pushed out across the estate now, multihop config only added on CRs. [15:58:53] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Decommission task for old cp hosts (cp1075-1090) - https://phabricator.wikimedia.org/T352253#9559872 (10ssingh) Hi dc-ops team: quick question: have these hosts already been hardware decommissioned? [16:02:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544#9559899 (10cmooney) [16:02:48] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867#9559902 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=343ed6db-68dd-4330-8851-9631da7da8d5... [16:02:54] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Decommission task for old cp hosts (cp1075-1090) - https://phabricator.wikimedia.org/T352253#9559903 (10ssingh) For further context: we have a request from @dr0ptp4kt for running a Blazegraph experiment and we are trying to free up a cp node for him. So we were wond... [16:03:12] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866#9559896 (10cmooney) 05Open→03Resolved a:03cmooney >>! In T355866#9556801, @Marostegui wrote: > @cmooney is there anythin... [16:06:29] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867#9559917 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=47f3a57d-6476-4782-ba82-9c2dc99042c9... [16:17:05] 10Traffic, 10SRE-swift-storage, 10Patch-For-Review: OpenSSL 3.x performance issues - https://phabricator.wikimedia.org/T352744#9560023 (10ssingh) Thanks to @Muehlenhoff, we have imported the forward port of OpenSSL 1.1.1 and have build haproxy 2.6 against it. We will be reimaging a cp host to bookworm. Shar... [16:17:10] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867#9560025 (10cmooney) All links moved successfully and all hosts responding to ping as before. [16:18:50] brett: those links are moved and cp hosts back pinging if you want to repool them [16:19:04] 10Traffic, 10SRE-swift-storage, 10Patch-For-Review: OpenSSL 3.x performance issues - https://phabricator.wikimedia.org/T352744#9560050 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp4052.ulsfo.wmnet with OS bookworm [16:20:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867#9560061 (10MatthewVernon) ms and thanos swift both OK post-move. [16:27:51] 10Traffic, 10SRE-swift-storage, 10Patch-For-Review: OpenSSL 3.x performance issues - https://phabricator.wikimedia.org/T352744#9560148 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp4052.ulsfo.wmnet with OS bookworm executed with errors: - cp4052 (**FAIL**)... [16:30:25] 10Traffic, 10SRE-swift-storage, 10Patch-For-Review: OpenSSL 3.x performance issues - https://phabricator.wikimedia.org/T352744#9560162 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp4052.ulsfo.wmnet with OS bookworm [16:39:26] Thank you! [16:59:34] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9560354 (10akosiaris) [17:18:26] 10Traffic, 10SRE-swift-storage, 10Patch-For-Review: OpenSSL 3.x performance issues - https://phabricator.wikimedia.org/T352744#9560423 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp4052.ulsfo.wmnet with OS bookworm completed: - cp4052 (**PASS**) - Removed... [18:14:43] 10Traffic, 10Security-Team, 10ContentSecurityPolicy, 10SecTeam-Processed: Stop sending X-Webkit-CSP and X-Webkit-CSP-Report-Only headers - https://phabricator.wikimedia.org/T357479#9540376 (10sbassett) [18:15:05] 10Traffic, 10Security-Team, 10ContentSecurityPolicy, 10SecTeam-Processed: Stop sending X-Webkit-CSP and X-Webkit-CSP-Report-Only headers - https://phabricator.wikimedia.org/T357479#9560711 (10sbassett) 05Open→03Resolved a:03sbassett [18:15:08] 10Traffic, 10Security-Team, 10ContentSecurityPolicy, 10SecTeam-Processed: Stop sending X-Webkit-CSP and X-Webkit-CSP-Report-Only headers - https://phabricator.wikimedia.org/T357479#9540376 (10sbassett) p:05Triage→03Medium