[08:33:46] 10Wikimedia-Apache-configuration, 10Security-Team, 10WMF-General-or-Unknown, 10Documentation, and 2 others: Add security.txt to Wikimedia sites? (2023 edition) - https://phabricator.wikimedia.org/T337949 (10LSobanski) Removing #SRE, please add us back when the decision to implement this has been made. [14:08:06] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10Jhancock.wm) This rack is physically ready for tomorrow. [15:47:39] guys cp2033 and cp2034 are in codfw rack B4 we're hoping to move hosts to newer switch in today [15:47:55] topranks: hello [15:48:06] we were just talking about those, we will depool and downtime [15:48:16] sukhe: great thanks! [15:48:29] no rush, let me know whenever you're ready [15:51:32] topranks: depooled, downtimed, all yours :) [15:52:12] sukhe: super thank you ! [15:58:49] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 3 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=1cb41722-6e24-4871-a903-cd... [15:59:19] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 3 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b2349fc0-73a1-418a-b3b8-28... [16:01:38] 10Wikimedia-Apache-configuration, 10Security-Team, 10WMF-General-or-Unknown, 10Documentation, 10Security: Add security.txt to Wikimedia sites? (2023 edition) - https://phabricator.wikimedia.org/T337949 (10sbassett) p:05Medium→03Triage [16:01:44] 10Wikimedia-Apache-configuration, 10Security-Team, 10WMF-General-or-Unknown, 10Documentation, 10Security: Add security.txt to Wikimedia sites? (2023 edition) - https://phabricator.wikimedia.org/T337949 (10sbassett) [16:03:22] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 3 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a3c16d29-3284-4390-9f38-03... [16:13:16] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 3 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10cmooney) All hosts moved successfully, all now responding to pings fine and MAC forwarding... [16:14:45] sukhe: hey we're done in codfw B4 if you want to repool those cp hosts [16:15:06] topranks: thanks, checking and repooling [16:15:13] cheers [16:15:21] nicely done, all involved! [16:16:21] Jenn is definitely up there with sensei pa.paul in terms of slick execution! [16:56:57] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 3 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10MatthewVernon) swift backends look happy, thanks :) [17:47:23] 10Traffic, 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Problem downloading large files from analytics.wikimedia.org - https://phabricator.wikimedia.org/T356792 (10BTullis) I have a feeling that this is more related to the CDN than to the webserver behind it. I tried the following test from stat1004, which... [17:48:10] 10Traffic, 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Problem downloading large files from analytics.wikimedia.org - https://phabricator.wikimedia.org/T356792 (10BTullis) I'm going to add the #traffic team to see if they have any insight as to why this might have started happening recently. [17:57:34] 10Traffic, 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Problem downloading large files from analytics.wikimedia.org - https://phabricator.wikimedia.org/T356792 (10Vgutierrez) I can reproduce via text@drmrs, I'll take a look ASAP :) [18:02:22] 10Traffic, 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Problem downloading large files from analytics.wikimedia.org - https://phabricator.wikimedia.org/T356792 (10Vgutierrez) @BTullis it's origin related: ` vgutierrez@bast6003:~$ curl -o /dev/null --resolve analytics.wikimedia.org:8443:10.64.21.14 --limit-... [18:10:54] 10Traffic, 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Problem downloading large files from analytics.wikimedia.org - https://phabricator.wikimedia.org/T356792 (10BTullis) @Vgutierrez - Thanks so much. Let me know if I can help with anything. [18:12:46] btullis: origin aka analytics issue :) [18:19:03] vgutierrez: oh, sorry. I understand now. [18:22:06] You could be hitting https://github.com/wikimedia/operations-puppet/blob/1a6c9d13ee7a499ee7a28e47449774a6a6dcdccc/modules/envoyproxy/manifests/tls_terminator.pp#L147 [18:24:00] 10Traffic, 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Problem downloading large files from analytics.wikimedia.org - https://phabricator.wikimedia.org/T356792 (10Vgutierrez) https://github.com/wikimedia/operations-puppet/blob/1a6c9d13ee7a499ee7a28e47449774a6a6dcdccc/modules/envoyproxy/manifests/tls_termina... [18:25:05] Oh, thanks. That's really useful. I'll try that. [21:09:01] 10Traffic, 10SRE: Lower geodns TTLs from 600 (10min) to 300 (5min) - https://phabricator.wikimedia.org/T140365 (10cmooney) Seems reasonable. There are some good reasons not to go too far (reducing load both our side and for recursive servers on the internet), but 5 mins seems ok to me. [21:22:19] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 3 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10cmooney) 05Open→03Resolved a:03cmooney Closing task, all looks good following change.... [21:22:29] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [21:35:55] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE, 10Patch-For-Review: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10jhathaway) a:03jhathaway [21:39:28] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE, 10Patch-For-Review: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10jhathaway) 05Open→03Resolved @bcampbell I assume this is resolved, please go... [21:59:31] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE, 10Patch-For-Review: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10bcampbell) @jhathaway Sorry for not closing the loop on this one. It is resolved...