[09:05:43] hello folks [09:06:28] while reviewing traffic that reads from kafka main I noticed that purged uses the Puppet CA bundle to validate TLS certs, created https://gerrit.wikimedia.org/r/c/operations/puppet/+/901118 [09:06:47] should be an easy one, but lemme know if I am missing something [10:12:08] 10Traffic, 10Sustainability (Incident Followup): cp3050 seemd more affected then otheres in recent incident - https://phabricator.wikimedia.org/T330682 (10Vgutierrez) @CDanis I'm guessing this is related to the base load esams handles + the extra debug logging added in ` profile::cache::haproxy::vars: - dir... [10:23:44] 10netops, 10Infrastructure-Foundations, 10SRE-Sprint-Week-Sustainability-March2023, 10Sustainability (Incident Followup): Cr1-eqiad comms problem when moving to 40G row D handoff - https://phabricator.wikimedia.org/T320566 (10Volans) [10:26:54] 10Traffic, 10SRE-Sprint-Week-Sustainability-March2023, 10Patch-For-Review, 10Sustainability (Incident Followup): Rate limiting for hotlinked images - https://phabricator.wikimedia.org/T317799 (10Volans) Removing I/F as all the proposed solutions falls into the Traffic realm. [10:35:12] 10netops, 10Infrastructure-Foundations, 10SRE-Sprint-Week-Sustainability-March2023, 10Sustainability (Incident Followup): Optimise WMF WAN Network Configuration - https://phabricator.wikimedia.org/T297355 (10Volans) [10:42:29] 10netops, 10Infrastructure-Foundations, 10SRE-Sprint-Week-Sustainability-March2023, 10ops-eqiad, 10Sustainability (Incident Followup): eqiad: upgrade row C and D uplinks from 4x10G to 1x40G - https://phabricator.wikimedia.org/T313463 (10Volans) [10:46:04] 10Traffic, 10SRE-Sprint-Week-Sustainability-March2023, 10Sustainability (Incident Followup): cp3050 seemd more affected then otheres in recent incident - https://phabricator.wikimedia.org/T330682 (10Vgutierrez) [10:51:25] 10Traffic, 10SRE-Sprint-Week-Sustainability-March2023, 10Sustainability (Incident Followup): cp3050 seemd more affected then otheres in recent incident - https://phabricator.wikimedia.org/T330682 (10Vgutierrez) [11:23:02] 10Traffic, 10SRE-Sprint-Week-Sustainability-March2023, 10conftool, 10Patch-For-Review, 10Sustainability (Incident Followup): requestctl can't act on cache hits - https://phabricator.wikimedia.org/T317794 (10Vgutierrez) [11:26:24] 10Traffic, 10SRE-Sprint-Week-Sustainability-March2023, 10conftool, 10Patch-For-Review, 10Sustainability (Incident Followup): requestctl can't act on cache hits - https://phabricator.wikimedia.org/T317794 (10Vgutierrez) [11:50:58] 10Traffic, 10DNS, 10SRE-Sprint-Week-Sustainability-March2023, 10Sustainability (Incident Followup): Automate DNS depools such that manual commits are not required - https://phabricator.wikimedia.org/T303219 (10Vgutierrez) [11:54:11] 10Traffic, 10SRE-Sprint-Week-Sustainability-March2023, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106 (10Vgutierrez) [12:50:07] Hi alL! Do we have stats on how many requests we see by logged in users, compared to the number of unauthenticated users? [13:49:57] duesen: yes per https://wikitech.wikimedia.org/wiki/X-Analytics [13:50:12] the Analytics team should be able to give you the actual data [14:15:37] Y'all doing something to lvs1018? [14:15:44] claime: yeah, me [14:15:47] downtimed. something is up? [14:16:16] sukhe: I just saw the alerts in -operations a few minutes late [14:16:20] :) [14:16:25] claime: sorry, I got a bit late in downtiming [14:16:30] No worries [14:16:30] but yes, all's good, thanks for checking [14:16:41] I didn't get paged so we're all good, I just wanted to make sure [14:16:50] yeah [14:17:02] I was deep into writing documentation >_> [14:17:17] you mean sprinting documentation :P [14:17:27] Yeah :D [15:37:34] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10Papaul) @cmooney We can move any servers racked from U11 up [16:23:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers to make space for new switches in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10Papaul) [18:54:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10cmooney) >>! In T327919#8710573, @Papaul wrote: > @cmooney We can move any servers racked from U11 up... [20:34:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10Papaul) @cmooney Please see first batch proposal. We can move all those servers next week. @aborrero ca...