[01:23:48] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host dns4004.wikimedia.org with OS buster
[08:48:33] <wikibugs>	 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10Vgutierrez) It looks more like a generic response to delegate a whole domain to shopify than what we are currently doing with store.wikimedia.org. As @Dzahn mentioned, we got store.wi...
[09:19:45] <wikibugs>	 10Traffic, 10SRE: Upgrade HAProxy on cp nodes to 2.6.x LTS - https://phabricator.wikimedia.org/T321775 (10Vgutierrez) 05In progress→03Resolved We are now running 2.6.8-2~bpo10+1 globally
[09:43:16] <wikibugs>	 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 10 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10MoritzMuehlenhoff)
[12:15:45] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10User-jbond: Sporadic RST drops in the ulogd logs - https://phabricator.wikimedia.org/T238823 (10Ladsgroup) FWIW, mw should not send this many cross-dc connections to databases but I assume it's a different aspect of this problem.
[12:24:10] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host dns4004.wikimedia.org with OS buster executed with errors: - dns4004 (**FAIL**)   - Downtimed on...
[14:55:26] <wikibugs>	 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10SHust) @Vgutierrez, thanks for your share. May I bug you for a comprehensive summary if possible, of what is needed, what I should put more pressure on, what to say no, etc, so I can...
[14:58:42] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Agree strategy for Kubernetes BGP peering to top-of-rack switches - https://phabricator.wikimedia.org/T306649 (10ayounsi) Circling back on the network side config now that there are a few patches out to improve the server side....
[15:09:19] <wikibugs>	 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10Vgutierrez) >>! In T128559#8618724, @SHust wrote: > @Vgutierrez, thanks for your share. May I bug you for a comprehensive summary if possible, of what is needed, what I should put mor...
[17:31:18] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host dns4004.wikimedia.org with OS buster
[18:24:44] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host dns4004.wikimedia.org with OS buster completed: - dns4004 (**PASS**)   - Downtimed on Icinga/Aler...
[21:58:21] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Add network-layer protections to avoid inadvertently lowering IRB MTU - https://phabricator.wikimedia.org/T329799 (10cmooney) p:05Triage→03Medium
[21:58:35] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall)
[21:59:35] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review, 10Upstream: Review cp2041 and cp2042 running bullseye - https://phabricator.wikimedia.org/T325557 (10BCornwall) 05Open→03Resolved Looks like this can be resolved!
[21:59:44] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Add network-layer protections to avoid inadvertently lowering IRB MTU - https://phabricator.wikimedia.org/T329799 (10cmooney)
[22:01:41] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Add network-layer protections to avoid inadvertently lowering IRB MTU - https://phabricator.wikimedia.org/T329799 (10cmooney) The above patch addresses the issue by ensuring Homer adds an MTU of 9192 on any L2 switch ports which don't have...