[00:44:40] 10netops, 10Infrastructure-Foundations, 10SRE: Eqiad Expansion - LVS Connectivity Options - https://phabricator.wikimedia.org/T292630 (10Jclark-ctr) [06:45:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:50:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [12:15:57] 10HTTPS, 10SRE, 10Traffic-Icebox, 10Upstream: Support ECH on Wikimedia servers - https://phabricator.wikimedia.org/T205378 (10Tranve) [12:16:31] 10HTTPS, 10SRE, 10Traffic-Icebox, 10Upstream: Support ECH on Wikimedia servers - https://phabricator.wikimedia.org/T205378 (10Tranve) ESNI has been superseded by the ECH, hence updating the task. [13:03:29] 10HTTPS, 10SRE, 10Traffic-Icebox, 10Upstream: Support ECH on Wikimedia servers - https://phabricator.wikimedia.org/T205378 (10jcrespo) unsubing, as I think I was added to this ticket by mistake. This is traffic/traffic security expertise, and they already triaged and aware of the task. [13:33:26] 10netops, 10Infrastructure-Foundations: Deprecate interface-range external - https://phabricator.wikimedia.org/T296935 (10ayounsi) p:05Triage→03Medium [14:45:34] 10Domains, 10HTTPS, 10DNS, 10SRE, 10Traffic-Icebox: Merge Wikipedia subdomains into one, to discourage censorship - https://phabricator.wikimedia.org/T215071 (10Tranve) 05Open→03Declined [14:47:16] 10Domains, 10HTTPS, 10DNS, 10SRE, 10Traffic-Icebox: Merge Wikipedia subdomains into one, to discourage censorship - https://phabricator.wikimedia.org/T215071 (10Tranve) Closing this task due to the fact that Wikipedia has been blocked in all languages since Mar 2019, which leaves this task meaningless. [14:52:06] 10Domains, 10HTTPS, 10DNS, 10SRE, 10Traffic-Icebox: Merge Wikipedia subdomains into one, to discourage censorship - https://phabricator.wikimedia.org/T215071 (10ssingh) 05Declined→03Open Please don't close this task pending further discussion. Thank you. [18:05:20] 10Traffic, 10SRE: HAProxy fails to reuse connections under some conditions - https://phabricator.wikimedia.org/T296874 (10Vgutierrez) As pointed out by Willy Tarreau on https://github.com/haproxy/haproxy/issues/1472#issuecomment-984745445 it's a matter of FD usage and increasing maxconn seems to solve it: {F34... [18:22:37] 10Traffic, 10SRE, 10Patch-For-Review: HAProxy fails to reuse connections under some conditions - https://phabricator.wikimedia.org/T296874 (10Vgutierrez) 05Open→03Resolved p:05Triage→03Medium a:03Vgutierrez [18:22:40] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez) [19:13:59] I'm currently helping with https://phabricator.wikimedia.org/T223053 and would like a bit of clarity on how cookies affect frontend caching for MediaWiki. I see that MW emits a `Vary: Cookie` header, does that mean the caches are already split based on the values of cookies? [21:48:38] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10wiki_willy) [22:28:47] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Dzahn) one case of mw2252.mgmt right now [22:33:32] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @AntiCompositeNumber Not for sure but it seems likely that it could also b...