[08:28:56] (EdgeTrafficDrop) firing: (2) 63% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [08:33:56] (EdgeTrafficDrop) resolved: (2) 63% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [08:44:11] (EdgeTrafficDrop) firing: (2) 63% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [08:45:26] (EdgeTrafficDrop) resolved: (2) 63% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [11:36:16] godog: checking data per request that can be logged on HAProxy, the session state at disconnection seems pretty useful to get some counters set based: http://cbonte.github.io/haproxy-dconv/2.4/configuration.html#8.5 [11:37:41] current idea is to fragment based on state at disconnection + status code + http method and http version [16:24:46] 10netops, 10Infrastructure-Foundations: Agree how to handle port-block speeds for QFX5120-48Y - https://phabricator.wikimedia.org/T303529 (10cmooney) p:05Triage→03Medium [16:56:42] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Jclark-ctr) @nskaggs Would we be able to Rack these in New Wmcs Dedicated racks? E4 , F4? [16:56:57] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Jclark-ctr) [17:10:45] 10Traffic, 10SRE, 10envoy, 10serviceops, 10Sustainability (Incident Followup): Raw "upstream connect error or disconnect/reset before headers. reset reason: overflow" error message shown to users during outage - https://phabricator.wikimedia.org/T287983 (10RLazarus) [17:10:53] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10RobH) Looks like I filed both T302981 & T299610, and T299610 has less recent details, and wasnt linked into T286588, so declining T... [18:21:01] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10nskaggs) @Jclark-ctr I would want confirmation from infa foundations that all the necessary network connectivity is present. From w... [19:11:31] 10Traffic, 10Data-Engineering, 10Event-Platform, 10SRE, and 2 others: Banner sampling leading to a relatively wide site outage (mostly esams) - https://phabricator.wikimedia.org/T303036 (10JMeybohm) [20:40:01] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: cp1085 memory errors on DIMM A5 - https://phabricator.wikimedia.org/T303183 (10wiki_willy) a:03Cmjohnson