[02:02:24] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Papaul) @Cmjohnson what i am seeing in the partman recipe that the server is using ,line 10 is removing any existing LVM ` 10 d-i... [06:56:56] (HAProxyEdgeTrafficDrop) firing: (2) 64% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [07:01:56] (HAProxyEdgeTrafficDrop) resolved: (2) 66% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [12:51:10] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: decommission atlas-esams - https://phabricator.wikimedia.org/T307026 (10ayounsi) I believe the anchors are linked to Faidon's RIPE account. @faidon, could you take care of it? [14:12:20] 10Traffic, 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) I am now investigating by capturing network traffic from the eventgate-analytics-external pods and looking... [14:17:38] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: decommission atlas-esams - https://phabricator.wikimedia.org/T307026 (10cmooney) From the experience with the one in codfw I think the process is to delete and then re-add. If @faidon can remove our existing one we can take care of the... [14:30:40] 10netops, 10Infrastructure-Foundations, 10ops-drmrs, 10ops-esams: drmrs-esams wave provisioning - https://phabricator.wikimedia.org/T307221 (10ayounsi) [14:32:29] 10netops, 10Infrastructure-Foundations, 10ops-drmrs, 10ops-esams: drmrs-esams wave provisioning - https://phabricator.wikimedia.org/T307221 (10ayounsi) [15:19:17] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ayounsi) a:05ayounsi→03None [17:08:22] 10Traffic, 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) Well, this is a bit confusing. I've examined packet captures from two pods in eqiad and another in codfw.... [17:35:48] 10Traffic, 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) I have a few errors logged by ats-be attempting to connect to `eventgate-analytics-external.discovery.wmne... [17:54:14] 10Traffic, 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10Ottomata) > perhaps this is a client browser opening a connection but sending an empty POST body This seems likely,... [20:19:57] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs, 10ops-esams: drmrs-esams wave provisioning - https://phabricator.wikimedia.org/T307221 (10wiki_willy) @RobH - here are the LOAs in pdf format below: {F35074530} {F35074529} Thanks, Willy