[06:38:56] (HAProxyEdgeTrafficDrop) firing: 67% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [06:43:56] (HAProxyEdgeTrafficDrop) resolved: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:11:53] 10Traffic, 10SRE: Text cluster is being hit with an average of 1.8k PURGE requests per second per host - https://phabricator.wikimedia.org/T318349 (10Vgutierrez) [08:12:56] (HAProxyEdgeTrafficDrop) firing: (2) 60% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:14:01] 10Traffic, 10Performance-Team, 10RESTBase-API, 10SRE: Text cluster is being hit with an average of 1.8k PURGE requests per second per host - https://phabricator.wikimedia.org/T318349 (10Ladsgroup) This seems to be mostly rest base (FYI perf) [08:17:56] (HAProxyEdgeTrafficDrop) resolved: (2) 64% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:20:20] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10Vgutierrez) After double checking on netbox it looks like we have some standing issues: lvs1017 and lvs1020 get connectivity to rows B and C... [09:29:15] 10Traffic, 10Performance-Team, 10RESTBase-API, 10SRE: Text cluster is being hit with an average of 1.8k PURGE requests per second per host - https://phabricator.wikimedia.org/T318349 (10Vgutierrez) The schema https://schema.wikimedia.org/repositories//primary/jsonschema/resource_change/1.0.0.json allows to... [09:48:59] 10Traffic, 10SRE: Varnish SLI is impacted by external components performance|behavior - https://phabricator.wikimedia.org/T317051 (10Vgutierrez) 05Stalled→03In progress actually there is some stuff that we can implement to avoid the issue described on the task description. the SLI should focus only on clie... [13:40:36] 10Traffic, 10Performance-Team, 10RESTBase-API, 10SRE: Text cluster is being hit with an average of 1.8k PURGE requests per second per host - https://phabricator.wikimedia.org/T318349 (10Krinkle) > non-PURGE requests VS PURGE requests hitting ats@cp3050 during the last 30 days: > {F35528462} Which dashboar... [13:48:13] 10Traffic, 10Performance-Team, 10RESTBase-API, 10SRE: Text cluster is being hit with an average of 1.8k PURGE requests per second per host - https://phabricator.wikimedia.org/T318349 (10Vgutierrez) The dashboard is https://grafana.wikimedia.org/d/kHk7W6OZz/ats-cluster-view?orgId=1&var-site=esams&var-layer=... [21:07:56] (HAProxyEdgeTrafficDrop) firing: 49% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [21:08:16] (VarnishTrafficDrop) firing: Varnish traffic in eqiad has dropped 65.88349738958671% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [21:12:56] (HAProxyEdgeTrafficDrop) resolved: 53% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [21:13:16] (VarnishTrafficDrop) resolved: Varnish traffic in eqiad has dropped 67.58995515154045% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop