[00:57:56] (HAProxyEdgeTrafficDrop) firing: (3) 47% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [01:02:56] (HAProxyEdgeTrafficDrop) resolved: (4) 63% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:13:56] (HAProxyEdgeTrafficDrop) firing: 48% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:18:56] (HAProxyEdgeTrafficDrop) resolved: (3) 66% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:29:20] 10Traffic, 10Prod-Kubernetes, 10SRE, 10serviceops, 10Kubernetes: service:.catalog entries and dnsdisc for Kubernetes services under Ingress - https://phabricator.wikimedia.org/T305358 (10JMeybohm) [12:47:35] 10netops, 10Infrastructure-Foundations: Cannot verify NTP satus asw1-b12-drmrs - https://phabricator.wikimedia.org/T305840 (10cmooney) p:05Triage→03Low [12:57:45] 10netops, 10Infrastructure-Foundations: Cannot verify NTP status asw1-b12-drmrs - https://phabricator.wikimedia.org/T305840 (10Volans) [13:15:08] 10Traffic, 10SRE, 10SRE Observability (FY2021/2022-Q4), 10User-fgiunchedi: Migrate Traffic Prometheus alerts from Icinga to Alertmanager - https://phabricator.wikimedia.org/T300723 (10lmata) [13:23:19] 10Traffic, 10Prod-Kubernetes, 10SRE, 10serviceops, 10Kubernetes: service::catalog entries and dnsdisc for Kubernetes services under Ingress - https://phabricator.wikimedia.org/T305358 (10JMeybohm) [14:20:05] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin: cp5002 memory errors on DIMM A4 - https://phabricator.wikimedia.org/T305423 (10RobH) p:05Triage→03Medium [14:21:27] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: ganeti4002 dimm error - https://phabricator.wikimedia.org/T303318 (10RobH) I'll chase this down today, I got the notice of processing but no shipment so I'll need to email Dell and find out what happened with this. [14:21:40] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: ganeti4002 dimm error - https://phabricator.wikimedia.org/T303318 (10RobH) p:05Medium→03High [16:23:01] 10Traffic, 10SRE: Denial of Service due to repeated hits from a particular IP - https://phabricator.wikimedia.org/T305863 (10RLazarus) Routing to #traffic to see if this is a VCL rule we're hitting. @ERayfield Can you provide some example requests, with headers and source IP? I'm going to preemptively make t... [17:45:56] (HAProxyEdgeTrafficDrop) firing: 55% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [18:15:56] (HAProxyEdgeTrafficDrop) resolved: 61% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [18:49:28] 10Traffic, 10Performance-Team, 10SRE: Enable HTTP compression for arclamp trace logs - https://phabricator.wikimedia.org/T305783 (10Krinkle) p:05Triage→03Medium a:03dpifke