[00:00:08] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552156 (10ssingh) Thank for reporting. Can you please try again from your home connection (without VPN) and let us know if that works? We depooled... [00:01:28] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552158 (10AlexisJazz) Works now, thanks! [00:02:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp3070:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=esams&var-instance=cp3070 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:05:00] RESOLVED: [32x] PurgedHighEventLag: High event process lag with purged on cp3066:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [00:05:31] yeah that's fine [00:07:36] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552161 (10AlexisJazz) {F71607077} There was a visible dip in editing and surge in error responses. [00:17:40] RESOLVED: VarnishHighThreadCount: Varnish's thread count on cp3070:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=esams&var-instance=cp3070 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [01:19:27] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552172 (10ssingh) We had a transient link failure between eqiad and esams that resulted in this issue. It should be resolved now, and esams is poo... [01:33:42] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552173 (10Peachey88)