[00:36:00] 10Traffic, 10SRE: Image load failing with 429 from varnish - https://phabricator.wikimedia.org/T285875 (10Legoktm) p:05Triage→03Unbreak! In general we've made some changes recently to rate limiting after repeated abuse/DDoS attacks. Could you please clarify what software (e.g. Firefox, Chrome, some other t... [01:03:53] 10Traffic, 10SRE: Image load failing with 429 from varnish - https://phabricator.wikimedia.org/T285875 (10RoySmith) > Could you please clarify what software (e.g. Firefox, Chrome, some other tool) you're using to access pages/images that is returning 429s? I'm not sure who that was intended for, but I get sim... [01:10:01] 10Traffic, 10SRE: Image load failing with 429 from varnish - https://phabricator.wikimedia.org/T285875 (10Legoktm) >>! In T285875#7188943, @RoySmith wrote: >> Could you please clarify what software (e.g. Firefox, Chrome, some other tool) you're using to access pages/images that is returning 429s? > > I'm not... [01:22:49] 10Traffic, 10SRE: Image load failing with 429 from varnish - https://phabricator.wikimedia.org/T285875 (10RoySmith) If you could generate the URLs for the other size images, I'd be happy to give them a try from here. [07:46:02] 10Traffic, 10SRE, 10User-notice: Rate limit requests in violation of User-Agent policy more aggressively - https://phabricator.wikimedia.org/T224891 (10ayounsi) p:05Medium→03High [09:01:09] 10Traffic, 10SRE: Preserve Server response header when generating custom error page via VCL - https://phabricator.wikimedia.org/T285926 (10ema) [09:49:56] (VarnishTrafficDrop) firing: (2) 60% GET drop in text@ulsfo during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [09:59:56] (VarnishTrafficDrop) resolved: (2) 69% GET drop in text@ulsfo during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [12:32:56] (VarnishTrafficDrop) firing: (2) 67% GET drop in text@eqiad during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [12:37:56] (VarnishTrafficDrop) resolved: (2) 65% GET drop in text@eqiad during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [13:31:43] 10Traffic, 10SRE, 10Patch-For-Review: Preserve Server response header when generating custom error page via VCL - https://phabricator.wikimedia.org/T285926 (10ema) p:05Triage→03Medium [14:23:32] 10Traffic, 10SRE: cp3059 Varnish child crash: Worker Pool Queue does not move - https://phabricator.wikimedia.org/T285953 (10ema) [14:43:13] 10Traffic, 10SRE: cp3059 Varnish child crash: Worker Pool Queue does not move - https://phabricator.wikimedia.org/T285953 (10ema) Relevant upstream issues: - https://github.com/varnishcache/varnish-cache/issues/2814 - https://github.com/varnishcache/varnish-cache/issues/2862 Related patch to look into: https... [14:44:48] 10Traffic, 10SRE: cp3059 Varnish child crash: Worker Pool Queue does not move - https://phabricator.wikimedia.org/T285953 (10ema) p:05Triage→03Medium