[11:34:10] <snitch>	 [[Tech]]; খাত্তাব হাসান; /* Help Needed */ Reply; https://meta.wikimedia.org/w/index.php?diff=24714911&oldid=24712413&rcid=26680358
[12:12:09] <snitch>	 [[Tech]]; AKlapper (WMF); [none]; https://meta.wikimedia.org/w/index.php?diff=24715101&oldid=24714911&rcid=26680664
[16:39:13] <roy649_>	 When a do a get on https://upload.wikimedia.org/wikipedia/commons/thumb/b/b2/Pete_Sutherland_playing_at_Bristol_town_tribute_%28cropped%29.jpg/280px-Pete_Sutherland_playing_at_Bristol_town_tribute_%28cropped%29.jpg, I get "Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes."
[16:39:24] <roy649_>	 Just in case this isn't already a known issue.
[16:40:18] <roy649_>	 Request from 2600:4041:51ec:c500:fdcb:d9c7:57cb:3c21 via cp1090 cp1090, Varnish XID 716451937
[16:40:18] <roy649_>	 Upstream caches: cp1090 int
[16:40:18] <roy649_>	 Error: 404, Not Found at Tue, 14 Mar 2023 16:38:25 GMT
[16:58:57] <AntiComposite>	 works for me
[17:18:09] <roy649_>	 Hmmm, working for me now too.  I'll blame this on everybody's favorite issue: cache invalidation :-)
[17:24:18] <roy649_>	 There's still something going on.  I'm getting similar results now for:
[17:24:31] <roy649_>	 https://upload.wikimedia.org/wikipedia/commons/thumb/archive/b/b2/20230314172235!Pete_Sutherland_playing_at_Bristol_town_tribute_(cropped).jpg/120px-Pete_Sutherland_playing_at_Bristol_town_tribute_(cropped).jpg
[17:24:32] <roy649_>	 and
[17:24:38] <roy649_>	 https://upload.wikimedia.org/wikipedia/commons/thumb/b/b2/Pete_Sutherland_playing_at_Bristol_town_tribute_(cropped).jpg/120px-Pete_Sutherland_playing_at_Bristol_town_tribute_(cropped).jpg?20230314172238
[17:25:23] <roy649_>	 Request from 2600:4041:51ec:c500:fdcb:d9c7:57cb:3c21 via cp1090 cp1090, Varnish XID 725720168
[17:25:23] <roy649_>	 Upstream caches: cp1090 int
[17:25:23] <roy649_>	 Error: 404, Not Found at Tue, 14 Mar 2023 17:25:14 GMT
[17:25:25] <roy649_>	 and
[17:25:43] <roy649_>	 Request from 2600:4041:51ec:c500:fdcb:d9c7:57cb:3c21 via cp1090 cp1090, Varnish XID 703262314
[17:25:43] <roy649_>	 Upstream caches: cp1090 int
[17:25:43] <roy649_>	 Error: 404, Not Found at Tue, 14 Mar 2023 17:25:32 GMT
[22:05:30] <bd808>	 roy649_: those look like more of the mystery failures from T331820
[22:05:31] <stashbot>	 T331820: Upstream caches: 404 - https://phabricator.wikimedia.org/T331820
[22:06:40] <roy649_>	 yup
[22:09:44] <bd808>	 typically swift (the media storage system) would not return a 404 unless the source file actually does not exist. When it looks for a thumbnail and cannot find it swift should call Thumbor to create the thumb from the original and then store and return the thumb.
[22:25:15] <bawolff>	 hmm, that is a weird one
[22:43:46] <bd808>	 much speculative discussion of T331820 is happening in #wikimedia-sre. No answer yet but some digging into logs and dashboards at least
[22:43:47] <stashbot>	 T331820: Upstream caches: 404 - https://phabricator.wikimedia.org/T331820