[11:08:58] <_joe_>	 vgutierrez: what is the difference between hit-local and hit-front?
[11:09:15] <vgutierrez>	 hit-front it's a hit triggered on varnish
[11:09:20] <vgutierrez>	 hit-local it's a hit triggered on ATS
[11:09:36] <_joe_>	 ok
[11:09:41] <_joe_>	 I assumed the contrary
[11:10:00] <vgutierrez>	 both include refresh hits (so requests actually go one layer up and get a 304)
[11:10:24] <vgutierrez>	 that's negligible for varnish but not for ATS due to latency between DCs
[11:10:25] <_joe_>	 one layer up meaning the applayer for hit-local?
[11:10:29] <vgutierrez>	 indeed
[11:10:45] <vgutierrez>	 that's why we got some hit-local responses with a latency about ~100ms
[11:10:53] <vgutierrez>	 even ~200ms for eqsin
[11:12:07] <vgutierrez>	 see hit-local p75 in eqsin: https://grafana.wikimedia.org/d/8T2XA-5Gz/frontend-haproxy-ttfb-latency?orgId=1&refresh=1m&viewPanel=34&var-site=All&var-cache_type=text&var-cache_type=upload&var-status_type=2&var-cache_status=hit-local&var-ttfb_max=1.5&var-percentile=75
[11:12:14] <vgutierrez>	 VS p95: https://grafana.wikimedia.org/d/8T2XA-5Gz/frontend-haproxy-ttfb-latency?orgId=1&refresh=1m&viewPanel=34&var-site=All&var-cache_type=text&var-cache_type=upload&var-status_type=2&var-cache_status=hit-local&var-ttfb_max=1.5&var-percentile=95
[11:12:41] <vgutierrez>	 that bump is ATS needing to go back to the applayer to refresh cached content
[11:13:53] <vgutierrez>	 current network latency between codfw|eqiad<-->eqsin: https://grafana.wikimedia.org/d/m1LYjVjnz/network-icmp-probes?orgId=1&var-site=codfw&var-site=eqiad&var-target_site=eqsin&var-role=cr&var-family=All&viewPanel=2
[11:17:37] <_joe_>	 yeah no I'm aware
[11:17:50] <_joe_>	 I'm trying to understand why there's a specific behaviour in restbase
[11:18:11] <_joe_>	 but also why content with 
[11:18:24] <_joe_>	 cache-control: s-maxage=1209600, max-age=0, must-revalidate
[11:18:34] <_joe_>	 gets evicted from varnish and ATS apparently
[11:18:48] <_joe_>	 I assume it's a fault of the backend
[11:19:10] <_joe_>	 that we send the request to the backend (because of must-revalidate) and that tells us nope
[11:19:26] <_joe_>	 the same backend behaves differently for different urls
[11:20:49] <_joe_>	 uhm ok, the fact that the backend (restbase) doesn't send a Last-Modified header but an Etag
[11:21:00] <_joe_>	 and that etag *varies* between backends
[11:21:03] <_joe_>	 isn't helping
[11:33:26] <vgutierrez>	 _joe_: you can filter those in the atslog-backend with TCP_REFRESH_MISS
[11:35:03] <vgutierrez>	 atslog-backend TCP_REFRESH_MISS |grep OriginServer:restbase will do it
[11:38:11] <vgutierrez>	 _joe_: it definitely looks like ATS isn't able to get any TCP_REFRESH_HIT from restbase
[11:39:18] <vgutierrez>	 we could definitely benefit from https://phabricator.wikimedia.org/T321484 :)
[11:43:35] <vgutierrez>	 I need to feed the cats, but I'll hack T321484 afterwards
[11:43:40] <stashbot>	 T321484: Track TTFB per Cache Status Code in ATS - https://phabricator.wikimedia.org/T321484
[11:44:40] <vgutierrez>	 (and I guess I should eat as well...)
[11:49:07] <_joe_>	 vgutierrez: the can of worms we just opened is deep
[11:49:24] <_joe_>	 it's a mix of restbase thinking it knows better than varnish, and flaggedrevs
[11:50:56] <vgutierrez>	 :)
[14:29:17] <vgutierrez>	 _joe_: it's brand new, but it shows what you suspected https://grafana.wikimedia.org/goto/4ZnVHxHVz?orgId=1
[14:30:20] <_joe_>	 vgutierrez: I have the full shit figured out sadly, I'm writing a comment
[14:30:29] <vgutierrez>	 nice :)
[14:31:46] <_joe_>	 vgutierrez: it's the result of hubris, specifically someone who thought they knew better how to cache content than our CDN
[14:32:06] <_joe_>	 turns out MediaWiki, which trusts the caches to a high degree, does it better
[14:32:13] <vgutierrez>	 as long as it can be fixed it's fine :)
[15:14:13] <Krinkle>	 "the can of worms we just opened is deep", I guess that's good news for vgutierrez's cat.
[15:14:37] <vgutierrez>	 I've cooked 1.2kg of chicken's breast for my 3 cats
[15:14:44] <vgutierrez>	 dunno if they are gonna be interested on some worms ;P
[15:15:28] <Krinkle>	 Ah, our house cats loved them. We didn't home cook for them though. That's sounds awesome.
[15:16:09] <cdanis>	 my cats are junk food addicts and only like crunchy kibble no matter what else we've tried
[15:17:23] <sukhe>	 cdanis: this reminds me of https://www.cbc.ca/news/science/raccoons-high-blood-sugar-1.4718418
[15:17:27] <sukhe>	 "Toronto Raccoons experiencing high blood sugar levels from eating our food
[15:17:30] <sukhe>	 "
[15:17:32] <cdanis>	 😂
[15:24:22] <_joe_>	 ahahahahahaha
[15:24:25] <_joe_>	 sukhe: that's amazing
[15:25:09] <vgutierrez>	 I've been feeding them 100-130 grams of chicken's breast + 30 grams of sweet potato/regular potato per day during the last week
[15:25:31] <vgutierrez>	 they seems really happy with me and they don't miss their usual human
[15:27:08] <Emperor>	 Ah, bribery :)
[15:28:04] <sukhe>	 _joe_: as you can see, vgutierrez's cats have a more balanced diet than most humans
[15:31:00] <_joe_>	 sukhe: that's the diet of a bodybuilder basically
[15:32:12] <claime>	 Why the sweet potato/reg potato? 
[15:32:27] <claime>	 I don't think there's anything in there a cat needs, is there? Is it just for bulk?
[15:32:49] <claime>	 (genuinely asking btw, I have two cats, so vested interest)
[15:33:39] <bblack>	 I assume cats need *some* carbs and other nutrients.  In the wild, they wouldn't be eating just a cooked muscle.  they'd be eating raw meat, more blood, organs, a bit of skin, etc.
[15:39:43] <vgutierrez>	 bblack: cats are strict carnivores.. so the amount of carbs they need is quite low
[15:41:13] <vgutierrez>	 ideally we'd feed them raw meat (same as our dogs).. reality is that adopted cats that are used to eat commercial stuff that's not really great for them
[15:42:32] <vgutierrez>	 sadly cats need supplements unless you feed mouses to them.. mainly taurine.. and essential (they need to eat it instead of produce it themselves) amino acid for cats
[15:42:40] <vgutierrez>	 s/mouses/mice
[15:43:01] <vgutierrez>	 anyways.. my partner is the expert about this.. I just follow the "how to keep the cats alive" howto
[15:43:06] <sukhe>	 vgutierrez: be honest, how many research papers have you read about cat nutrition?
[15:43:26] <vgutierrez>	 me? 0, my girlfriend? hundreds of hours at this point...
[15:43:47] <vgutierrez>	 and talking about her... she is going to land pretty soon..
[15:43:50] * vgutierrez headed to the airport!
[15:46:57] <vgutierrez>	 But first...
[15:47:03] <vgutierrez>	 Some cat spam
[15:47:26] <vgutierrez>	 https://usercontent.irccloud-cdn.com/file/x75OxpkE/irccloudcapture1557245249351489709.jpg
[15:48:36] <jynus>	 not sure why I spected cans of SPAM specially indicated for cat nutrition
[16:08:02] <cdanis>	 vgutierrez: Oh are we posting cats? https://media.discordapp.net/attachments/1023929743831224381/1035585222902820904/IMG_1042.jpg
[16:46:29] <vgutierrez>	 <3
[17:13:01] <inflatador>	 Moar plz
[18:09:56] * TheresNoTime notices the cat tax is being paid..
[18:12:13] <TheresNoTime>	 https://usercontent.irccloud-cdn.com/file/5eymiGYe/PXL_20220925_175721892.jpg
[18:19:26] <brett>	 https://cloud.i--b.com/s/8x3rLEbwHWiZc7R/preview
[18:20:32] <brett>	 TheresNoTime: such chubby cheeks :D
[18:21:09] <inflatador>	 Oldie but goodie https://ewr1.vultrobjects.com/fun/angelfish.jpg
[18:21:46] <inflatador>	 and one to getcha ready for the weekend! https://ewr1.vultrobjects.com/fun/sleepy_pringles.jpeg
[18:22:24] <brett>	 The eyes match the netflix mail D:
[18:22:43] <brett>	 I see a bonus hidden cat behind the couch too
[18:23:10] <TheresNoTime>	 brett: she was asleep in a box, woken especially for a photo.. was not impressed :D
[18:24:51] <inflatador>	 brett oh wow, I never noticed that myself!