[00:53:04] https://usercontent.irccloud-cdn.com/file/UkdLMD2a/triple_count_mismatch_btw_eqiad_codfw.png [00:54:53] Hadn't noticed this before. I'm thinking some of the recent streaming updater issues mean that codfw is in an uncertain state. That could explain the (presumed) worse performance on queries thus leading to the issues [00:55:14] Likely we should do a data transfer from an eqiad host to a codfw one and then seed the rest of codfw with that host's journal file [00:56:43] The delta is about 300 million triples [00:57:37] I don't have a great mental model of how blazegraph internally stores its knowledge graph, but I imagine there must be a mechanism whereby despite the % difference in triples being extremely low between eqiad/codfw that in practice the overall graph is more sluggish [00:58:39] Need to take off for now so will hold off on doing any data transfers for the timebeing [01:06:30] lol did my math super wrong, I think it's 30k not 300 million. but yeah anyway I do still find the mismatch to be odd, doesn't seem like it's explainable by eqiad being more behind in updates or something