[11:15:13] lunch [13:04:41] I've been talking to Olja about where we're at in the WDQS Graph Split. Our goal being to have a data reload under 10 days. [13:06:08] 7 days were mentioned as our current reload time. But I think that number is just for loading the data into Blazegraph once it is available locally on the servers and does not take into account all the previous steps (loading dumps into hive, splitting the graph, exporting, transferring to the target servers...) [13:06:42] We haven't really done all those operations in one go, so I don't think we have real measured numbers on the time it takes end to end. [13:07:15] Also, not having automation means that we're loosing a lot of time between the steps. [13:07:55] My current estimation of the full reload time would be more 12 days than 7 days. But we know we can gain a lot by just automating the process. [13:08:04] Does this sound reasonnable? [13:08:57] Also one big caveat is that while we are confident that we will be able to reload split graph in < 10 days, we don't yet have validation that the resulting split is usable. [13:27:24] the reload in < 10days was in comparison with what we usually see in T241128 (in the column import time) [13:27:25] T241128: EPIC: Reduce the time needed to do the initial WDQS import - https://phabricator.wikimedia.org/T241128 [13:28:31] at least that's my understanding so not taking into account the delay eaten by waiting for the dumps to be present into hive [13:38:50] if we want to be end-to-end the kr should be more like: "the initial lag of a freshly imported wdqs node is < 10days" [13:45:06] I think we discussed an initial value of > 30 days, which matches more with multiple attempts of a full data import [13:46:24] ok, it looks like we at least need to clarify what the KR exactly is [14:17:07] o/ [14:41:58] Checking the current wording: "Wikidata knowledge graph can be reloaded within 10 days for a graph of up to 20 billion tuples." Not entirely clear what we mean here. [14:42:49] o/ [14:42:50] I wouldn't be entirely happy with "initial lag" as a measure, as it does not take into account the need to restart the loading process multiple times in the cases where it crashes. [14:46:32] gehel: true, depends on what we want to cover, initially I thought of this as: the "blazegraph import step" being the most fragile is what we want to measure and optimize as a kr for the graph split [14:47:28] I'm thinking more about the overall process, as it includes retries and is thus a better indicator of the stability of the system. [14:47:29] if we want this kr to cover the work of automating the whole process we might want to reword that a bit [14:48:53] I'm also thinking that we need a fully working and automated solution to be able to call this done, so the overall process makes more sense in this context. If we just use the Blazegraph step, we could call this done right now, but that's obviously not the case. [14:49:09] true [14:49:15] sounds fun! [14:49:44] In the end either way is fine, as long as we agree on the definition. We could use just the blazegraph process, but make it clear that additional work needs to happen outside of getting the number down. [14:50:18] this 10days thing was initially for me a precondition for a split, as in, we would not accept a set of splitting rules that result in an import time > 10days [14:51:01] I'm using this more as a measure of the overall progress of our work and of the impact we have on the system. [14:51:11] Both approaches are valid, just different :) [14:52:34] could we have multiple krs? the import time is I think interesting to validate that the rules of the split are meaningful, we could add more to cover the automation and the rest? [14:56:05] At the SDS3.1 level, we should have a single KR. But to drive our engineering work, we can very much add more metrics and details. [14:56:35] And we already do that: we know that one important thing that we also need to measure is the impact on query time and functionality for example. [16:00:07] \o [16:00:20] o/ [16:17:28] Thanks to jayme, we finally got envoy to follow our retry + timeout instructions. We now also see 504 (gateway timeout) upstream responses. [16:17:59] \o/ [16:53:56] this kind of pages: https://www.wikidata.org/wiki/Wikidata:Database_reports/Constraint_violations/P1807 might explain why wikidata is so slow to fetch&parse from the SUP, every line is calling a template to load the item label... [16:54:46] bot's and the wikitext parser, name a better match :P [16:55:21] at least it's not editing that page multiple times a day [16:55:28] yes... [16:55:36] ryankemper just a heads-up so we don't get our wires crossed...I'm hacking on https://gerrit.wikimedia.org/r/c/operations/puppet/+/989244 ATM [17:08:39] dinner [17:22:20] getting my laptop, back in ~1h [17:29:06] dinner [18:19:15] * ebernhardson wonders if that's right, somehow i have 120 hours of sick leave in dayforce. Seems excessive [18:20:09] sent a msg to discovery-internal, lots of distraction from sinus issues today. Taking the day off [19:03:35] sorry to hear it! Feel better! [19:10:14] restoring my laptop, probably take a few hours [19:17:10] ebernhardson: you mean nasal sinuses? [19:17:51] i tend to treat such with steam inhalaation (not at 100C , maybe at 60C) above a pot, or in a bathroom full of steam [19:20:42] if we're sharing our weird tips, I like xylitol+saline nasal sprays. xylitol's really good at degrading bacterial biofilms (this study is about dental plaque specifically but is illustrative https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7325245/) [19:27:04] +1 to xylitol. Pur (a Swiss company!) makes really good xylitol gum https://thepurcompany.com/ [21:23:22] Re: the notes above from g.ehel and d.causse, quick braindump (I think this mirrors shared knowledge) - [21:23:37] Agreed we need to automate. Of course right now we're focused on having the endpoints internet facing and having some query coverage (David's program) and performance comparison pieces (I'll be working on that piece over the course of the next week and want to kick it off before heading out for the weekend next week). Looking forward to this in following months for sure. [21:23:53] Here's a rough view of steps to orchestrate, I think: [21:23:59] TTL import: 3.5 hours, but let's round up to 4 (it seems to infrequently take that long, and in the rare cases sometimes 4.5 hours). I actually think this could be made to be faster, but it's not something we want to mess with, as there are probably other bottlenecks to address. [21:24:07] Split into separate HDFS tables: 2 hours or less. This also could be sped up a little (for that matter technically it could be inlined in parts of TTL import, but it's not worth it - this is better as a discrete step), although same thing probably better to look at other bottlenecks I think. [21:24:17] Write new N-Triples files: 10 minutes (I think last time it took 7 minutes) [21:24:24] Copy files to server: 30 minutes-ish [21:24:36] Import new split-graph N-Triples files: 7 days (https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&var-cluster_name=wdqs-test&viewPanel=7&from=1701907200000&to=1702598399000) - [21:24:38] it actually appears it can be less than that but there's some sort of reoptimization thing going on in Blazegraph from the point where really the data have been imported (you'll notice also that one side of the graph loads in faster than the other; but remember we may have some mild artifacts of host OS systemd things...whatever, though). [21:24:45] 10B for one side of a maximal 20B triple graph (assuming equal distribution) / 7.67B triples baseline = 1.31 time scaling factor, if linear (y'never know, but let's guess) [21:24:51] 7d6h40m * 1.31 = 9.53 days if everything played nicely. [21:25:04] Before that the Wikidata dumping process takes what, maybe 2.5 days if we speculatively load without any checksum? (I looked at how this works a month ago, but am just eyeballing dumps.wikimedia.org directory and file timestamps right now, and maybe someone here knows better.) [21:25:35] So I think in total 12 days (g.ehel is a number magician?), maybe 11 if you're looking from the point of a dump process initiated to the point of import onto the test servers when test system is stable; at that point queries ought to be possible and then streaming updates come in to get it up to speed. [21:25:43] This is glossing over other details, of course. I think the biggest thing is driving down the second to last part: importing the split-graph N-Triples files - because my machine can do some 8B triples in sub-5 days, I'm very keen to see how a depooled prod node might do for comparison. The detail being how much is CPU bound, disk bound, or some aspect of configuration of the channel between them. [21:28:34] Unrelated: re: xylitol, makes me think of other saccarides, makes me think of ice cream! [21:34:00] back [21:39:33] I do love making ice cream...I should do that again sometime soon [22:26:03] A detail in the timing for the dumps is some arrive in different order because of weekly runs. There are HDFS-side jobs to check daily, but the dumps run weekly - see https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/snapshot/manifests/systemdjobs/wikidatadumps/rdf.pp and https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/snapshot/files/systemdjobs/wikibase/ [22:26:48] So it may be interesting to have something running more like daily if the servers can handle it, so that we can cut out any weekly rhythm requirement...but future problem, not now