[08:45:51] jbond and I are disabling puppet fleet wide for few minutes to merge 2 changes [08:46:27] Ack, thanks. [08:56:16] <_joe_> volans: lmk when you've done, I have a couple things to apply [08:57:47] _joe_: ack, will be few minutes (hopefully) just checking few things to make sure it's all ok [09:02:54] <_joe_> yeah np, just let me know [09:07:20] * volans waiting for jenkins [09:14:38] _joe_: all re-enabled now [09:14:46] <_joe_> :* thanks both [09:50:30] <_joe_> trying again a full appserver fleet restart [10:20:15] I'm disabling puppet for ~5 min on mw/mc (and a handful more) to merge a patch [10:20:52] k [10:29:12] everything back to normal [10:29:26] (or what passes for normal around here ;)) [10:30:33] heh :-) [12:11:00] please confirm or deny: mcrouter does not replicate data across data centers. [12:46:11] <_joe_> duesen: I cannot confirm or deny [12:46:19] <_joe_> duesen: we replicate only deletions [12:46:34] <_joe_> and only for specific prefixes [12:46:54] <_joe_> that's what WanObjectCache uses IIRC [12:47:00] _joe_: nothing is ever simple :) [12:47:22] <_joe_> duesen: I blame.... timo and aaron and effie! [12:47:47] <_joe_> I'm joking I'm the one to blame for the complexity of this, they were just playing along :) [12:48:33] <_joe_> but basically, we allow mediawiki to decide itself what should be replicated and what should not [12:49:39] The reason I'm asking is data-parsoid stashing. We are moving the functionality out of RESTbase, but it's currently unclear where to store the stash in the future. I'm looking at T212129 since it seems similar. [12:49:40] T212129: Move MainStash out of Redis to a simpler multi-dc aware solution - https://phabricator.wikimedia.org/T212129 [12:50:54] duesen: each dc memc is independent, the only thing we replicate with evantual consistency is delete tombstones. this is why our general pattern is get-with-set with purge on change, and not proactively set, the expectation is that memc is on-par with replica db (not master db) - high level overview at https://wikitech.wikimedia.org/wiki/Memcached_for_MediaWiki#WANObjectCache [12:50:57] I'm still talking to the editing team about requirements, but what we may need to reliable storage for 24 hours, replicated across data-centers, but not write/write. [12:51:18] for unstructured data that should not be subject to LRU, mainstash is indeed the right place. [12:51:48] although right now it is limited to 500M globally for all wikis, we're moving this out from redis to mysql to make space for new use cases among other things. It was shrunk a lot after we moved sessions to kask. [12:53:03] Krinkle: and we just put the data-parsoid stash in main stash for now? And let it be migrated away from Redis with PageEditStash and friends? [12:53:17] I was thinking we should avoid putting more stuff into Redis. I'm not sure 500MB will be sufficient. [12:53:18] duesen: overview of all caches (not just memc) is at https://www.mediawiki.org/wiki/Manual:Object_cache - which should guide you what store is appropriate [12:54:17] stash is usually only for a few minutes around the edit submission cycle, best-effort in memcached for non-Restbase use cases. [12:54:56] Krinkle, _joe_: I shared with you the overview doc I wrote for Editing, please edit/comment as you see fit. [12:55:31] Yea, the data-parsoid stash is much more long term, needs to be pretty reliable for 34 hours [12:56:25] <_joe_> duesen: I'll take a look [12:59:30] *24 hours [12:59:40] _joe_: thank you! [12:59:43] <_joe_> duesen: frankly, my first guess for an editstash would be... cassandra [13:00:36] _joe_: yes, but how? via kask, or directly? and using an existing cluster, or do we need a new one? also, we don't need write/write... [13:01:03] _joe_: can we get away with just using redis? Or with setting up another SQLBagOStuff?... [13:01:16] that'S what I'm currently trying to figure out :) [13:01:24] <_joe_> duesen: redis? What did I do wrong to you? :P [13:01:48] <_joe_> joking ofc, but no, redis isn't a good choice for something we care about if it gets lost [13:07:05] duesen: I don't know if it helps, but recently we've been working on opening up the aqs_next cassandra cluster so that it's usable by multiple applications (no longer just AQS). [13:07:51] _joe_: well, Krinkle said that the "main stash" concept is designed for handling this case, which is currently backed by redis, so... [13:08:18] <_joe_> duesen: and I've asked a long time ago that we stop doing so :P [13:08:41] It is going to be used for the Image Recommendation Feedback data: (T295405) - We're currently making this cluster multi-DC. [13:08:41] T295405: Implement Image Suggestions Schema in Cassandra - https://phabricator.wikimedia.org/T295405 [13:09:00] <_joe_> we have a task for moving all the stuff that needs cross-dc replication off of redis [13:10:51] duesen: the way parsoid uses this sounds a lot more like parsercache than stash, i'm curious where the 34h comes from, is that a new value or based on what restbase does today? [13:11:16] e.g. this is not for stashing during an edit, but for comparing against the unmodified base and ensure no dirty diff, selser etc. right? [13:11:20] <_joe_> btullis: yeah that seems like a good place for small cassandra datasets [13:12:12] Krinkle: sorry, typo: 24 hours. It's the max time between opening VE and submitting the edit. If your edit takes longer, your edit fails. [13:12:30] <_joe_> btullis: wait so you're telling me that we'll have multi-dc aqs? [13:12:31] Yes, it's for comparing against the unmodified base rendering [13:16:21] Right but we don't need this if the html of that revision is in the parsercache. That'll naturally be there either from page view miss (once default) or from opening VE. And that would have a 30d TTL anyway [13:17:37] I think we may want to sort that out first instead of approaching it as a stash which makes it seem much lighter than it is. I guess an estimate would help here. Eg unique rev IDs opened by VE in a day globally and add up the html expanded sizes [13:18:11] Or if it already uses a separate store or namespace today we can compute it empirically [13:18:17] * Krinkle back in a bit [13:28:49] does anyone have experience with facebook's zstd compression? I read https://www.percona.com/blog/dump-performance-comparison-mysqldump-vs-mysql-shell-utilities-vs-mydumper/ and it claims a 2x speed increase on mydumper speedup [13:30:05] _joe_: Yes, multi-DC AQS is a work-in-progress: T307641 [13:30:06] T307641: AQS multi-datacenter cluster expansion - https://phabricator.wikimedia.org/T307641 [13:52:48] Krinkle: it uses a separate Cassandra store today. Eric has the details. We can't rely on PC, because the rendering in the PC at the end of the edit may not be the one that we started the edit with. [14:25:08] jynus, a tiny bit of discussion here: https://phabricator.wikimedia.org/T222985 also somewhat dated but maybe better than nothing [14:25:46] apergos: thank you, that is actually quite useful [14:25:58] 👍 [14:29:25] I may have to do some tests for specific datasets and to balance cpu, speed and disk usage [14:29:36] apergos: at the moment, that is not an option provided, right? [14:29:45] no [14:30:13] e.g. if it was already in place that would be "a point in favour" [14:30:44] but seeing it was discussed means it is not super-obscure [14:31:32] I would have to check, however, what is the largest bottleneck for my use case [15:06:16] jynus: do we regularly make plain sql dumps? Which feature, service or tables do we do that for? [15:44:38] could you be more specific with your question? To me that would be like asking "what do you have backups of"? Which the simple answer would be "everything that makes sense" [15:48:03] Maybe this would help answering your question: https://phabricator.wikimedia.org/source/operations-puppet/browse/production/hieradata/role/common/dbbackups/monitoring.yaml although we have more backups than those- just not regularly created (e.g. ES read only hosts) [15:49:59] the other answer would be to wait for the new backupmon dashboard soon to be deployed, which may enable you to answer the specific question you have [16:00:37] then there is the definition of "plain sql"- basically it is not simple to answer without more context of what you want to know [16:14:07] jynus: I'm responding to the article you shared and was wondering where we'd use it in prod. The various export/dump use cases I'm aware of use a different format, either higher level or lower level. Eg json/xml for mw and more sanitised, or eg replication or partial disk backups. [16:14:29] As opposed to adhoc use cases when debugging [16:14:37] I see now [16:15:07] I think you are confusing exports vs backups, which will make Ariel not super-happy :-D [16:15:41] jynus: ack, well 2/3 commands are called dump [16:16:03] he he [16:16:25] but you're saying we use sql exports of our core dbs regularly? Full unfiltered backup? I guess that takes a while, from a depooled replica that then has to catch up again? [16:17:33] let's go pm, so I can get super-pedantic with meanings of "use" and "unfiltered" without anoying anyone else :-) [16:18:38] the whole explanation is at https://wikitech.wikimedia.org/wiki/MariaDB/Backups for anyone else curious, and I am very happy to get additional questions [17:33:52] I think I have Krinkle a satisfying explanation of current backups and improvements coming soon :-) [17:33:58] *given [17:34:41] it doesn't help that dump is a super-confusing word that is overloaded in mediawiki context :-(