TerarkDB, ByteDance's RocksDB replacement
github.com
github.com
RocksDB is a fork of Google's LevelDB [1] at Facebook. Optimized to exploit many CPU cores, and make efficient use of fast storage, such as solid-state drives (SSD), for input/output (I/O) bound workloads.
LevelDB [2] is an open-source on-disk key-value store written by Google fellows Jeffrey Dean and Sanjay Ghemawat. Inspired by Bigtable.
Bigtable [3] is a compressed, high performance, proprietary data storage system built on Google File System, Chubby Lock Service, SSTable (log-structured storage like LevelDB) and a few other Google technologies.
I hope the info above is useful for others as I dont have the slightest idea what RockDB is, but I do know LevelDB.
[1] https://en.wikipedia.org/wiki/RocksDB
[0] https://github.com/cockroachdb/pebble
[1] https://www.cockroachlabs.com/blog/pebble-rocksdb-kv-store/
I am one of the maintainers of this project you can ask any question here.
What are the reasons for the perf improvements we see here?
"An error occurred. This browser is not supported, click here to learn more."
[1] https://bytedance.feishu.cn/docs/doccnZmYFqHBm06BbvYgjsHHcKc
I'm on firefox on MacOS too, and it happens with my ad blocker on or off. Chrome works fine. The support document linked in the error explicitly says only chrome and safari are supported on macos. I'm confused why my grandparent comment has been downvoted - this is a real bug report stopping me (and maybe others) from reading documentation that looks to have a lot of thought put into it. And given the content seemed to be loading fine before the error message came up, well, it feels forced.
I want to address that we are not meant to "get rid of" RocksDB (which lots of KV engine claimed). What we want to do is provide another solution for storage engine users with different road path (focusing on new hardware and heavy-write workloads).
For simple use cases, there will be no difference no matter what engine you use.
And for most cases, upgrade your hardware (e.g. SATA SSD to NVMe SSD) or tuning your RocksDB parameters would save you lots time, just make sure you understand what you are doing.
There's no cue for every workloads, try TerarkDB if RocksDB happens not fit your scenario.
1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD/Memory and others with ZNS...) may participant in this project so we want to handle all commits ourself to make sure everything is under control.
EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.
Please someone feel free to correct me, but if I recall correctly a lot of the improvements in Vim 8 were a result of the popularity of functionality in NeoVim?
Second, even if you do the first, the whole thing gets screwed up again when you start trying to introduce vendor code into the mix. Generally, no one upstream gives a crap that you have super compelling business reasons to compromise on code quality (or even trivial things like how code is committed: tarballs vs good git hygiene), and vendors sometimes compromise a lot.
So it’s not surprising that sometimes groups choose to do the expedient thing to get something to market instead of doing things “the right way.” In a lot of respects, the original Android did this with Linux.
Competition is good.
Android vendors keep doing this over and over again with Linux, which explains why so many phones are stuck on old versions of Android.
Are they giving back the source? And letting Mongo merge their changes if they wish?
Because that's what open source is all about.
There are much more egregious and fundamentally different violations to open source namely those you mention in your comment.
The comparison figures presented looked really impressive, thanks for sharing it.
So RocksDB came from LevelDB and here we go again.
But now when I look at the new repo and the online doc there is no mention of succinct data struct anywhere.
Also, the benchmarks back then claimed 10x or more faster than RocksDB. Now the performance claim is much more modest.
Does that mean TerarkDB no longer uses succinct data struct? Or are you just open sourcing a lower-end version of the software without the secret sauce?
Can you talk about what makes TerarkDB faster than RocksDB?
Here's the reasons: 1. Our `all-in-one` docs are still under writing, we will cover that part later. 2. For the performance part, we are now showing real-world cases, not a well-designed benchmark.(We selected the best result to show our work few years ago, don't want to do it anymore) 3. About why TerarkDB is faster than RocksDB will be explained in our `all-in-one docs` in one week, and most of the reasons are not magic, just engineering efforts
Thanks again for your remembering us.
So... You're saying there is magic? :)
Almost all kinds of workloads are here since TerarkDB runs under too many database clusters (each cluster only serves a single application)
Hopefully, that will also brings better Java test coverage and integration.
I remembered that we tried it on Flink in early versions are the result is pretty good.
I would be happy to be contacted about specific Java API issues with RocksDB; Maybe I can help.
By separating out large values there's less write amplification and things get faster because more of the SSTs fit in RAM cache. RocksDB wasn't historically a great choice to hold things like file uploads - you'd use the traditional filesystem for that. But that's quite constraining. When large values work better, it not only is a performance increase, but it enables new software designs too.
[1] https://www.usenix.org/system/files/conference/fast16/fast16...
Why not explain your project to me first? Assume I know nothing about the project and followed a link from HN.
"TerarkDB is a RocksDB replacement"
doesn't help me, what is RocksDB?
This is a huge missed chance for projects to get new users. Start every release note with a sentence explaining your project. Assume people reading your release notes are non-users.
There's probably a mutually beneficial filter being applied here by not letting beginners stumble in.
So things I don't know exactly about are not for me? That seriosly hinders my personal development.
As a two decade CTO I think I should be interested in things I don't know about.
I'm saying that unless you've used and hit the limits of RocksDB - and it's already absurdly fast - there's zero reason to utilize this project.
Maybe it'll mature one day, have multiplatform support and a wide array of client libraries, and be to RocksDB what RocksDB was to LevelDB. But today is not that day.
For now, developers that don't immediately understand what this project is for would best be served with a simple link to RocksDB.
You first try rocksdb/lmdb, learn it all, break things, hit limitations, and lower your standards enough to search for other things, that don't have comprehensive documentation but just a small readme/paper and checking the code.
Don't expect to build a better Postgresql on the first try.
You wrote the initial post saying they're hurting themselves by not being more clear. I don't think that's true.
Now you're saying they should be more clear to help beginners in the field. I completely agree.
It had several points where the first point was about me as someone reading dozens of HN posts a day following links and needing to go on a googling spree to find out about a product I might or might not be interested in.
The other point was about the missed opportunity to attract users.
And even the second point used "projects" and talked about relase pages which made it clear that the it's about the generality of the problem (the linked page wasn't even a release page) Picking out the specifics you either haven't read the post or are misresenting it intentionally.
In an organization, it's easy to recognize consumers -- they typically say things like: "I don't understand this. Is there a training course for this that I can sign up for?" and expect to be assigned to an internal training session or to some external course.
An autodidact on the other hand goes: "I don't understand this. Let me do some research on my own and try to teach myself."
I've been both at various junctures in my life but I've learned that in order to progress to higher levels, it's better to be an autodidact instead of a consumer. When it comes to new knowledge, there's rarely someone who will feed it to me -- I have to take the initiative to learn it myself.
There's nothing wrong in asking for a clarifying blurb (good marketing aims to make things frictionless for potential customers). But RocksDB is its own universe and it's actually pretty well known. I don't work in this space, and even I know what RocksDB is because it has come up a lot in technical discussions about storage engines. When I first encountered it, I had no idea what it was, but I gathered from comments that people were excited about it, so I googled "wiki rocksdb". It took 2 seconds.
Truly curious people are autodidacts, not consumers.
p.s. the HN comment section is a great venue to "overhear" what the community is talking about and what they find exciting. It provides a good signal to dive into certain topics. Knowledge acquisition is very much a sociological exercise as much as it is an individual one.
As someone who is very autodidact, I can tell you, that just because I don't understand something doesn't mean I go and learn it. There are far too many things to do and to learn just to go and learn things when I have a reason to. Just like consumers will ask for training sessions when they have a reason to.
This is marketing 101, you have a product and you want people to use it. Even in open source, you still want people to use it, you want there to be value in the thing that you built. Build it and they will come doesn't work.
> There's nothing wrong in asking for a clarifying blurb (good marketing aims to make things frictionless for potential customers). But RocksDB is its own universe and it's actually pretty well known. I don't work in this space, and even I know what RocksDB is because it has come up a lot in technical discussions about storage engines. When I first encountered it, I had no idea what it was, but I gathered from comments that people were excited about it, so I googled "wiki rocksdb". It took 2 seconds.
I have heard of RocksDB before, but I still don't know the use-case, why? Because there are so many other database systems. And even if I was using RocksDB looking at that paper I don't know why as a company I would invest in a rewrite to switch over to this new one, since the performance benefits don't seem massively clear.
> Truly curious people are autodidacts, not consumers.
What I think you think autodidacts are, are people with no focus and spend time researching every new thing that pops up. The sort of people that "Jack of all trades, master of none" is made up to describe.
I'm interested in a many different things, as you I have read about RocksDB before but my time is limited between my family, hobbies and work.