TidesDB – High-performance, durable, transactional embedded storage engine | Hacker News Reader

TidesDB – High-performance, durable, transactional embedded storage engine

github.com

4 pointsalexpadula9 comments

Hello my fellow hackernews community. I hope you are all doing well.

I'm Alex a super passionate and driven engineer who loves building database systems and data structures. I've created a new open-source database engine called TidesDB. TidesDB is a log structured merge tree based storage built for high-performance written completely in C++. Still in it's early stages but moving quite rapidly.

Some features:

- Embeddable storage engine

- Variable-length byte array keys and values

- Simple API (Put, Get, Delete)

- Range functionality (NGet, Range, NRange, GreaterThan, LessThan, GreaterThanEq, LessThanEq)

- Custom pager for SSTables and WAL

- LSM-Tree data structure implementation (log structured merge tree)

- Write-ahead logging (WAL queue for faster writes)

- Recovery/Replay WAL (RunRecoveredOperations)

- In-memory lockfree skip list (memtable)

- Transaction control (BeginTransaction, CommitTransaction, RollbackTransaction) on failed commit the transaction is automatically rolled back

- Concurrent safe

- Tombstone deletion

- Minimal blocking on flushing, and compaction operations

- Background memtable flushing

- Background paired multithreaded compaction

- Configurable options

Do let me know your thoughts, advise, I'd love to hear what you think.

Thank you.

That is an amazing featureset for less than 4500 lines of code. Can you describe how you did that?

I appreciate that comment! I'd say mainly practice. Coming up with the design before writing what you plan. Once you have what you had in your mind or on paper you can then optimize and cutdown the fat. I've always done this. :)

It's pretty interesting how this design turned out, RocksDB for example I believe is well over 200,000 lines. I originally wrote it in GO, well the base of it:

https://github.com/guycipher/lsmt

Then once I felt ok with what I had I took what I had in GO and painstakingly rewrote it to C++, (originally was gonna be C). Why GO first? I honestly write a lot more GO than C++ lately so I'm faster at writing in GO. Not to say I don't write C++. I've been writing code for 15+ years, C++ was one of my first languages at 14 years old, just haven't been writing it everyday. This project has changed that pretty dramatically!

Similar! I learned C and C++ when I was 16, a very long time ago.

I'll definitely be stepping through the code later. It has been over a decade since I did any C++, Rust is better fit for what I need now.

I have used a GCd language in a similar manner on two projects, where I wrote version 1 in Python and then rewrote the production version in Java and for another project in C. If you know that the final project is going to be in the target language, you can kinda code in that style from the beginning. We coded in Java-lite Python and the translation was mostly mechanical, used the same integration tests.

Version 1 is for figuring the problem out.

Version 2 is for shipping it in production.

That's awesome. I'd appreciate that honestly. I tried to keep it very approachable.

See Go is a procedural language like C, that's why I was gonna rewrite it in C once I had my design down. I felt like C++ was the best choice and what I wanted to truly continue with so I bit the bullet and committed with C++. I'm glad I did. As like you it's been awhile since I wrote C++ everyday or read it everyday but I do enjoy it. Very broad language, many ways to do things so, lots of research involved.

Very cool! I'll try it out and I'll be watching its progress.

Hey david927, thank you for the kind words! Do let me know how your experience goes. Oh and there's lots more to come :)

Happy to hear that, and yes, will do!

Reply on news.ycombinator.com