Crate: Elastic Data Store | Hacker News Reader

Crate: Elastic Data Store

crate.io

22 pointspanarky15 comments

Have anybody any experience with this DB they would like to share here?

I took a spin a few weeks ago, it seemed like it would be a great fit to our needs: - we store alot of metrics for custom KPIs - we don't do joins - we don't need transactions - we're write once read heavy, sometimes re-write - crate's sharding, partitioning and replication, as well as "sql" api is great for our use case

I even implemented some custom aggregation functions to check out the code, and it was easy. But what I liked the most was the responsiveness of the devs, the project seems to be moving at a good pace and the devs were very helpful over github.

I wished the site made clear that it's using ElasticSearch under the hood, though. I don't know a thing about ES so I can't really comment on how the sharding, partitioning and replication are achieved. What I can remember from aphyr's blog (jepsen series) is that it's not recommended as a primary database though.

Previous discussion on HN, 6 months ago: https://news.ycombinator.com/item?id=7609323

OT: the domain name crate.io confused me into thinking this was about Cargo, rust's package manager, which is documented on crates.io

This domain used to be Python's new package manager as well.

It's still live under /packages/: https://crate.io/packages/requests/

They mention that here: https://crate.io/thanks-dstufft/

This confused me greatly as well...

crate.io is older then crates.io, though.

Looks great, really. I'm pretty tired of Postgresql all I want is a simple modern data store that can handle arrays and blobs. Relationships and data validation is all handled at the application level for me. I'm very interested to try this out as soon as there is an implementation for active record.

Thanks for the heads up.

> all I want is a simple modern data store that can handle arrays and blobs

I'm curious in what way Postgresql doesn't fit that, for you. Is it not "simple"?

It has a lot of features that aren't needed in modern application development. I just need indexible columns. Arrays were added later so you have to include a module, I mean maybe it's just a feeling but it all feels dated.

More importantly it isn't elastic. Constructing a heavy load or large database server cluster is difficult. More complicated than it should be. If there is on offer an elastic solution with simple arrays built in, I'm all over it right now.

Blob support in postgresql is seen as something you shouldn't do as a best practice. If it's a data store then what I want is to be able to use it to store data easily without a fuss. Postgresql is the king right now and I use it but I'm very open to looking at alternatives.

Your last two paragraphs make a bunch of sense - clustering should be easier and blobs definitely aren't the "happy path".

I don't relate much to the first paragraph though. You don't have to include a module to use arrays. I'm mystified by the implication that data integrity, normalization, and ad-hoc querying are useless for "modern application development".

That stuff is handled at the application level a data store should just be there to store and retrieve data, I think databases used to need to do a lot more than they do now. I thought Postgres required one to include a module in order to gain array support.

Looking into it again now it seems I was confusing Arrays for Hashes, with regard to the hStore extension.

The application is in charge of what data goes into the database. It is performing sanitisation, validation, returning errors, all of that before persisting data. I don't want to have to deal with errors from the database nor do I want to have to manage data integrity from two separate locations.

I want to be able to tell the database to store something and it stores it modern development frameworks can handle data integrity stuff.

"modern development frameworks can handle data integrity stuff". Wrong. They cant. If you have 2+ applications working with the same db, there will be problems with data integrity that no "modern development framework" can solve.

If you have two applications making use of the same database, then build a shared ORM layer and have the applications interact with that. Modern applications do not interact with the database directly there is an ORM.

Separate that out and share it between however many applications you want you should be able to swap out data stores not be confined to them.

Your solutions to these problems (like "do validation and integrity in the application layer" and "build a shared ORM layer for multiple applications") are definitely possible solutions to the problems, but you state them as if they are obviously the best solutions. They actually sound a lot harder to me and seem to be driven by some theoretical purity in having a dumb data store that I don't really understand. Building a shared ORM layer for multiple applications to use is just a hack around having a shared data store layer doing validation and integrity, which is what a smarter database already does. It's reinventing one of the really hard things that databases are already good at. What is the advantage?

I've also never made an application that doesn't eventually want to do something that the ORM I'm using doesn't know how to do, at which point it makes a lot more sense to interact directly with the database than to shrug my shoulders and say "I guess I can't do that".

Maybe my applications just aren't "modern" enough.

Reply on news.ycombinator.com