Hacker News Reader

Comment by bontaq | Hacker News Reader

I'm not sure it's still true, but awhile ago their EULA banned the publication of performance statistics. Be very careful using this and do your own performance testing.

View on HN

Still true based on what I see:

Restrictions:

The Licensee hereby agrees, without the prior written consent of Cognitect, which may be withheld or conditioned at Cognitect’s sole discretion, it will not:... j) publicly display or communicate the results of internal performance testing or other benchmarking or performance evaluation of the Software;

This is a similar style I believe to the Oracle licensing model which used to prevent similar stuff I think.

Building your stuff on a platform (proprietary) with this attitude towards licensing... I'd say run away... fast.

Performance testing is a fickle art and publicizing misleading or non-generalizable results can be extremely damaging to a small business like this, even if the results are not representative nor applicable to most people.

Then Datomic should at least release their own, properly done and fair, benchmarks.

You say that like it's an easy thing to do.

For a relatively novel data model with potentially unusual performance characteristics, the chance of finding a properly done, fair, and generally understandable benchmark is virtually impossible.

I think the model from the user point of view has been seen many times over the years.

See eg https://en.wikipedia.org/wiki/List_of_SPARQL_implementations

https://en.wikipedia.org/wiki/Datalog#Systems_implementing_D...

Also the various RDF things from the days of semantic web enthusiasm.

For active competitors, maybe the various GraphQL things and

Then they should constructively respond to these misleading results not suppress publication with legal measures.

A damaged reputation is hard to repair, if even possible.

As a small business, the choice seems to be pretty straightforward--pay multiple salaries to do PR, damage control, reputation management, etc. (and compete with huge corporations with deep pockets)...or pay those same salaries to engineers to build a better product for your customers, and get your customers to promise not to create PR headaches for you.

That seems like a very negative way to approach benchmarks. It could also be a PR boon--out of nowhere, the company could get positive attention for someone else doing work on the company's behalf. Assuming the technology is performance competitive. (But I don't know if it is or isn't--I've not seen any benchmarks).

Tech is a ruthless meritocracy, because people face challenges constantly that exceed their capabilities.

If something sucks, they will shout it to the skies. If it performs better or saves their ass on a key feature, they will shout it to the skies.

Let those shouts echo, and the equilibrium is how we decide on what's next.

Censorship is almost always a bad idea. Furthermore, you shouldn't chill the ability of people to talk about a platform if you eat your own dogfood and believe in it.

A bad benchmark is something everyone can understand and use as criteria to disqualify a technology.

A license that prohibits a public benchmark is something only a few people care about.

Having seen hordes of developers on IRC, Reddit, and HN develop outdated gut reactions that they still invoke years after the fact has led me to believe that benchmark prohibition is easily worth it.

This is absurd. All databases, at some point in time, run into scaling and performance issues. Documentation by blogging and performance metrics are the only guiding light of sanity we have when these hard problems hit your app.

Having used products that bragged about their performance benchmarks (won't name any names), I respect the Datomic team's position on this. Benchmarks lead to gamed results and you still have to test with your workload.

Indeed, I just think the ban on it makes certain things harder to find before you're using it in a production system (with proper production loads). Pagination and sorting, for example, do not exist in the usual sense.

https://groups.google.com/forum/#!topic/datomic/gr9fscuF6oE

https://groups.google.com/forum/#!topic/datomic/NgVviV9Sw8g

It'd be swell if they put up a list of differences, because using the pull query API + clojure really is quite cool -- it's just good to know what you're getting into.

Our team signed a contract with Cognitect for Datomic late 2017.

This clause was in-place and stood out to me as well. I had a chance to ask their legal team about it. The clause is written in legal-ese, which always sounds overbearing.

I asked the question in the positive sense, "what if have some really nice metrics from our use cases, and want to talk about them at a conference?" They simply asked to be consulted and request written permission to share. The intent, like others have noted, is to request (legally: insist) that Cognitect have a chance to review and point out potential implementation issues (good or bad) prior to customers making performance statements about their product.

The clause can/does put a damper on 'notes from the field' reports, which often help when deciding on tech direction.I look for community based reports to reinforce perceptions of a tool (to a degree). Completely agree with OP, do your own performance testing.

One thing I will say is that it would be hard for someone who hasn't invested in learning the inner-workings of Datomic's decoupled architecture to pick apart storage speed vs. transactor speed. For example, storage speed (SQL, DEV, Dynamo, etc.) is not a concern of Datomic, but a key dependency to measurable perf. This may change in the AWS service announced today, and become more uniform on dynamo and S3 "storage resources". https://docs.datomic.com/cloud/whatis/architecture.html#stor...

Datomic is a unique product and there are many ways to make it sing (or blow up) depending on how you use it. We designed data models, streaming processes, and queries with Datomic in mind and have had success. Exactly how much success, I'm not at liberty to say just yet.

https://danluu.com/anon-benchmark/ - "That time Larry Ellison allegedly tried to have a professor fired for benchmarking Oracle" - Looks like other database vendors do the thing. That doesn't excuse the behavior though.

Then people using PostgreSQL remember how their database is free, rather than a $100,000 a year despite having more flexibility and features, and you see why Larry Ellison would do this.

!@#$ Oracle...

My experience with Datomic was that it is not appropriate for a system that has performance requirements. We developed a system using redis while the data team was developing the data service on Datomic. When we tried to integrate, the whole system slowed to a crawl and we needed to do a rewrite.

My advice is to develop with Datomic from the start and not separate it out into a service. Any other database will be so much faster that you won't know how bad your performance is until it's too late.

Another piece of advice would be to seriously consider if you need immutability in your database. If it's not a hard requirement, I would not use Datomic.

Having built a variety of products with Datomic and performance requirements I can only disagree. Especially queries are insanely fast compared to other databases.

I don't know what your data team was doing, but they must have done it wrong.

My advice is to develop without a data team from the start ;)

Datomic is interesting. It is somewhat similar to object databases like ZODB. I use an object database called Durus (spiritual derivative of ZODB). Both ZODB and Datomic make use of aggressive caching in the application. That is a huge performance boost for applications that do a lot of reading and little writing. ZODB essentially puts the indexes in the application, rather than having the DB layer take care of indexes. Again, I understand this is similar to how Datomic works.

Developing an application on top of ZODB vs on top of an SQL database leads to a very different data model design. You can't just swap one database layer out for the other and expect things to work well. I'm not surprised if a team who is used to developing on top of SQL-like DBs did not have good luck when moving to Datomic. It is just a very different model.

would you give an example?

The main advantage is that you can do time-consistent queries against any past database state across the entire database without any read locks. This frees resources and eliminates the necessity of timestamp tables.

Then there is Datomics caching model in combination with its local data based query engine. The vast majority of critical queries only read from memory. Data that is fetched doesn't block other consumers.

What RDBMS are you comparing to pretty much everything is MVCC so writers don't block readers and readers do not block writers mature RDBMS also are fairly highly optimized written in C with critical paths hand written in assembly so again would be interesting to learn for what workload Datomic is significantly faster. The biggest performance penalty in a modern RDBMS is GC of snapshots so if you need a design that is append only it will make modern RDBMS more performant.

I think a big difference is that Datomic can scale out reads in a distributed way. While also keeping writes consistent and transactional.

So its specifically designed for such case where you need consistent transactional writes, while needing high scale reads.

PostgreSQL can do few million reads per second on a high end x86 box. You can scale our reads easily by running multiple slaves.

I don't care so much about optimized tight loops written in assembly, rather about the ability to scale nowadays. Datomic uses databases like you describe e.g. Postgres as its storage. So it would be foolish to say that it beats their performance on a bare bones level. Instead, Datomics architecture and information model make it much easier and significantly reduce the overhead to design and implement applications that provide insane performance. I won't argue that you can hand rewrite every Datomic application in its underlying storage database and get more performance out of it if you do your caching and coordination right. It will take you much longer though (I'd guess a tenfold at least), likely have some very difficult to find bugs, and the result won't be as easy to extend. With Datomic, I get memory speed performance out of the box for the heavy hitters and so much more that it take would take some very uncommon requirements for me to choose something else nowadays.

Well I would buy the developer productivity argument but most applications have a mix of reporting requirements that are generally extremely hard to implement on anything "distributed". Also in a distributed system you are either running some consensus algorithm (paxos, RAFT etc) that will def. not be "insane performance" or you it will have issues with consistency.

Datomic uses distributed storage, writes are coordinated by a single instance (transactor). Reads don't block writers and immutability allows to query consistent snapshots. Does that address your concern?

>> Especially queries are insanely fast compared to other databases.

Nice! Insanely fast? Awesome. What kind of queries compared to what databases?

Interesting!

> not appropriate for a system that has performance requirements

What kind of performance requirements?

Write performance, Query performance, latency? Other?

Where was the bottleneck?

I've found it's very hard for other databases to beat Datomic for query performance.

> Another piece of advice would be to seriously consider if you need immutability in your database.

Remember to check with your audit team, your reporting and analytics teams, your business users, and your data warehousing team.

> ...it is not appropriate for a system that has performance requirements

This is an unreasonably simplistic approach to assessing performance. A useful approach would be to describe what workloads Datomic may or may not be suited for. Having used Datomic (On-Prem) 2 years in production, here's my take on it:

1. For a system that writes a lot (eventually leading to 10 billion datoms or having high peaks of write throughput), Datomic is not a great fit, because it has a single writer thread and does quite a bit of indexing. 2. Reads are horizontally scalable, and because Datomic's semantic allow for pervasive and reliable cacheing, reads can handle a huge load, and have low latency for OLTP-style queries. 3. Datalog is relatively slow for aggregations, and offers no facilities for trading accuracy for speed: if you need big, low-latency aggregations, you should offload it to a specialized store like ElasticSearch. This is especially easy to do with Datomic, because Datomic makes it trivial to implement change detection, in-real time if needed (see Log API and txReportQueue) 4. When writes are overwhelmed and become unavailable, reads stay available, which is an awesome situation to be in.

I recently read that a lot of DBs do this, or at least have done so historically.

Might be like patents - being more of a defense mechanism, rather than an actual device intended to be used against customers.

I found it really fun to use for 3 years in production. But in retrospect I think Postgres is probably a better fit for what we needed.

> But in retrospect I think Postgres is probably a better fit for what we needed.

Would love to hear more about this.

If after three years you think so only "probably", you probably still aren't confident about your requirements.

You don't say to your girlfriend "We had fun the last three years, but probably I'm better off with my ex" unless you have no clue what you are looking for.

I'm using Postgres for a financial app, and the idea of never updating a value in place sounds very appealing over PostgreSQL's construct of updating in place.

I'd love to hear your thoughts about how using Datomic in this immutable sense plays out in practice, because Rich Hickey's talk in favour of it is compelling. https://www.infoq.com/presentations/Datomic-Database-Value

Reply on news.ycombinator.com