Hacker News Reader

Comment by jb1991 | Hacker News Reader

jb1991

Man, am I ever a dummy! I look at the diagram on that page and it might as well be Egyptian hieroglyphs to me. I'd have to study that thing one long time to even understand what problem Ions is solving. I'm sure it's a real problem with an interesting solution, but geez the modern Cloud world has really gotten quite complex.

I feel particularly silly because I've been writing Clojure professionally for many years.

View on HN

This is about Datomic, not really Clojure. So if you don't have experience with Datomic Clojure understanding would not really help you.

Seems to me this is a combination of Datomic and Amazon Lambda to deploy programs that can use Datomic without all surrounding deployment problems.

>Ions let you develop applications for the cloud by deploying your code to a running Datomic cluster

I could be totally off base here as I have no real experience with clojure, but this intro sentence threw me off right away.

>Ions let you develop applications for the cloud by deploying your code to a running Datomic cluster

What is an ion? What is a "datomic cluster"? Are these terms just new jargon for what is essentially a deployment system? If they're a clojure thing then ok, but a quick search makes me think they're not. I understand software and I understand deployment. I read the rest of the page and I feel like I would have to problem understanding what this really is if they cut down on the jargon.

Datomic is an immutable database, written in Clojure. It's pretty cool, you should check it out.

The company behind Datomic (Cognitect, which also develops the Clojure language) recently started offering the database as a cloud service. This was something Datomic users had wanted for a long time, but one problem with the new service was that one of Datomic's primary features, i.e. having the database reside in memory with your application, was missing.

Ions is a new service that addresses this problem by allowing you to deploy your app to the same VMs that are hosting Datomic in the cloud (a "datomic cluster"). It's basically PaaS for Datomic-backed applications.

I'm a longtime Clojure user and I had some trouble understanding all this as well, so I agree that could do a better job with presentation.

> one of Datomic's primary features, i.e. having the database reside in memory with your application

I'm not sure I follow - are you saying it is desired to have an entire database also in memory? You don't probably mean that, but I thought a key feature of databases is for them to be your offline storage so things don't need to be in memory. I wouldn't be able to fit the contents of my SQL databases into memory on any machine I own.

Datomic keeps your database in the same memory space as your application code, backed by SSD, backed by EFS, backed by S3/DynamoDB. Each layer of this cache provides a different value, and the whole cache system is automatic and eternally consistent because the database is an immutable value.

You want the illusion that your database is entirely in memory, just as want the illusion that all your memory is in your L1 cache.

> Each layer of this cache provides a different value

What does that mean?

And what kind of actual memory usage should people expect with a "big" database?

Hm, I should have said "value proposition", e.g. SSDs are low latency but can fail, S3 is high latency but "never" fails, etc.

Datomic is designed for data sets larger than memory, and also to do a good job caching working sets that do fit in memory.

> Datomic is designed for data sets larger than memory

Of course. But you didn't answer my question.

Let's say someone has an application with 100GB of data in Datomic.

What kind of memory usage should he expect for the "peer" process?

That depends on what queries the peer runs. In both On-Prem and Cloud, Datomic maintains an in-memory LRU object cache. Also, you can send queries with different working sets to different processes without having to shard data on ingest. For example, some peers might handle queries related to user transactions while other peers handle analytic or batch work.

> You want the illusion that your database is entirely in memory

I don't want illusions. When it comes to databases I want simple and predictable.

Datomic basically does the same thing your database does - caching the working set in memory and running the queries. The difference is that with datomic the db is running in your java process rather than on another server. This should get you better scalability (for reads) and make complex queries faster. (Writes still go through a designated server.)

The data is modeled as a collection of (entity,property,values,txn) tuples (with schema on property). They are stored in a log and a few ordered sets (indexes) that are structured somewhat like 2-layer btrees with large pages. These pages and the tail of the log are loaded as needed while running queries. The nature of the data model also gets you point in time queries and queries across time (e.g. all previous values of a property).

There is some cleverness going on behind the scenes, so for a large database you wouldn't necessarily have the whole thing in memory at once. But one of the big selling points of Datomic is that you get to treat the entire database as a value, and when you configure your app as a Datomic Peer (the feature I referred to above), it loads most of the Database into memory as it is needed.

I think the idea of databases being on-disk (vs in-memory) is getting quite outdated. You want durability of course, but most datasets can fit in memory these days.

(Which is natural if you consider the retarded development of disk/ram vs processing power; cf the memes of "disk is the new tape" "ram is the new disk")

> Datomic is an immutable database, written in Clojure.

Ah, that helps (I guess my search was a bit _too_ quick). So Ion is really the only jargon there. I'm not even going to delve deeper as I just don't know the ecosystem at all. I was hesitant to even comment in the first place. Thanks.

They should get you to write their website copy. Thank you for the summary/translation.

Ion is a name as far as I know.

Datomic is a database consists of multiple services so you can deploy it on one machine or multiple. I called it a cluster because on AWS of course the idea is to deploy it on multiple.

Amazon Lambda is a way to deploy code without telling Amazon about the underlying machines/VMs. Ion is a way to deploy Clojure and Datomic without knowing anything about the underlying machines or VMs.

Ions is the name of a service. Datomic is a database. These aren't jargon.

Are you also bewildered by "Postgres cluster" or "MongoDB cluster? Why isn't it your responsibility to spend the 10 seconds to see that Datomic is the name of another database?

In addition to the linked page there's also a tutorial https://docs.datomic.com/cloud/ions/ions-tutorial.html

and a reference https://docs.datomic.com/cloud/ions/ions-reference.html

There are always two sides to this... I think it’s badly communicated. Someone said (I don’t find the quote), that if you can’t explain a complex topic in simple words, you haven’t understood it thoroughly.

Einstein said this, but he also said: Everything Should Be Made as Simple as Possible, But Not Simpler.

If you know Clojure and AWS I really don’t think it is poorly communicated. There’s a limit to that idiom of simplicity and understanding being correlated.

This was a walk through of the mechanics. The TL;DR is if you are comfortable enough in AWS and Clojure implementing your system in a scalable and simple and wholistic way just got a bunch more accessibile.

I've written a clojure app, used aws and lambda functions and skimming this I really wasn't sure wtf they were talking about. Datomic... Something something lambda functions, deployment. I know that they have an existing ami that uses a cloudformation template to let you set up a datomic server on AWS, this is something similar maybe? I think I could get it if I read in detail but this is definitely poorly communicated, it reads like old Microsoft enterprise sales copy.

Reply on news.ycombinator.com