Hacker News Reader

Comment by yrashk | Hacker News Reader

It's a lower-level "database engine" that allows you to build different types of higher level databases based on a very simple foundation:

1) BTree-based K/V engine (which gives you an ability to iterate over lexicographically sorted keys) 2) Strong immutability guarantees (data can not be overwritten) 3) ACID transactions 4) Server-side executable imperative language that gives you a control over querying costs

In a sense, it's as much of a database constructor as different MUMPS systems (GT.M, for example: https://en.wikipedia.org/wiki/GT.M)

PumpkinDB also aims to provide a good set of standard primitives that help building more sophisticated databases, ranging from hashing to JSON support, and more to come.

View on HN

I am a big fan of this approach: Low-level, high-performance, immutable-by-default database engine as a building block, with bindings to multiple languages for easy application development.

TrailDB (http://traildb.io), which has many elements of event sourcing in it, follows this philosophy and it has proven to be pretty successful for its intended use cases.

I was delighted to notice that PumpkinDB has an imperative query language inspired by Forth. We recently open-sourced a similarly imperative query language inspired by AWK, http://github.com/traildb/reel :)

I will definitely follow PumpkinDB with great interest!

Thanks. I read the entire "Documentation" page and still didn't feel confident that I understood what this really was. "Event sourcing" to me implies that it's generating events.

Well, its a fancy word. Thats basically what I took away from it. Also, I agree, read the whole doc and took way longer to understand what it does.

Clearly, they are smart, but bad copywriters :)

Please write better docs.

Doing our best, one step at a time! It's clear to me that we haven't spent enough time on documentation yet. The whole project is only 3 weeks old and our initial focus was to get a working version out with some level of documentation (a goal that we somewhat attained, I believe)

Fair enough. I'm gonna be done this shitty program and have a piece of papers with some fancy seal on it in a couple of months. I'd be happy to contribute.

Bookmarked until then.

Also, if this topic is interesting to anybody who's good at writing, we are very open to contributions of all kinds!

We are definitely looking to improve the documentation. This is 0.1, after all.

This project was started as a backend for a lazy event sourcing approach (https://blog.eventsourcing.com/lazy-event-sourcing-ed7e59007... , https://m.youtube.com/watch?v=aqv8d1pjmU8) and beyond.

The idea behind it is that it provides primitives for building systems that are designed around immutable events, journals, indexing, etc. Hence the current positioning. We thought it would be useful to be targeting fairly narrowly early on.

Either way, we will definitely need to expand on that in our materials

Amazing looking project! I've been daydreaming about using LMDB's memory mapping to provide a flexible low level db primitive for quite a while, figuring a combo of erlang like actors and flexible data scripting would be killer. Needless to say I love the design.

Any thoughts on whether this could be used to implement a Q/kdb+ like computation system? Seems like PumpkinScript could be extended with a library of computational array primitives. (https://news.ycombinator.com/item?id=13481824)

That being said, it'd be great to be able to read how the "actor" system is implemented. The documentation alludes to actors and pub/sub channels. Not sure I can help much at this time, but will keep an eye on it to see!

Event Sourcing is a technical term, it's the idea that instead of mutating your database, you should instead have a log and insert a new log entry saying that you changed this value to that, etc. The idea is that this helps you do cool stuff like temporal queries (i.e. make a query "as the entire database looked like a month ago") or look at historical values and changes of things. This matters a lot in some fields. Of course then there's the matter of how to do this efficiently. You can build event sourcing on top of a regular RDBMS but if there is database-level support (as in PumpkinDB), then maybe some things are more efficient. Read more:

https://martinfowler.com/eaaDev/EventSourcing.html

This sounds like my (probably wrong/incomplete) understanding of Datomic?

I believe you could use Datomic for event sourcing, though it wasn't explicitly designed for the job and may not be the best choice depending on the requirements of your system as a whole. Bobby Calderwood at Capital One has a talk on a system which includes both event sourcing (using Kafka, IIRC) and Datomic:

https://speakerdeck.com/bobbycalderwood/commander-better-dis...

Nice, thank you.

Virtually every db works like that under the hood. They expose it differently, though. Kafka has nothing but log, for example.

e.g. https://dev.mysql.com/doc/refman/5.6/en/binary-log.html

They do but it's usually not exposed in a useful way. Postgres once had this feature... https://www.postgresql.org/docs/6.3/static/c0503.htm

We actually expose something to support event sourcing: https://www.postgresql.org/docs/current/static/logicaldecodi... - although that's very different from the old time travel feature.

Edit: missing word

Is the idea that apps talk to PumpkinDB in order to achieve this layering, or do you see it as a library?

Not often that you see MUMPS referenced on HN, by the way. It's one of those oddly productive niche languages that are (as far as I know) alive and well, but rarely encountered except if you work in that niche, e.g. finance or healthcare.

Some of the layering in terms of basic building blocks and higher level languages will become a standard part of PumpkinDB. For the rest, yes, it's expected that end applications or frameworks will compile higher level constructs to PumpkinScript.

I've been following MUMPS and using it for some ideas for some time and that's how some of their ideas became inspirations for PumpkinDB. As quirky M is, MUMPS was indeed oddly productive and I wanted to piggyback on that.

I hope that PumpkinDB is an alliterative reference to MUMPS :)

It seems to serve approximately the same purpose as Berkeley DB.

Well, that would have been true if we had no PumpkinScript. Then it would have been just a tiny wrapper around LMDB that we are using as a storage backend, indeed.

So basically something similar to FoundationDB?

To a certain degree, yes

From what I see in the published documentation, it's way too early to make it "show HN". If I didn't know what event sourcing is, I couldn't make heads or tails of your daemon, and even then, I don't understand how is user supposed to interact with it in a production-like deployment.

What to Submit

Show HN is for something you've made that other people can play with. HN users can try it out, give you feedback, and ask questions in the thread.

Its never too early to show HN.

Especially for open source projects how do you suggest you onboard people to get this thing progressed?

The voting mechanism helps us determine whether it is interesting anyway.

Reply on news.ycombinator.com