Hacker News Reader

Hacker News

Comment by aboodman | Hacker News Reader

Full thread

aboodman

Hi all. I'm one of the creators of Noms. Happy to answer any questions!

View on HN

Are you aware of https://github.com/bup/bup? It's a git file format based backup incremental backup tool.

Do you also use rolling checksums (like bup) to prevent re-storing data when only a few bytes have changed?

--> What are the major use cases you imagine for noms?

I've read the section on Github, but can you give some specific examples that you see as good use cases?

Yes, we credited bup in various places, such as the design overview:

https://github.com/attic-labs/noms/blob/master/doc/intro.md

We were also heavily influenced by camlistore (which I hacked on for awhile), irmin, ipfs, and others who have done a lot of interesting work in this space.

We do use rolling checksums, but I think we have done some novel work here: https://github.com/attic-labs/noms/blob/master/doc/intro.md#...

FYI, the "HTTP protocol" link[1] in the intro page above is broken.

PS: I love the name "Prolly Tree".

[1] <https://github.com/attic-labs/noms/blob/master/datas/databas...

:) Thanks, we like it too.

bup is focused on backups. Would you say that noms is useful as a backup utility?

Do you have any mechanisms for securing integrity, specifically repairing the store in case of inconsistencies?

Is there any plans to support any data retention policy/functionality?

Noms should be useful as a backup utility, but I'd say it's especially useful for backing up data which is not files. Think about backing up data which you only have access to via API.

You can take the JSON output of an API and drop it into Noms, then do the same thing tomorrow, and Noms will automatically deduplicate the data as well as give you a nice structured API to read and interact with it.

We have an example of this here: https://github.com/attic-labs/noms/tree/master/samples/js/fl... but it's not working atm due to a bug introduced right before launch. You can look at the code though.

Another question to help me better understand the tool: who do you see as your competitors, technically? What do you see as viable alternatives to Noms, but worse? (Or better!) I do see that you were inspired by git, but clearly your use-cases are different.

Git is a competitor. It is fairly common to check data (e.g., csv or json files) into Git today.

However, this falls down pretty rapidly. In order to get reasonable diffs, the data has to be sorted, and line-oriented. Also Git just doesn't scale well to larger repos or individual objects.

Otherwise, we see the competitors as the way that people distribute data today - custom APIs, zip files full of CSV, etc.

How would you say Noms compares to Datomic[1]? Both projects are working on the same idea of representing a database as tree of commits over time.

From my quick inspection, it looks like Noms shows some focus towards working in multiple branches, whereas Datomic, at least in its marketing materials, just talks about preserving a single timeline.

[1]: http://www.datomic.com/benefits.html

https://news.ycombinator.com/item?id=12213886

> Also Git just doesn't scale well to larger repos or individual objects.

I guess the above means that Noms does scale to larger repos... Do you guys have any numbers, comparisons, benchmarks against git?

If so, it would be useful to include in the readme.md as it would be kind of a big thing, and quite attractive to many people.

Why would you exclusively support schema inference, rather than also allowing users to manually specify their schemas?

Schema inference is very difficult to do correctly and safely, especially with small initial samples of instances (source: work on https://github.com/snowplow/schema-guru).

There's a little bit of terminology overloading going on here.

In Noms every value has a type. It's an immutable system, so this type just is. The type of `42` is `Number`. The type of `"foobar"` is `String`. The type of `[42,44]` is `List<Number>`. And if you add "foo" to that list, the type becomes `List<Number|String>`.

We don't try to infer a general database schema from a few instances of data. We just apply this aggregation up the tree and report the result.

That all said, we do want to eventually add schema _validation_, by which I mean the ability to associate a type with a dataset and have the database enforce that any value committed to the dataset is compatible with that type (following subtyping rules).

It sounds like Noms is dynamically typed, rather like SQLite — types are associated with values, not (just) with datasets. The difference is that SQLite (like Python or JS) only types leaf/atomic data, while you're also typing aggregate data. Is that right?

Are you planning on writing complete reference documentation at some point, like https://www.sqlite.org/limits.html, https://www.sqlite.org/howtocorrupt.html, https://www.sqlite.org/lang.html, https://docs.python.org/2/reference/index.html, and https://golang.org/ref/spec? Or is using Noms going to be more of a UTSL kind of thing? The documentation I've found so far seems to be purely tutorial and introductory in nature.

(I'm really glad you're writing Noms, by the way. There's an enormous need for it.)

Right - the challenge is that with dynamic typing and without schema validation, it's incredibly easy to break any strongly typed client/consuming application. You think you are dealing with a `List<Number>`, you have Go/Java/Haskell/whatever apps which are consuming that in a strongly typed fashion using their idiomatic record types, and then suddenly a user accidentally sends in a single value which turns the tree of values into a `List<Number|String>`, and all your consuming apps break.

Given that schema validation ("does this instance match this type?") is simpler to implement than schema inference ("what is the type of this instance?"), it's surprising to me to deliver inference first...

i don't understand how we could have gone in the opposite direction.

Schema validation for us is just looking at the type requirements of the dataset and the type of the value and seeing if they are compatible. How can we do that without first knowing the type of the value?

How does Noms compare to ipfs?

https://ipfs.io/

First, IPFS is awesome.

IPFS is essentially providing a globally decentralized filesystem. Noms is providing (or hopes to provide) a database.

By database, I mean:

  - small individual records
  - efficient queries, updates, and range scans
  - ability to support complex queries
  - ability to enforce structural data validity

These are all things that IPFS could eventually grow to support, but in order to do it, I think it would have to grow into or layer something like noms on top.

Hi, is there any place where you talk about how merging of two different branches works?

How would NOMS work for massive amounts of geo-temporal data? with many inserts and queries but few (zero) updates? Efficient queries on geo-temporal keys are useful.

There's a question about what you consider "large data" above.

does noms understand sql and can it do joins?

The linked page says, as its last line, "many important features are not yet implemented including a query system."

It doesn't support queries. It's a datastore, not a database. This is from their FAQ. https://github.com/attic-labs/noms/blob/master/doc/faq.md

I mean, by their definition it is a database, but i can understand your usage. Then again, they both say it is a database, and in your link, they say it "isn't quite there yet", so /shrug heh.

It'd be fantastic to combine something like noms with https://arrow.apache.org/ and then use Spark/Drill/Impala to query it.

You probably want to prepend a "Show HN: "

Whoops -- @ahl says it's too late now.

perhaps dang will change it for you?

Added.

Reply on news.ycombinator.com