Hacker News Reader

Comment by Tarrosion | Hacker News Reader

I wish this website spent its banner space on "what it is" rather than "star us on GitHub!". I doubly wish this because it's pretty confusing what JuliaDB is! From the name, I expect it to be a database, but the website immediately compares it to Pandas. I think of Pandas as a library for in-memory manipulation and analysis of tabular data, not a database / persistence engine. So is JuliaDB not a database at all, but in fact an alternative to Pandas / DataFrames.jl?

In fact, I've been using Julia for work and following the ecosystem since version 0.4 (we're at 1.5 now), and I'm still not sure what JuliaDB is. No doubt this is mostly due to me not having reason to look very deeply (and/or not being very perceptive), but certainly doesn't feel like the marketing copy is giving me any help...

View on HN

I think it is actually a database, so it seems they haven't committed the cardinal sin of putting "DB" in the name of something that's not a database.

> JuliaDB is a pure Julia analytical database. It makes loading large datasets and playing with them easy and fast. JuliaDB needs to support a number of features: releational database operations, quickly parsing text files, parallel computing, data storage and compression.

Got this from https://juliadb.org/talk/juliacon2018shashi/ which appears to go into more details

> This talk is a bottom-up look at the construction of JuliaDB. We will talk about the scope and implementation of underlying building block packages, namely IndexedTables, TextParse, Dagger, OnlineStats and PooledArrays.

> they haven't committed the cardinal sin of putting "DB" in the name of something that's not a database

It doesn't seem to store data in any way - so it is definitely a data processing engine, but a database without an "INSERT" command feels a little off.

From that point of view, this looks a lot like what original MapReduce did - the data lives outside it & is referenced as urls, but the engine itself does processing out-of-core and in-memory for very large datasets.

"It doesn't seem to store data" is a complaint that doesn't really make sense to lodge against a library rather than a standalone program. If your program is using a database library, it's your job to write the line of code that tells the library to load data from a particular file into memory. The library cannot persist as a running process of its own across multiple executions of your program that is using the library. This is as true of SQLite as it is of JuliaDB.

I agree that it's a bit odd to not have a direct analog of SQL's INSERT, but you can definitely add rows to an existing table by making a new one and doing a merge operation.

What do you mean by storing? JuliaDB tables can be stored in memory or disk (mmap-able).

There's a better description on the front page of the docs: https://juliadata.github.io/JuliaDB.jl/latest/

Ok, we've switched to that from https://juliadb.org/. Thanks!

Side note: this is what we tell every startup about talking to HN: Lead with a clear statement of what your company does. If you don't, the discussion will consist of "I can't tell what your company does". Same for open source projects of course.

Looks like they give a short overview when you follow "Read the docs" (just under the "Star us" ;)

That's too far from being a first impression.

I came here only to find and upvote this comment

Not every database is necessarily a persistence engine

Can someone explain the difference between JuliaDB and Apache Arrow? The latter has just reached stable 1.0 version last month.

Arrow is a file format, albeit one designed to allow easy interoperation of different analytical frameworks, but it doesn't provide any computation functionality itself.

JuliaDB is more like Dask, or DataFrames.jl - it provides that computation functionality.

Reply on news.ycombinator.com