Hacker News Reader

Comment by KenoFischer | Hacker News Reader

Just to set expectations while this package is decently stable and used by a number of people, both commercially and in the open source, it is basically in maintenance mode at this point in time. The intention a few years back was to build a complete analytical database on top of Julia, but have it be a mostly independent project. That vision didn't quite materialize, partly because not enough people wanted to do distributed analytical work (often a single big box is enough, particularly with a language as fast as Julia) and partly because the main developer left to pursue a PhD. I wouldn't be surprised if it got revived in the future, but for the moment, it's basically just the distributed tabular computing part (which is useful if you have that problem, but more limited than the name might imply).

View on HN

> not enough people wanted to do distributed analytical work (often a single big box is enough, particularly with a language as fast as Julia)

As someone who's in the midst of a plan to migrate off of Apache Spark, because we've discovered that, for our particular purposes, a single-machine implementation in a language with better crunching and multithreading chops than Java, can easily achieve comparable performance to a Spark cluster, while being good deal easier to develop and maintain, this comment strikes me as being very poignant.

I have come to suspect that the big data ecosystem is a castle that was largely built on a nice, thick technical foundation that is largely composed of pointer chasing.

I think the problem is that people don't really appreciate how big your data has to be to be Big Data.

To be fair that threshold has risen substantially since the paper out of Google that kicked the whole thing off in 2003.

Well, there is that. But, rewind 10 or 15 years to when Big Data was really taking off, and the world was also different in ways that meant that your data didn't need to be nearly so big, even after adjusting for hardware advances, to qualify as Big.

Most the programming platforms and libraries that one might choose instead of Hadoop & friends didn't exist back then, or weren't or weren't particularly robust. So there weren't a lot of games other than Java in town. And Java (idiomatic Java, not benchmark-friendly Java) tends to reduce the volume of work you can squeeze out of a given hardware setup by introducing a hefty does of memory overhead.

Big Data was initially a solution to a problem that basically nobody had except Google. It has remained such.

> It has remained such.

You should take a look at genomic data. The volume of data is one of the main challenges[1].

[1] https://spectrum.ieee.org/computing/software/the-desperate-q...

Omg you are so right!

See diskframe.com a single machine focused framework

I often find data scientist (from python persuasion) come into engineering organizations with this mindset and undo alot of progress. They end up realizing that engineering really does have value and isn't just engineering for engineering's sake and end up layering on a thousand hacks that replicate what was already there. Simplicity for simplicities sake is just as dangerous as complexity for complexities sake. There is a reason major enterprise organizations (LinkedIn, Facebook, etc) have java backends. Also it's not a matter of misappropriation. Some of what they do does apply outside the enterprise. You see a ton of python libraries make this mistake, like dask by saying "it's just pandas that runs on multiple machines". "It's just" is usually a sign that there is some complexity they are ignoring or don't have (that most others will)

Yeah, just look at kdb+/q performance compared to Spark. The former is orders of magnitude more efficient and you realize that a cluster is only necessary because the performance and memory usage of the JVM is very poor on numerical workloads. Same goes for Hadoop.

The amount of money and engineering resources thrown at a problem that actually has a pretty simple solution (integrate your db and language, make your program and language runtime fit in L1 i-cache, optimize the hell out of your language) is a little disheartening. But hey, it keeps people employed managing and debugging an unholy mess of DBs, K8 clusters, load balancers, KV stores, and message brokers..

Sure, but there are 12 people in the world who know how to use k and q efficiently, and there is no fail recovery or any way to deal with data that actually does not fit in ram

That is not true. RAM is cheap (terabytes per node is not uncommon). Moreover, a common pattern for kdb+ databases is to have two parts: a real-time database (rdb) that is typically memory backed and a historical database (hdb) that is serialized to disk (mmap'd for near-RAM speed). In practice, most data fetching is sequential (cache friendly), so you get good performance regardless. kdb+ is commonly used in the financial industry, where data volumes are large.

The language itself is extremely minimal, so the learning curve is actually much lower than most languages (it forces you to write in a "vectorized" style, which is not too different than how numpy/R/matlab works). I was able to get up to speed in a couple of days.

Source: Used to work with kdb+ (still do sometimes). Not a shill (I don't even like k/j/q/APL).

I am kdb user, I am not arguing against it. More power to you if you can afford to endlessly scale kdb vertically. Rest of the world have to deal with distributed systems, because people do not want to pay for kdb or the machines to run it on

kdb+/q mmaps everything, so the difference between in ram and on disk is immaterial. You're right that not that many people know k or q, but they're pretty easy to learn and a lot as fun as well!

Also you can set up clusters using a very nice IPC system. You can seamlessly just send over code and data with very little fuss.

Maybe the solution then is to train more people to use kdb, not to build new languages/frameworks.

I gave a talk at some kx/kdb usergroups[2], and definitely saw more than 12 people there. There were hundreds.

Its not a trivial language to learn, you have to be ok with very terse languages in general, and ok using the language as a way to mentally model the algorithm. Somewhat "mathematically".

I'm not using/working with it any more, haven't in a few years. Most of my recent work is in Julia, by choice.

Two very different philosophies at work here. Julia provides a sane, more or less classical language to work with, inclusive of numerous macros and modules to help drive productivity. k/q provides a fairly modern apl like experience, leveraging Iverson's idea of a programming language being a "tool for thought".

FWIW, the forces behind k/q (Arthur Whitney, et al) are now working on Shakti[1]. Looks like an interesting project, more of a platform than kdb was.

Bringing this back to JuliaDB, I had been hoping for a very julia centric/idiomatic persistence engine as well as an analytical engine. I'm no fan of SQL or its variations. In reading this, I've learned DataFrames is supposed to be this. However, my impression working with DataFrames has been that it is woefully slow on small data sets (100-ish rows). It cannot handle one of the slightly larger 50k line tabular data sets that I am importing via excel sheets.

For that work, I may need to resort to Perl (my fall back programming/data munging language). This is important, as Perl has a capability of tieing (binding) a variable to an underlying data structure/file/... . This enables trivial persistence, in a completely idiomatic manner (update/insert into the tied variable by setting a value). It would be interesting if this capability existed within julia as well.

[1] https://shakti.com/

[2] https://kx.com/media/2018/03/joe_preview.jpeg (actual image of me talking)

My impression hasn't been that DataFrame are slow at all. What do you mean it can't handle 50k rows? Like it takes a long time to load or what?

Because right now I'm using a DataFrame that has 7m rows and 50 columns and it works very well.

Hmm ... could be some of the other tooling around it. Right now I can't get XLSX to talk reliably to DataFrame. I rigged up some tests for 10, 100, 1000, ... points and performance above about 500 points fell off a cliff.

I should look into this more. Thanks for letting me know it can handle things that size.

I dunno man, I think Kx deals with out of memory data in the most sane way imaginable if you assume time oriented problems and understand how computers work (aka latencies in squirting the data across different layers of hardware into the CPU). I have no idea what they're giving people for manuals for setting up for different query patterns these days, but the old K3 manuals were great; classics of database engineering and practical computer science. Petabytes before it was routine.

The APL family takes some getting used to, as it's a significantly different paradigm from Julia and Python looking languages: arguably as different a paradigm as Lisp. It's also fairly natural if you've done any work with Matlab, NumPy or other array oriented packages, but you do have to put in the work.

It doesn't get as much attention here, but Jd is pretty good also. It's also much lower barrier to entry than Kx if you're building a personal project with it (aka its free for personal use). J is a more complex language than K, but I like it better; more oriented towards mathematics.

https://code.jsoftware.com/wiki/Jd/Overview

if your use case lets 'your program and language runtime fit in L1 i-cache', it's not a big data use case - engineering for big data tools is not the problem, deciding this is the 'problem' is.

Spark runs pretty nicely in single-machine mode, and it forces you to structure your logic as a nice clean map-reduce. So I think it's fine to use it for "small" data.

Have you seen Frank McSherry's COST paper? (and follow ups) http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...

Really fun reading on exactly this issue - definitely changed my view on Spark and compute clusters. People definitely don't realise just how powerful a modern computer is, and how far you can get on a single one.

Distributed computing for data science not only benefits when we go big, but also when we need go small with independent nodes.

e.g. Cluster of scalable ARM SBC nodes or even multi-architectural nodes could be highly efficient.

I've been exploring this kind of setup for a while now, frameworks like Dask, Ray[1], Modin can do this with Python to some extent. But they are still finicky and Dask seemed more stable than other frameworks for this setup.

I wanted to try out language level distributed computing setup, but Julia required same setup (OS/Arch/Path) replicated on all their nodes last time I visited it.

I feel, distributed computing as such hasn't got much love as it deserves in the consumer market, especially since many have several computing units in their houses now(PC, tablet, smartphone, Watch, TV, Game console). If interoperable distributed computing layer was fundamentally baked in with all modern operating systems minimising the latency with Network, Storage, Memory; Then we could leverage huge compute power on demand, which not possible without investing lot money in single compute unit nowadays.

Then again, Compute power is a major strategic advantage for the manufacturers. Why would Apple share its A13X for compute with a Snapdragon or Intel?

[1]https://gist.github.com/heavyinfo/aa0bf2feb02aedb3b38eef203b...

Good point. I used Spark and DataFrames at my last job just because the data I needed was conveniently available in that environment. Using a high-memory single server seems better on general principles, until your data won't fit.

I call this the: "large-but-not-really-all-that-big data"

For if you can fit it in RAM on a high-end machine, and on a single harddrive in a low-end machine.

and its practically a more useful case. Since if data is truely big many algorthms can't be used (since if it is O(n^2) its not practical).

I would add that while JuliaDB is really great for some applications, the data ecosystem is very much coalescing around DataFrames, which has recently caught up in performance with JuliaDB and has a broader more flexible API. (I'm one of the original authors of JuliaDB, so this is not a knock on the package at all.)

Any plans for DataFrames to support larger-than-memory data though? JuliaDB is super useful for this I've found, basically a solid Julia alternative to Dask.

My current thinking is that it would make sense to implement the standard DataFrame abstraction (with all the generic functionality it offers) with a couple kinds of distributed data frame. One approach is to make a distributed stack of local data frames; another is to make a single data frame where each column is a distributed vector. Not sure which is better. Fortunately, Julia makes it really easy to experiment with these kinds of things. Of course, as soon as you have a system like that you need a scheduler for work at which point you want something like Dask or Dagger (https://github.com/JuliaParallel/Dagger.jl).

We are so strong with the more general Tables.jl abstraction. Tables.jl took us a step away from having lots of types of `AbstractDataFrame`. Subtyping `AbstractDataFrame` is hard, I have to deal with two packages that do it and it is a big and not entirely documented interface.

More natural is extending Tables.jl (like DataFrames and JuliaDB does). and we continue to build more tools that are table agnostic, and have APIs that DataFrames and a distributed table package special case when they can do it more efficienctly

The basic Dask-like functionality is in Dagger.jl, we have plans to move out the distributed array functionality out of it and keep just the scheduler in there.

I'm working on a set of functions to work with many DataFrames (or anything) in parallel and do it out of core if possible, it's basically like JuliaDB but with a FileTree abstraction rather than a table abstraction.

http://shashi.biz/FileTrees.jl/lazy-parallel/

Would Query.jl[1] be an alternative to consider?

Their README says:

The package currently provides working implementations for in-memory data sources, but will eventually be able to translate queries into e.g. SQL. There is a prototype implementation of such a "query provider" for SQLite in the package, but it is experimental at this point and only works for a very small subset of queries.

Still early days, but sounds like they're working on the same problem.

[1] https://github.com/queryverse/Query.jl

This is really sad. kdb+ is still really popular in the financial industry (with very few if any competitive alternatives, open source or otherwise). Julia has the potential to breakthrough in this area.

Reply on news.ycombinator.com