Hacker News Reader

Hacker News

Comment by teacup50 | Hacker News Reader

Parent Full thread

teacup50

Why do you find this surprising?

What other language and runtime platform provides:

1) Productive, high-level programming languages.

2) Portability to any OS.

3) High runtime performance for the server-side usage model.

4) Real shared-state SMP (necessary for memory and core efficiency).

5) Standalone, single-directory distribution of the applications and all library dependencies.

6) A broad, stable, mature base of well-documented, well-tested libraries.

The CLR has some advantages over the JVM, but it also shares most of the JVM's attributes.

View on HN

I disagree that the JVM is the only platform on which you could (or would) build such a system. I know because we built a similar system at Joyent called Manta, which provides a distributed, Unix-based, map-reduce environment. It's built almost entirely in Node.js. We use nginx for serving raw data from the storage tier and PostgreSQL for storing and querying metadata (as well as compute job state), but job supervision and actual execution are implemented in Node.js.

Sorry for the self-reply, but to clarify: the _implementation_ is Node.js. Users can run their actual workloads using any program they want.

A mature, reliable, performant, but unadventurous platform which attracts corporates. I am not knocking it - I think it's obvious that the JVM is well suited, but isn't there room for at least one or two competitors on other platforms? Do any exist that you know of? I tend to be a bit more adventurous in my coding projects and given that embarking on projects requires one to be enthusiastic about one's tools, I cannot use this as Java leaves me cold. This sounds flippant but really I don't think it is. I love the idea of DAG workflows but I don't want to do it in Java, even with all its strengths.

The other benefit of the JVM is the multiple languages that run on it - did you try Clojure or JRuby already? Or do you mean Java as in JVM and you don't want that in your stack for some reason?

I am not a fan of any compatibility layer technology as I am a bit of a bare-metal purist, and I believe Linux does this already. The original post waxes lyrical about the Unix philosophy, but part of the Unix philosophy is supreme efficiency and this is violated by the JVM. I know I know - performance is comparable but then coding is a bit like art - you have to love what you are doing.

As for using other languages - I played with Python on Storm but I very quickly found that I had to basically use Java because the entire community and documentation was java-centric (last I checked - a few months ago).

I may be being obtuse here - in fact I know I am - but my point is that there is a large community out there which is not Java friendly (logically or not) and so I lament the fact that all the advancement in this new, very exciting, field, is JVM-based. My dislike for Java was sealed by the Bloomberg terminal API, some of which's basic functionality is 5 to 6 dots deep. Seriously I had 80+ character function calls. This.that.that.this.this.finally()!

Perhaps we'll just have to kick in, leave our biases behind and start running with the JVM.

FWIW my use case is streaming fixed income (bond pricing) data analysis. Python/Numpy is running out of steam fast for me and we're using quite a bit of C. I basically come from the scientific computing set. We're applying advanced statistical analysis to pricing (not HFT - we're operating in the 5 minute thru 1-week horizon - not 5ns).

If you're interested in Python + Storm, we are making it happen via our open source project streamparse. See the project on Github[1] and also my PyCon US 2015 talk[2].

Your point about documentation is well-taken. We are trying to document streamparse + Storm usage from a Pythonista standpoint via our online documentation[3], e.g. here is our detailed Python API documentation[4].

[1]: https://github.com/Parsely/streamparse

[2]: https://www.youtube.com/watch?v=ja4Qj9-l6WQ

[3]: http://streamparse.readthedocs.org/en/stable/

[4]: http://streamparse.readthedocs.org/en/stable/api.html

In one breath "I am a bit of a bare-metal purist" and in the next "As for using other languages - I played with Python on Storm".

So which one is it? :-)

I am not the OP but it is not necessarily a contradiction.

The underlying goal in Pythonic data-science is that one pushes as much of the computation into pre-packaged loops that have already been written in a close to the metal language (C, C++, Fortran). Well, that's as far as the intent goes, practice deviates from it by degrees.

This does not work quite as well in Java (or JVM) because JNI is just supremely god-awful. JVM semantics are overly strict, this over-specification kills optimization opportunities. Is heavy on memory, I would rather use the memory for loading more data than fill it with overhead. Finally there is this OOP culture that gets in the way.

That said, JVM is one of the most mature, and well engineered VMs out there (Java is another story), but not very well suited for number crunching, because there is more to number crunching than calling BLAS APIs.

last time I checked it was Java Scala or Python. None is bare metal. We use Python as glue. Any heavy lifting is done using Numpy ecosystem or C.

What do you find missing in the python/numpy space? Inability to handle analysis over streaming data?

Inability to scale easily to clusters, unless the problem is embarrassingly parallel (ours isn't always), and no equivalent to the conceptual beauty and efficiency of distributed directed acyclic graphs that Storm amongst others uses. We could hack Python up to do this, but if the problem became TRULY huge, hundreds or thousands of nodes, we'd probably end up spending more time building a processing infrastructure, and repaying technical debt, than doing the actual algorithms, while Spark / Storm etc already have massive scaling ability built in. Still love Python and especially the Numpy ecosystem though and we stream thousands of ticks per second through it no problem, with a bit of help from multiprocessing, MKL libs and Numbapro. But as we get towards tens of thousands we're hitting the buffers, and most importanly, problem complexity is rising so the DAG approach is looking very attractive.

I'm looking to build something like Naiad on top of python, disco and numba. Mind if I pick your brain on the kind of tooling you'd like to see around big data python? My email is in my profile.

Clojure is a fascinating language. Rich Hickey gives a lot of his rationale for why he picked the JVM for it [1], which boils down to VMs being the platforms of future, not OSes like Linux.

[1] http://clojure.org/rationale

That's why I highlighted the platform strengths -- what others platforms can reasonably compete?

That said, you can use these tools as complete applications without actually dipping your toes into Java (or even the JVM, other than running it). If you do want to write code directly against them, the choices (of which I'm sure you'll well aware) range from Clojure to Scala to Java to JRuby.

Scala is one ugly duck, but personally I find it to be a sufficient language for writing JVM-based code without losing my lunch.

Well, first, a lot of this stuff is actually written in Scala, which is like Java's younger and more fun cousin. Still runs on the JVM but has many fun language aspects I won't get into because there's already a billion posts online about that.

But second, not quite a "platform" but I've been enjoying Frank McSherry's blog posts in which he builds up a complete distributed dataflow engine in Rust: http://www.frankmcsherry.org/

I'm sure I'm being stupid and underinformed, but it doesn't seem like these systems do anything you can't replicate easily enough with htcondor/gridengine and so-forth alongside cluster computing filesystems. It almost seems like Java people decided they wanted Java only in their stack, so they ignored all the existing unix cluster computing stuff.

Somebody's downvoting me but I doubt they know anything about streaming and short-job support in gridengine.

Scala/Python using Spark Streaming.

C++11 perhaps.

Precluded by requirement #1 (which I say as someone who has written an awful lot of C++).

That's subjective. I'm quite productive writing system software in C++11/14.

Reply on news.ycombinator.com