Comment by nextos | Hacker News Reader

Comment by nextos | Hacker News Reader

Slightly off-topic, but what's your preferred stack for developing data-intensive applications that involve a lot of preprocessing, heavy statistics and machine learning plus a web front-end?

Julia sounds quite appealing as a replacement for MATLAB or R with some Dylan-like semantics plus types and really efficient code generation on LLVM. I wish the Racket - Chez merger lead to something that targeted LLVM to be able to do front-end and back-end stuff using Scheme.

A JVM-centric stack is one of my preferred alternatives. Clojure is great for data preprocessing and manipulation, plus ClojureScript for coding all front-end. Datomic, core.spec, core.logic, anglican, plumatic.plumbing just to name a few are a joy to use. Then there's Scala, which is also a great asset, and tons of fantastic Java / Scala libraries like Stanford NLP, Markov Logic Networks (Tuffy, RockIT...), Factorie, Deeplearning4j, etc. Sadly, Scala-Clojure interop is not very good.

Python is the other obvious option, with tons of good libraries, including a fantastic data analysis ecosystem built around NumPy, SciPy, Matplotlib and Pandas. Plus most deep learning libraries targeting Python first. I just feel the language doesn't scale that well, although things like Numba or Cython help.

Have you considered Lua? You can develop web apps with Lapis [1] and do your data analysis with Torch [2].

The tradeoffs compared to Python: LuaJIT is much faster, which should ease your scaling worries, but the ecosystem is not as developed.

[1] http://leafo.net/lapis [2] http://torch.ch/

It’s really not fair to compare CPython with LuaJIT.

If you compare LuaJIT with PyPy or Numba, the “much faster” argument will simply not be true. What’s different is that both of the above only cover a part of the language, but for specialised (e.g. numeric) application that’s often not a problem.

I think LuaJIT is generally faster, not only because Python is harder to optimize.

LUAJIT is often on par with C. How would that make it slower than Pypy?

> If you compare LuaJIT with PyPy or Numba, the “much faster” argument will simply not be true

Source? AFAIK it does hold true. Last time I benchmarked those, LuaJIT was significantly faster, with PyPy almost being on pair with vanilla Lua, due to Python being a much more complex language and harder to optimise.

Just want to note a ton of the torch people have largely moved to pytorch. Lua with LuaJIT probably works for some people, but I don't see it really growing in marketshare over python..

This is true, but the reason is just people already knowing python / python being more popular. Lua is a much simpler language than python or julia though, I'd absolutely still recommend it for data scientists who are new to programming.

Right but with a good enough FFI and wanting to actually deploy models in some way, having the infrastructure to deploy general purpose applications in the same language (even if it's slower than say: go or the JVM) is really appealing. Lua itself while simpler owns a very different part of the application space than python does. There's a reason more peopple know it.

I would actually recommend against lua because of the lack of libraries for doing every day data science tasks.

I mean look at what facebook had to do to justify using lua, it had to invent the notebook for it.

Hi,

My team builds deeplearning4j. We're aware of the massive demand for python and built a bridge to our tensor library: https://github.com/deeplearning4j/jumpy

This library does direct pointer mapping between our JNI based tensor library and cython (no network!)

So you could off load some of your work to the JVM using pyjnius (which this library uses underneath)

It's not a full solution yet but it's definitely a start to something promising!

We also import python models. We only support keras right now but our new autodiff library (samediff which will also be usable from python!) will handle onnx and tensorflow.

For visualization we tend to use zeppelin which has worked well enough in practice. If you have any specific suggestions or use cases I'm more than glad to take input though. We would love to build a python friendly JVM backend.

Other notable work in this space is what wes is doing with arrow. We are looking at using their tensor interop (it's still kinda green field yet..but it holds promise!) to do zero copy ETL between python and java. That should help as well.

R plus C++ or Fortran. R comes with a whole bunch of stuff built in for preprocessing, stats, it has some web front ends, etc..., and it's trivial to hook up Fortran or C++ code.

Reply on news.ycombinator.com