Comment by tavert | Hacker News Reader

Comment by tavert | Hacker News Reader

Parent Full thread

> including fast user defined types

Can numba do this yet? Without having to write special "numba classes" code?

Blaze has a giant serialization wall between it and the JVM which it needs to solve to ever be competitive in its target market. Ibis is the much more compelling effort here.

Dynd is an overly complex C++ disaster that is trying way too hard to solve problems that aren't what anyone really struggles with. The number of real world problems for which homogeneous dense multidimensional arrays of floating point numbers is the solution isn't growing much. Real problems have interesting structure that your data structures need to be able to exploit on a problem specific basis. Doing the array layer all in templated C++ with Python bindings is not a recipe for making something easy to use and effective at problems it wasn't previously designed and compiled to solve. Displacing numpy is not going to happen in the Python space.

Dask has good ideas, but the insistence on implementing everything in pure python means you're stuck with the GIL and lousy performance of your scheduler. This needs to be low overhead if they want anyone to actually use it.

Granted special Numba class syntax, and nowhere near as elegant or integrated as Julia.

Dask distributed interfaces directly with HDFS...no need for serialization.

Dynd is specifically designed to deal with arrays of custom types...criticism sounds like it would be more aptly directly towards numpy.

once it gains Numba binding it will lose the Python overhead.

Re Dask...is 1ms per task too much? It has a threading scheduler that can work with numpy arrays to release the GIL....so not bound by that with numerical code.

Regarding regular list and text processing... Julia's poor performance with heterogenous data actually makes it slower than python for cleaning the corresponding dirty and text datasets.

Interfaces to HDFS how? PyObject has to get translated somewhere.

Arrays of custom element types is a good first start, but boy is dynd a seriously overengineered way to accomplish that. I'm referring to array structure, sparsity, symmetry, linear algebraic properties that should be reflected in the type system. Python's type system is lousy for this, and C++ isn't extensible.

Julia can fix performance on heterogeneous data with gradual compiler improvements, and the string representation is due for a major rework. Python can't fix the fact that the language and libraries were not designed to be efficiently JITted, and extension interfaces are closely coupled to the CPython interpreter's API.

I don't see why all that can't be reflected in dynd type system and array metadata. It's just early and foundations are still being laid.

Revelation julia union etc performance...are these improvents a given, hypothetocal or hope? Doesnt fast code on these types fly kn the face of julia static optimization ethos?

Also serious question: dask's use of fast python datastructures like dictionaries gives it a 1ms per task overhead. Is that slow? How does it compare to other dag frameworks like julia etc

> Doesnt fast code on these types fly [in] the face of julia static optimization ethos?

What does that even mean? Julia's union types aren't intentionally slow, they just aren't implemented very efficiently yet. Major revisions of how they're implemented are definitely on the roadmap, not far away.

For fine-grained parallelism of the type you'd use MPI for, 1ms overhead (assuming that's pure overhead above and beyond the actual cost of data movement) could be significant, sure. If you have calculations that need to go for thousands of individually cheap iterations, it adds up.

It means that I thought Julia wanted to avoid inching too much towards tracing JIT optimization.

Well, luckily dask isn't meant for MPI style parallelism. I was wondering how it compared so something like computeframework.

Good to hear that union type performance will be optimized. Are there any issues I can follow?

A better compiler is a good thing, as long as its performance model can stay reasonably transparent.

https://github.com/JuliaLang/julia/issues/8974

Agreed.

Reply on news.ycombinator.com