Julia parallel task runtime (WIP) | Hacker News Reader

Julia parallel task runtime (WIP)

github.com

70 pointsboromi11 comments

This work has been a monumental collaborative effort and I think the eventual outcome will be pretty spectacular.

I gave a talk at an MIT Julia meetup a few months ago that the author of this PR also spoke at; I found his talk to be an insightful overview of this work: https://www.youtube.com/watch?v=YdiZa0Y3F3c

Very excited to see this come to fruition. Julia is really nice to use on parallel clusters and this will really improve that experience. Especially since that exception stack PR just merged: https://github.com/JuliaLang/julia/pull/28878

Is there a non-developer overview of what this patch does? Pull requests often aren't the best thing for Hacker News submissions as they assume a very deep level of understanding of a project… especially when they chronicle long development tasks.

checkout https://www.youtube.com/watch?v=YdiZa0Y3F3c which explains the new features of the task scheduler in a simple fashion.

Basically Julia is employing a new algorithm to schedule tasks, where traditional methods would get stuck/blocked. Essentially, if you run a job with parallel tasks and one of the threads finishes, it's extremely difficult to schedulate the free thread, since the data may still be used by the other threads that are busy; thus you need to have clever way to determine how to allocate computational resources. Julia is implementing results of a new paper on how to optimally do this.

I've been researching parallel programming languages for my thesis and Julia has been really fun to work with. Alongside Chapel, they really make parallel programs fun to write. Although, in my experience, it's been really hard to make them run as fast as C+OpenMP. Mostly because I got trapped in a lot of performance pitfalls as a newcomer to the language. E.g.: parallel TSP branch-and-bound algorithm where I can't seem to improve the performance - https://github.com/novoselrok/parallel-algorithms/blob/maste...

Judging from the linked code, you're using multiprocessing. Using multiprocessing is not similar to OpenMP. If you want something like C+OpenMP, then you'd need to use threading. This parallel task runtime will use multithreading like OpenMP, and should be similar in speed.

I really look forward to using improvements in parallel execution within Julia. I've been experimenting with it for years and in practice parallel execution has been fairly hacky, though it has been improving. When it works, it does seem to work quite well.

It's just me or everytime Julia is in HN a veritable army of shills follows

Care to elaborate? I think a lot of HN posts around popular technologies will clearly have a strong group of followers, not sure why it would be appropriate to call them `shills`.

So maybe there is or maybe it's just genuine interest in the novel parallel task runtime. I'd like to think it's the later.

Posting like this breaks the site guidelines. Please read https://news.ycombinator.com/newsguidelines.html and follow them when commenting here.

Reply on news.ycombinator.com