Efficient Aggregates in Julia | Hacker News Reader

Efficient Aggregates in Julia

julialang.org

49 pointsluu26 comments

Julia is looking more and more like a language that could replace C++ either for systems programming, scientific libraries or plain old business applications requiring high performance.

> could replace C++ either for systems programming, scientific libraries

AFAIK Julia is GC'd and is only embeddable, so meh.

System programming languages with GC were already used at various Xerox PARC systems.

UNIX brought us into the wrong path.

I think Unix did right. Good GC's are enormously hard to implement if the language is complex, they touch nearly every aspect of implementation.

That's why a lot of interesting languages don't have GC's comparable to those of the JVM or CLR, which are the work of large dedicated teams on decade-scale projects. And which are still barely to not-at-all usable themselves in some domains, such as numeric programming (as with Julia) or OS development, where even teams within MS found CLR wanting.

That is why UNIX took us into the wrong path. If those languages were already widespread back then, then we would enjoy the ca 30 years C has enjoyed from compiler development support from the industry.

I don't know whether it would have been a good thing (really don't know).

It seems that GC's developed so far are basically inseparable from individual language implementations, or rather closely related families of languages, which have similar approaches to programming (e.g. rooted in single virtual dispatch on heap referenced objects). These systems require complete buy in to a particular approach to computing in order to be usable from new languages, forcing them to use the same object layouts, calling convention, stack usage conventions, unwinding support, view of concurrency, etc, which can all be very limiting. [Edit for clarity: the high performance GC and access to existing libraries is the honey, the forced view of computing is the resulting bee sting.]

I think if such a siloed GC VM system would have become dominant, we would not have such a huge set of reusable routines to call from really innovative new languages. That's something for which we can be grateful to C I think, despite all of its flaws.

So I think even the high performance VM's delivered by the JVM and CLR are too limiting, really. I'm a lot more excited by the prospect of LLVM making precise GC easier for language implementers. From what I understand, the API is being standardized now, so maybe in a few years we can start enjoying some common work on high performance GC's usable by a really wide variety of languages which don't all have the same narrow approach to computing.

> I think if such a siloed GC VM system would have become dominant, we would not have such a huge set of reusable routines to call from really innovative new languages.

No, it would just mean you would have GC as an another OS service, and use whatever OS ABI was in place.

> So I think even the high performance VM's delivered by the JVM and CLR are too limiting, really

They already good enough to replace C++ systems in high performance trading systems.

I only use C++ in hobby projects nowadays. At work, when I get to see C++ code in enterprise systems, it is usually as part of migration projects to replace them by JVM/.NET based ones.

Cheers to the discussion between pjmlp and sixbrx - classic hn. I feel like computing is such a large beast that one can find two very informed people disagreeing while each being right in wide area.

I don't think you can even call memory allocation "just another OS service", (much less garbage collection). Memory allocation is going to be ubiquitous in any application and if it's performance degrades, so will the application's performance. Further, one knows nothing about "what's behind the curtain", performance will degrade for any medium performance or higher application.

Java is perfect for enterprise code because most enterprise software is more forcused on data-safety and generally playing well with the rest of the data zoo. Enterprise software isn't high performance and thus the costs of c++ outweigh the benefits. Or even more

C++ is needed in system programming and shrink-wrap software where performance is crucial.

On the other, other hand, if a dominant OS forty years ago had imposed a GC model that became ubiquitous, perhaps the sweet-zone where the programmer doesn't have to worry about the allocation process would have been larger and the area where you did have to worry about the allocation process might not have been harder to deal with. Especially, a single ubiquitous program layout model would force hardware makers to adapt and what I understand is that a lot of the need for multiple memory/program-layout models comes from innovative hardware getting the last once of performance through constant novelty in low-level memory access.

+1 for the comment.

Just some info on my side.

Like many developers I do have my issues with C++, but I do like it, since my Turbo C++ days.

However, my experience with GC enabled system programming languages dates back to being an Oberon user in the mid-90's.

So I experienced first hand that such systems are possible and have been ever since dismayed by the lack of attention from mainstream OS vendors.

We would have needed hardware support for GC if it was to be used in systems 30 years ago. We simply didn't have the memory bandwidth or the cpu cycles to run GCd languages.

Not really, one reason why Lisp Machines failed was because common hardware became fast enough to run compiled List natively without special CPUs, besides the AI Winter of course.

Besides Xerox PARC, there were other similar research systems like the Oberon and Modula-3 ones, but without industry support, they never left the institutes using them.

If the path had been chosen 30 years ago, the millions poured into Java and similar systems would have been much earlier and who knows what might have came out of it.

But Java is only 20 years old. Modula-3 was/is good but it couldn't displace C++. Yes, it would be wonderful to be in the alternate universe where we were using GCd provably correct functional languages back on our 128k home machines.

We live in that alternate universe right now! Unsafe, crappy native code is going the way of the _____. Rust, Dlang, Golang, Haskell, Nimrod all native, all safe. You can compile ClojureScript down to native using Gambit Scheme. Probably can AoT compile Clojure using RoboVM.

>System programming languages with GC were already used at various Xerox PARC systems.

With what performance? Handling modern day like multimedia/number-crunching/10M connections etc high workloads?

Good enough to be used as personal workstations for what were the first DTP systems.

Surely if those systems enjoyed the 30 years of commercial investment that certain languages compiler backend's enjoyed, the performance would be quite similar.

  > Good enough to be used as personal workstations
  > for what were the first DTP systems.

If you're referring to Star on the D-series, that was written in Mesa, which did not have GC.

Alto system software was written in BCPL, which did not have GC (or much else).

Bitsavers has a good deal of the documentation for these systems.

Mesa was followed by Cedar, which used reference counting with GC for collecting cycles.

Not to mention the Interlisp systems as well.

Mesa, Interlisp, Smalltalk each had their own instruction sets, effectively VMs, which were written by hand. The D-machines' native instructions were wide words with multiple dedicated timesliced hardware 'threads'. The interpreter ran in one 'thread' and core system functions, also handwritten, ran in the others. It's never turtles all the way down.

(I currently own a Dandetiger and two Daybreaks.)

Thanks for the info, I never saw one such system live, just a few papers about them, so some wrong info on my side.

Still if those systems had become mainstream surely there would have been an investment making them run as fast as possible. After all, Hotspot is a consequence of an commercialization attempt of such systems.

Does anyone know how aggregates in Julia compare to aggregates in SQL in terms of speed? I like the power of SQL, and know it well. I wonder if Julia is worth investigating.

This question isn't that bad, so why the downvotes? I'd like to see a comparison between Julia's performance on an in memory dataframe against the same data in an SQL table. Julia should have the advantage but how fast is it really?

I wasn't voting but to me they are orthogonal questions unless you frame it in specific way. Also to me that is not what the article was about. It was about defining immutable structures.

Julia is a client language and data must be local/in-memory to manipulate which SQL is generally has more going on and is usually being interacted with remotely. Julia might be faster than Sqlite3 but I sort of doubt it for an in-process manipulation where sql can do the work. Julia has an advantage that you can probably do more operations than what sql typically allows for.

I frequently hand code SQL to do complex aggregates in it so that the server can do the calculation fast and only send the result and I don't have to pull the whole data set to work on it. When Julia is client/server and I can write remote functions to aggregate data on the server and then return the result then this question can be asked again I think.

Note that this is a year old. Not that that makes it any less accurate.

Does Julia have an equivalent to interfaces?

Kinda sorta not really. Abstract supertypes currently behave somewhat like an interface. But because multiple inheritance is still an open issue [1], it doesn't really fulfill the requirements of an interface.

But it looks like it's definitely on the roadmap [2].

[1] https://github.com/JuliaLang/julia/issues/5#issuecomment-330...

[2] https://github.com/JuliaLang/julia/issues/4935#issuecomment-...

Great, thanks for the links.

The website is down.

Reply on news.ycombinator.com