Romeo and Julia, Where Romeo Is Basic Statistics
b-lukaszuk.github.io
b-lukaszuk.github.io
I've not done a lot with "numeric" programming, but I was able to pick up Julia really quickly, and I was able to do a more-or-less straight port of my dad's code to Julia. What I was extremely impressed with was how much faster it went out of the box; for some of the more crunchy bits, running on the same hardware, the Julia code ran quite literally 150x faster, and this was without any Julia-specific optimizations other than adding types to the function arguments.
Since then I've learned a lot more about Julia, to a point where it has sort of become my go-to language for starting new projects (assuming I don't need to use a specific JVM library or something). It's basically guaranteed to give me fast code out of the box, it has most of the concurrency constructs that I actually care about, it has a better-than-average macro system (at least compared to something like C++), and personally I just think that the language looks pretty once you get into some more of the details. I wish it were a more popular language, because the only thing that has held me back from using it more has been the lack of library support. Most of the libraries are fine, but for example I had a lot of trouble getting the Bcrypt library working, which is something I wouldn't really have a problem with on a JVM language or C++ or something.
Of course, this is (hopefully) a temporary problem, as the language slowly grows in popularity the library support should just get better.
ETA:
I haven't done proper benchmarks on this so this is purely speculation, but that said I think that the biggest boost comes from the fact that Julia properly optimizes loops. My dad is a smart dude, but not being a software engineer in 2024, didn't really know that using loop constructs in most higher-level language is considered a code smell. I don't now about Octave, but I know in Python it's generally considered way better to try and avoid vanilla "loops" and try and convert looping things into some kind of vector operation.
I think the JIT part of Julia is smart enough to optimize loops out of the box, and so my straight port of my dad's code just "magically" got faster, and I was extremely impressed, but I suspect that if my dad had utilized a lot of the vector operations of Octave and/or Matlab, the results wouldn't be nearly as impressive. I suspect Julia would still have performed better (if nothing else because I was able to add type information), but I suspect it wouldn't be the 100+X speed increase that I was getting.
I know this wasn't the point of your post, but just noting that adding types to function arguments "usually" shouldn't impact performance at all :) when it does, that might mean there is some type-instability being papered over (i.e. there is a better fix lurking around the corner)
I didn't do a control test to figure out exactly why, but I think it was having some issue with a parsed CSV being read as a matrix. I think explicitly putting that requirement into the function signature allowed for a lot of low-level optimizations to be used that it couldn't use before.
I'm not 100% sure though; basically when I was getting some performance bottlenecks I did some preliminary searching and some of the results said "try adding types to your function signatures", and I really couldn't think of a good reason not to add them for most of the variables, and so I did and it worked.
Concrete types in a function signature also work as assertions, so it's one way to force the compiler's hand this way. In many cases it's better to assert the arguments, rather than the parameters, to keep the method itself generic. The main reason to add types to a method is to allow multiple dispatch to pick that method of the function for values of the specified type.
The slow function I added the types to didn't really make any sense to be really generic. For nearly all the arguments (of which there was like 20), there was only one type that made any sense for each. I don't think keeping it more generic would have bought me much, and adding types worked as an assertion that made things much faster.
But yes, once I started adding types to thing, I did start utilizing multiple dispatch, since I've always been a bit fan of Clojure-style multimethods and this got me something more or less comparable to it.
No they do not, this is not how it works. Functions auto-specialize on the types they see and do a multiple dispatch behind the scenes on these specializations. Type parameters in a function signature thus only define what specializations are allowed. See https://book.sciml.ai/notes/02-Optimizing_Serial_Code/ for details. As such, you don't need to add any types to a function signature to get performance.
> So I guess it’s gonna be about a programming language named Julia and its usage for basic statistics.
This writing is weird, you are the author right?
this is just a normal sanity check and is not a strike against Julia (or R)
> Still, if you are new to (Julia) programming and statistics then most likely you should calibrate your tools first. Before you run some statistical analysis you may want to try it out on an example from a textbook written by an expert (not me though) and see if you get the same (or at least comparable) result on your own. Although this is a sound approach, I suspect you are more prone to visit some statistical blog or internet forum and go with the examples that are contained there. One such option is rseek.org, i.e. a search engine for the R programming language.
> ... Once I got both outputs that are similar enough I can be fairly sure I did right. Otherwise I should investigate where the differences come from and possibly make some necessary adjustments.
It doesn't say to compare it against R, it says that if you are new to programming and statistics you should check your work against known-good answers or against a second implementation before rushing ahead, and it gives R as an example of a place to find code that you can use to check your work.
This isn't advice about one language being better, it's the usual advice to solve the same problem in two different ways to make sure you got it right!
I did the same to r using ibm spss. Not that I trust that more. Just to make sure when newly program you have used the right parameters. If they compare, you finish learning and switch over. A cautious approach when your job is at stake.
Since the LLM breakout on Huggingface, I am back in Python world.
I mean, I tried it a few times a decade ago, but the environment was not mature, and performance improvements were not in the places I needed. Though, I guess, things moved from there.
Python has the richest environment for data science. For statistics, there are some tools in R, such as the stunning ggplot2.
For numerics-heavy arrays, we have PyTorch.
For custom low-level numerics, Rust rocks.
I tried to use it for quick scripts and calculator-like-use, but the JIT compile times made it unfeasible. (Julia's math syntax is the best I've seen)
For more complex and performance-intensive programs, I don't see the case for choosing it over a general languages like Rust and C++.
I mean, the fact that it's getting comparable speeds to C++ or Rust, while having full garbage collection and nice built-in concurrency support is a selling point to me. I'm not saying you couldn't write code faster in C++ or Rust, but I'm going to argue that Julia will often be just as fast or faster from "default implement".
> It isn't faster if you are leaning on garbage collection. Just like any other garbage collected language, you want to avoid the garbage collection.
I agree in theory but not in practice. Absolutely, if you are good at properly freeing memory, and knowing when to free memory, and how much to free and all that fun stuff, you can definitely get much better performance than a GC, with fewer (or zero) pauses and better throughput.
However, I don't think I agree that that is the average case. A lot of people will simply malloc and free at the beginning and ending of the function, or worse they will do things incorrectly and so the data won't actually free, or it frees before it's completed. A GC does incur some overhead, but they can also be extremely optimized and and have the advantage of doing things basically correctly, or at least safely. If you through multiple threads into the mix, then the problems get much harder.
Julia's doesn't do this, but GC in languages like Go also basically don't cause hiccups or hangs, at least not long ones. The pause times are sub-millisecond [1], and it's pretty impressive. Even Java, the language that initially gave GC's a bad name, has the Shenandoah GC in some builds of OpenJDK which also gets sub-millisecond pauses [2].
Personally, I tend to believe that an application that stops working is generally "slower" than one that keeps working. A lot of the time savings you get from avoiding a GC will be eliminated if your program crashes and restarts.
(That said, I will acknowledge that Rust probably makes this much better; I haven't done anything "real" with Rust so I cannot comment on it.)
What I was trying to say was that Julia, at least in benchmarks, does get speeds getting close to something like C++, even with GC enabled, and it will certainly run circles around something like Python. If you want something that gets around at least 80+% of the speed of C++, while also giving all the prettiness and easiness of Python, I think Julia is a very good choice, and I would happily choose it over something like Go.
[1] https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8
[2] https://wiki.openjdk.org/display/shenandoah/Main#Main-Perfor...
I don't think programming mistakes in other languages make a point when talking about garbage collection performance.
but they can also be extremely optimized
Every language I've ever seen with garbage collection has gone through decades of "now the garbage collection is better" or "just wait until the next version, garbage collection will be better". I've seen it in java, D, julia and every scripting language. It's always a thorn people have to work around when talking about performance interactivity. Julia is right in the middle of this common problem and that is fine for what it is because it isn't a typical systems language and avoiding the gc is not difficult, but to pretend that it is the same as C++ is just not true.
A lot of the time savings you get from avoiding a GC will be eliminated if your program crashes and restarts.
This is besides the point of performance and no longer talking about reality, it's just FUD from a "what if" future.
Julia, at least in benchmarks, does get speeds getting close to something like C++, even with GC enabled
Right, but you get it by avoiding allocation and avoiding the garbage collector the same way avoiding allocation in C++ is important, but in julia it won't be woven in to the performance, it will cause big pauses.
Ok but the Go example I linked is already in production, right now, you can use it. This isn't a "it will get better in two releases" situation, Go's GC as of today has pause times that are sub-millisecond. The Java Shenandoah example I linked is still mostly in beta, but it's also something you can use right now, though admittedly it'll probably be awhile before it's in a mainline release.
> This is besides the point of performance and no longer talking about reality, it's just FUD from a "what if" future.
It's not "just FUD", there are dozens of reported security issues that have happened because of bad manual memory management problems. Off the top of my head, Heartbleed was a famous case.
This isn't me badmouthing anyone; manual memory management is hard to get right, even for very smart people.
> Right, but you get it by avoiding allocation and avoiding the garbage collector the same way avoiding allocation in C++ is important, but in julia it won't be woven in to the performance, it will cause big pauses.
Fair enough, I did look at the code for the official benchmarks (https://github.com/JuliaLang/Microbenchmarks/blob/master/per...) and outside of the integer parsing code it does indeed seem to avoid dynamic allocations so I will concede that the benchmarks might be a bit more skewed compared to real-world code.
I still get a hunch that if you compared it allocation-heavy Julia to malloc+free-heavy C++ the differences wouldn't really be that far off, but that's just a hunch and I don't have data to back that up; might be a fun test to write though, so maybe I'll try that this weekend.
-----
Sort of tangential, but I also do think that there's value in having decent concurrency constructs built into the language. With C++, if you stick to built-ins you are basically stuck with mutexes and despite what people like to pretend, getting correct code with mutexes is really really hard to get right, and very easy to screw up in a non-obvious way. If you allow yourself to use libraries, then you have stuff like ZeroMQ and OpenMP and stuff, so it's really not that dire realistically. However, I think there's value in having nice, easy to use concurrency constructs in the language other than mutexes, and I do wonder if as a result of that it encourages people to utilize multiple threads more frequently, because they don't have to worry about weird deadlock situations as much.
Again, I believe Rust actually does address this because of the single-owner-enforced-at-compile-time stuff, but I haven't used it enough to really draw a conclusion on it.
This was a thread about julia and garbage collection, you keep talking about unrelated things and trying to go off in all sorts of other directions.
manual memory management is hard to get right, even for very smart people.
This is very different in modern C++, I basically never have memory problems since the actual allocation is minimal and inside value types.
if you compared it allocation-heavy Julia to malloc+free-heavy C++ the differences wouldn't really be that far off
The point is that if you care about performance you don't do either of these. If you try this you will see that they are both much slower than they need to be, but julia will pause and hiccup because of the gc.
With C++, if you stick to built-ins you are basically stuck with mutexes
This is a bizarre and unrelated tangent, but it's so wrong I figured I would say something. Not only does the standard library have great primitives, but there are great libraries out there for lock free queues, lock free hash maps and openmp for fork join parallelism. I don't think anyone has concurrency nailed down but julia is not any better.
Yes, but I was referring specifically to your statement "Just like any other garbage collected language, you want to avoid the garbage collection." I took issue with that statement. I think you know that and you're pretending to not.
> This is very different in modern C++, I basically never have memory problems since the actual allocation is minimal and inside value types.
Fair, maybe my C++ is out of date.
> The point is that if you care about performance you don't do either of these. If you try this you will see that they are both much slower than they need to be, but julia will pause and hiccup because of the gc.
You keep asserting different, slightly contrary things, and are restating things that I've already conceded to. The statement "if you really care about performance" is doing a lot of work; most realistic code isn't hyper optimized to remove all allocations and the like. Most code is written by mediocre engineers. Julia gives very good speeds, comparable to C++, in those situations.
If you're doing high frequency trading or something then sure, maybe you're writing this hyper optimized code. In my experience that's not typical, and the initial point I argued that you took issue with for reasons I have to admit I still don't understand is this: Julia can be used for general purpose stuff in place of C++ and Rust, and you'll get decent performance while also having full garbage collection.
> This is a bizarre and unrelated tangent, but it's so wrong I figured I would say something.
In the thing you're responding to I said it was a tangent so acting like I'm changing the subject unprompted is weird. I also mentioned OpenMP and mentioned that this wasn't a problem realistically, so I don't see why you felt the need to restate it.
Are these the concurrency primitives you're talking about? https://en.cppreference.com/w/cpp/thread
Sure, those are fine, but I dispute that they're as generally-useful as CSP style message passing. Moreover, Julia has parallel for loops out of the box (https://julialang.org/blog/2019/07/multithreading/), in addition to having built in support for not only parallel computing, but also distributed computing across multiple nodes (https://docs.julialang.org/en/v1/manual/distributed-computin...).
Now I again will restate that I am aware that C++ has libraries to do this, you don't need to pretend that I didn't say that. You can get nice distributed computing support with ZeroMQ or Nanomsg and probably a dozen other libraries.
Honestly though, this entire topic is frustrating to me because you're getting to the point of outright dishonesty.
Julia has parallel for loops out of the box
Every major C++ compiler has openmp built in. Julia is great but I think you should study up on C++ if you want to compare them.
Worth trying again unless that was in the last 18 months. Precompilation has gotten dramatically faster, like 50x improvement in load times faster. So has startup time in general, `time julia -e "return 5"` gave 161 milliseconds of user time on my computer. I don't actually use Julia for simple script munging, simply because I'm used to Python in that role and it works fine, but it's suitable for it.
Calculator-like use? Julia, all the way. Of course I keep the REPL open at all times. Gotta use tools the way they're intended, y'know?
I think the most striking moment I had was when I realized through experiment that rust, which has a notoriously slow compiler, was faster to compile and run a small plotting program than Julia. That was ~5 years ago though!
The big push now is to add everything needed so that making object/dylib code in Julia of comparable size to the usual systems languages is possible. The team has a track record of finishing what they start, I'm optimistic that they'll get that right as well.
Python, but for engineers and scientists. Syntax is friendlier for doing mathematics, barrier to entry to first calculation script is lower
a = [1 2 3]
b = [4 5 6]
a + b
import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
a + b
Python is for software people. Matlab, Julia and R are for people who only write code because they want answers.
Or, Julia can be for both.
Julia lets you start from scientific calculations scripts and grow your code to a larger software package. Same as Python grew popular 20 years ago because it let you grow your small sysadmin scripts to larger software.
Matlab and R are too limited in their "real programming language" constructs, and it gets painful when your codebase grows larger. Julia has all "real programming language" constructs a software engineer would need.
Also, Rust specifically doesn't have built-in rational numbers, complex numbers, bigints or proper matrixes. There are lots of crates that implement them but that's precisely what prevents a cohesive ecosystem from developing.
This could be achieved by a mix of Python and Rust, but it's just too annoying to write scripts in two languages, where one is compiled and the other has terrible package management. Much easier to write it in Julia and have it be fast out of the box