Retro on the Julia Programming Language
blog.staffjoy.com
blog.staffjoy.com
Happily, the present situation is more rosy than it has ever been. Funding agencies are beginning to realize that academic software development is an activity worth funding in its own right, not necessarily tied to research. Julia Computing has also been fortunate to receive funding from organizations such as the Moore Foundation to work on precisely the tools that users most need, but are difficult to justify development with research funds.
I almost feel like there should be two versions of Julia: a commercial version that focuses on stability and is usable in production systems, and a scientific version that undergoes constant revision/feature overhaul to produce the best programming language for numerical computing.
At work most people agreed with the technical insights - it's just a super language, but everyone thought that it was impossible to use it in any real project and very quickly discarded it even as a hobby tool. The clue is / was in the versioning - 0.3; 0.4 now 0.5. Things are broken and major things will change.
I think that this is really good, and important. Getting Julia right may really matter in the future. Code that costs less to write, runs faster and is easier to maintain is an underpinning capability for our economy and for science. Julia could be a huge enabler.
Having said that I find myself having a meltdown over 0.5 (specifically I had a tantrum yesterday when everything broke). I wanted to do some stuff on a private laptop so set up an env under windows. I jumped versions from 0.3.9 to 0.5.3478 because - why not? I'd also been using a 0.4 dev version sometimes, it worked ok but there was one issue that made me normally run 0.3.9 which was fine. However, Atom couldn't cope because HttpServer is @unix_only (whatever the that means, even the complier didn't know, it just said "documentation error")... I had about 100 compile errors in my code due to "' ' not allowed before '(' in foo (bar, bee)" but ok I just fixed those, one of the things that attracted me to Julia was whitespace liberalism, a minor irritation with Python, I find it easier to read code with a bit more whitespace. The trip wire was that I got the same thing from dependencies - like DataStructures.jl
This is intentional by the devs (I put an issue on GitHub and it got closed with that note). I do like "lets get it right", but even as someone just messing about with prime numbers in the evenings I can feel the pain of anyone who is trying to do more significant things. On the other hand a fork like the one you describe is going to be bad - Python 2/3 bad, and Julia is in no way as established as Python was when it had it's fork. The consequences are that the stable libraries are going to have to focus on the stable build and the problem is that at that point the dev version is irrelevant.
For Julia that is a big problem. There is a lot to do in terms of developing the core infrastructure in the language, see the idea of a registry in 0.6 or shrinking base and having a standard library for example. I do think that there is a worry that in covering off all cases and requirements the lovely simplicity of Julia may get diluted, the registry is especially a worry here because dealing with config files and issues is one of my bugbears and I was hoping Julia would stop this nonsense by "just use Git".
Still - I picked up Java at 0.7 and it really became useful at 1.4 Julia is on a much better trajectory already!
Not sure why I thought 0.7, perhaps I was wishful thinking wrt. how much change happened in early Java, but I think that the conventions of version numbering hadn't settled in the same way as they have now.
I also shared the same feeling, given the chaos of writing portable C++ code back then.
I've had my eye on it for the last two years and really look forward to its 1.0 release. There is no doubt there is an audience for it!
I probably could have done it with python, but IMO the language just is cleaner and easier to read. The macro system helps, a lot. The typesystem also gives me confidence in the correctness of some of the stranger code I'm emitting.
It's a really great prototyping language for mathematical concepts.
I was playing with Legos recently and thought of a new way to approach the shift decomposition problem, and I'm amid a prototype in Go that is promising.
We're not using Dantzig-Wolfe decomposition, but our original scheduling algorithm used a dynamic programming approach that was quite similar.
We're currently amid rewriting the Chomp decomposition algorithm to be a more specialized branch and bound variant.
We also migrated from julia to python. Another vote for the MIT teams that worked on the project! It really helped us at the beginning!
We are also playing with the google glop solver that gives pretty good results so far.
The main drawback of python for column generation (and as far as I understand, the same flaw exists in juliaopt/JuMP) is twofold. 1) most libraries are "math formulation oriented" which makes implementation hard to maintain/debug 2) biggest issue: the wrappers around the underlying solver (cbc, glpk, etc) keeps all references (to variables and coefficients) and do not offer a clean add/remove interface (only add/set bounds). Even playing with coefficients isn't always straigthforward.
So we had to build our own wrapper on top of the LP wrappers out there (Pulp, pyomo, cylp etc). When solving large scale problems with column generation, you generate hundreds of thousands of different variables even if only a fraction of them are "active" at a time leading to memory and runtime leaks in the usual wrappers. We had cases where the LP library was spending about 85% of the time just writing and reading files for CBC. The or-tools library wraps the C library itself via SWIG and is our best horse as of writing this.
For sure we will think of rolling our own binder too. Last year I was hoping to do it is scheme :) We might get "stuck" with python for the time being though.
You labast the lack of testing tools but one of the other blog posts [1] is about how you built a testing framework.
We're just just being kind in that blog post because you didn't really spell out any shortcomings then ?
[1]
The `@test foo` command basically raises an exception if the result is false. You have to manually register all of the tests and call them individually. setUp and tearDown has to be done manually. The testing library keeps no data about how many tests were run - it just throws an exception if an `@test` encounters a false.
It works, but there is no sophistication.
Absolutely not.
> The developer tools and communities shape the culture and ultimately the roadmap of the language.
Absolutely so. My fellow PL-ers sometimes forget this part :)---Thankfully this is changing.
He said 'most' not all, so he is correct.
While the work on JuMP is impressive, isn't the fact that all major devs are from the same place worrisome?