Hacker News Reader

Comment by plafl | Hacker News Reader

Julia is the Haskell of numerical computing.

It seems like AD is a solved problem, which is not. I can personally think two instances in which I have hand rolled my own gradients: in the context of recommender systems (dense matrices are evil) and right now in the context of real time collision detection (memory allocations are evil).

One of the domains where I hope Julia will excel is precisely as you comment on physics + ml (I check frequently if MuJoCo source code is available at last).

Anyway, I encourage everyone with any background in scientific computing or DS to have a look at Julia. The ecosystem is nowhere near Python yet, but the language is very good and the tooling is getting better. The performance Julia provides without interfacing C/C++ or Fortran is not just a convenience, it has architectural consequences. It's not about of coding faster, it's about coding "further".

View on HN

> The performance Julia provides without interfacing C/C++ or Fortran is not just a convenience, it has architectural consequences.

These statements set the expectations really high. Yet they omit the ugly truth of increasingly slow compilation times, pure compilation caching, JIT which might affect overall performance due to type instability.

Yeah, like Haskell has a nice Quick-sort example prominently displayed at the front page, Julia is sleek for scientific computing, yet it can bite you really bad as soon as you start wrapping the developed model into a real world REST app.

Do not get me wrong, I see some of the benefits of Julia. On the other hand, I do not think this uncritical hype about "coding 'further'" does good for the language.

Compilation times are still an issue, but what do you mean with 'increasingly slow' compilation times? As far as I know and can tell, compilation times have been getting faster not slower, and by a significant amount.

Let's even put raw numbers to it. DifferentialEquations.jl usage has seen compile times drop from 22 seconds to 3 seconds over the last few months.

https://github.com/SciML/DifferentialEquations.jl/issues/786

Yeah, I should not have used "increasingly", yet what you are pointing out is an example of an improvement in a single module, not the language compiler & llvm. It does not actually prove the original comment wrong.

It's not due to effort that is specific to a single module. It reflects an improvement in the community's understanding of what patterns causes slow compile times and also tooling that helps improve compile times by reducing method invalidations.

There may well have also been improvements in the Julia compiler.

Almost all of the improvements came from changes to Julia's Base that improved compilation times and changes to tooling which suggest trivial changes to improve compilation times. In v1.5 or so this would've been a heroic feat. In v1.7 this is something any undergrad could do in a weekend to any module. So yes that proves that things have changed quite a bit.

I manage the MuJoCo.jl wrapper, and having the source code wont (naively) help with AD. Internally, there's a number of iterative algorithms that you wouldn't want to automatically differentiate, and if analytical derivatives were apparent they would be in there (the MuJoCo creator was my advisor). As it stands, finite differencing is the best way to extract derivative/gradient information from the underlying physics model of MuJoCo, and is the suggested method.

We had a paper that included MuJoCo (with custom defined adjoints) within Julia's DiffEq framework to learn continuous control policies (https://arxiv.org/abs/2012.06684) which revealed a whole host of other issues, namely that gradients are not all that helpful when your optimization landscape is insanely non-linear. But that's a different problem.

Open to chatting about; Julia is definitely the tool for this kind of work.

> namely that gradients are not all that helpful when your optimization landscape is insanely non-linear

Did you try multiple shooting? https://diffeqflux.sciml.ai/dev/examples/multiple_shooting/

I want to restructure the docs because I have come to think that any non-trivial fitting problem requires multiple shooting in order to be stable. Otherwise the loss becomes dominated by "loss values calculated beyond where the simulation has already fallen off". So I'd like to dig into some of the non-fitting examples and see if this or some other tricks are the right answer.

I'm not thinking just about AD. I would love lots and lots of little simulation modules instead of monolithic solutions.

Now some uninteresting story: the reason to have a look at MuJoCo is personal: I studied Mechanics I, II and Analytical Mechanics at college and thought non-continuos mechanics a solved problem. When doing my masters thesis about helicopter simulation I needed at some point to model ground contact and I thought "easy, just add a damped spring". I feel that thesis was nice for an undergrad overall but that part as you can imagine was a complete and utter piece of crap. And now at my forties I want to have vengeance.

> Julia is the Haskell of numerical computing.

Is that a compliment or an insult? When Haskell is involved I can never tell.

Me neither, and yet I maintain my opinion.

That's because Haskell is like the programming language equivalent of The Gig That Changed The World[0], although I guess Algol60 also has a strong claim to that title.

Let me explain by quoting these paragraphs Roger Ebert's review[1] of 24h party people:

> As the film opens, Wilson is attending the first, legendary Sex Pistols concert in Manchester, England. (...) Wilson is transfixed by the Pistols as they sing "Anarchy in the U.K." and sneer at British tradition. He tells the camera that everyone in the audience will leave the room transformed and inspired, and then the camera pans to show a total of 42 people, two or three of them half-heartedly dancing in the aisles.

Sounds like the average language designer entranced and inspired by their first time grokking Haskell.

> Wilson features the Pistols and other bands on his Manchester TV show. Because of a ban by London TV, his show becomes the only venue for punk rock. Turns out he was right about the Pistols. They let loose something that changed rock music. And they did it in the only way that Wilson could respect, by thoroughly screwing up everything they did, and ending in bankruptcy and failure, followed by Sid Vicious' spectacular murder-suicide flameout. The Sex Pistols became successful because they failed; if they had succeeded, they would have sold out, or become diluted or commercial. I saw Johnny Rotten a few years ago at Sundance, still failing, and it made me feel proud of him.

I could rephrase that last sentence at "I checked out a contalk by Simon Peyton Jones from a few years ago, still avoiding success at all costs[2], and it made me feel proud of him" and it would be absolutely true.

And no, I also did not expect to find a parallel between Haskell and punk music, but there you go.

[0] https://openculture.com/2015/06/the-sex-pistols-1976-manches...

[1] https://www.rogerebert.com/reviews/24-hour-party-people-2002

[2] https://www.youtube.com/watch?v=re96UgMk6GQ&t=1372s

This is the best HN comment I have seen for quite some time.

If you watch the linked SPJ video, I timestamped it at the moment where he explains what he means with "avoiding success at all costs", but also: if you watch a bit longer he shares the story of a GHC compiler bug where if you compiled a module on windows, and in a different directory than the current directory, and it contained a type error... it would tell you there was a type error, and then delete the source file.

And instead of people going berserk over this, all he got was some friendly emails from people who just wanted to inform him they bumped into this problem, but it's fine, they had a workaround. Because Haskell people in those days were used to Breaking changes with a capital B.

I think that's at least a little bit punk

> it would tell you there was a type error, and then delete the source file

Which makes all the sense because Haskell is about avoiding bugs at compile time

I wonder what language would be considered post-punk?

Scala (experimenting with various styles) comes to mind.

How about proto punk? Something like ML?

I think I have a good news for you. Deep mind acquired and open sourced the project a few months ago.

https://github.com/deepmind/mujoco

I know, I know, but the source code is not yet available. The github repo is a placeholder for now

I thought Julia is the Dylan of numerical computing. :)

>One of the domains where I hope Julia will excel is precisely as you comment on physics + ml (I check frequently if MuJoCo source code is available at last).

Somebody already did a comparison of DifferentialEquations.jl's standard usage against MuJoCo and DiffTaichi and showed that it outperformed them by about 5x-10x.

https://arxiv.org/abs/2012.06684 https://homes.cs.washington.edu/~thickstn/ctpg-project-page/...

That's all showing the raw iteration count to show that it algorithmically is faster, but the time per iteration is also fast for many reasons showcased in the SciMLBenchmarks routinely outperforming C and Fortran solvers (https://github.com/SciML/SciMLBenchmarks.jl). So it's excelling pretty well, and things like the automated discovery of black hole dynamics are all done using the universal differential equation framework enabled by the SciML tools (see https://arxiv.org/abs/2102.12695 for that application).

What we are missing however is that, right now these simulations are all writing raw differential equations so we do need a better set of modeling tools. That said, MuJoCo and DiffTaichi are not great physical modeling environments for building real systems, instead we would point to Simulink and Modelica as what are really useful for building real-world systems. So it would be cool if there was a modeling language in Julia which extends that universe and directly does optimal code generation for the Julia solvers... and that's what ModelingToolkit.jl is (https://github.com/SciML/ModelingToolkit.jl). That project is still pretty new, but there's already enough to show some large-scale models outperforming Dymola on examples that require symbolic tearing and index reduction, which is far more than what physical simulation environments used for non-scientific purposes (MuJoCo and DiffTaichi) are able to do. See the workshop for details (https://www.youtube.com/watch?v=HEVOgSLBzWA). And that's just the top level details, there's a whole Julia Computing product called JuliaSim (https://juliacomputing.com/products/juliasim/) which is then being built on these pieces to do things like automatically generate ML-accelerated components and add model building GUIs.

That said, MuJoCo and DiffTaichi have much better visualizations and animations than MTK. Our focus so far has been on the core routines, making them fast, scalable, stable, and extensive. You'll need to wait for the near future (or build something with Makie) if you want the pretty pictures of the robot to happen automatically. That said, Julia's Makie visualization system has already been shown to be sufficiently powerful for this kind of application (https://nextjournal.com/sdanisch/taking-your-robot-for-a-wal...), so we're excited to see where that will go in the future.

MuJoCo was purchased by Deepmind and open-sourced, a few weeks ago: https://deepmind.com/blog/announcements/mujoco

Reply on news.ycombinator.com