Hacker News Reader

Comment by systemvoltage | Hacker News Reader

The negativity is warranted if you've ever run Julia or had to maintain it in production. Use it for your own Jupyter notebooks and personal analysis? Great! Need to debug some weird obscure error (which Julia does a poor job of reporting to the user, let alone which line in the stack trace) while having production pressure to get it up and running again? Julia is unquestionably, unarguably and utterly unsuitable. Do not use it in any production workflow, heed my advice as a maintainer of Julia repos and the tech debt that graduate students have created in our company (I am sorry but just stating the facts). You're still unsure? Let's switch sides, you maintain our company's production Julia repositories and I will take your spot to write beautiful code in a couple of hobby projects. I guarantee you, with absolute certainty, you will come down to your knees. You'll lose sleep. You'll hate management for letting this happen.

View on HN

I'm empathetic about your experience but I have been using Julia in production (financial services use case) for over 2 years and it's absolutely working great for us - developer productivity, performance, etc. We are not stopping and writing more Julia code at the moment.

To be honest, you can incur technical debt with any language. Software must be designed and maintained properly if you need a long-lasting solution. Research projects are also different than production code. Proper training and involvement with the developer community could help a lot.

My suggestion to everyone reading this thread - if you are new to Julia programming and need to work on a production project, do talk to Julia Computing folks. Their consultants can lead you to the right track. In addition, join the Julia Slack and Discourse community. You can almost get instant answers to anything you ask there.

Lastly, here's a selfish plug to my book: Hands-on Design Patterns and Best Practices with Julia https://www.amazon.com/Hands-Design-Patterns-Julia-comprehen...

Thank you, I can definitively say that some of the problems arise from poor programming practices than the language itself. But some languages are better at preventing people from writing bad code - see Rust and Go (impeccable packagement, project structure, memory management, RAII, etc) and Python (strict syntax PEP8/black).

That's fair... although I generally found bad code that are not just bad syntax but rather bad design. Bad syntax or even project structure is easy to fix. Bad design is not.

BTW, this kind of discussion is healthy. The Julia core developers are already taking notes and I'm certain they will continue improving the language or ecosystem.

Yes, as a grad student I'm not working on large production code-bases, but I am in contact with quite a few people who are working on large julia production code-bases and my impression is that it has worked well for most of them, hence my dismay at the sorts of comments I see here.

In my experience, one of the biggest problems is that people approach julia as "python or matlab but faster", which is going to be a recipe for failure. Julia is a very different language with its own idioms and very different style and if people try to just write Python with Julia syntax I can see how problems arise.

To be clear, I do definitely believe there are some big pain points in julia at scale, but depending on the usage domain and needs, my impression is that julia is an appropriate tool for at least some niches.

I wonder if the issue here is the graduate students part more than the Julia part (although obviously the tooling needs work). Maintainability is an art that comes with experience (or proper guidance), and I had to deal in college with a ton of really scary C code from other students (like all the program in the main function with 2000 lines scary). And in the same way in a professional scenario (in a polyglot company with mostly young developers) we had to deal with unmaintainable code in multiple languages, from Python to Scala. Even short scripts that were extremely tricky to reproduce in a more sane approach sometimes.

If my job was just that then I would certainly be heavily burned by it and I wouldn't want to touch the language. But the better solution would be to instead learn the language properly and create a serious guideline that everyone has to follow. If people can't use macros properly (or any of Julia's most powerful features), then every PR with a macro requires a very convincing justification and a review from a more experienced developer. If you're adding a new functionality then you have to be sure there isn't cyclic dependencies with other functionalities, and possibly making it a small independent library instead. And more importantly code is not done until it's properly documented, reviewed and unit tested.

I think that this is very insightful. At least in my experience, maintainable code isn't something that naturally arises. It comes from experience and interest in writing code that will need to be maintained. A lot of students (or academic researchers in general) don't really pay much attention to these things, which is totally understandable given their overall incentive structure.

I'm working with a computer science undergrad at the moment who put multiple full versions of his code to into the Github repository. He didn't use branches, he didn't use pull requests for updates, the diff log is just a new file that is the previous one with changes, all of which sort of defeats the purpose of using source control like git. But, as you say, it's something that takes some guidance and experience to understand. I do find that it makes me lean towards tools that encourage good coding practices by design and which make untying knots easier, which it sounds like Julia might not make super easy (I don't have any production experience with it). Obviously, the practices you laid out definitely help to make sure things don't slip through the cracks, though!

Julia is definitely trickier to control because it's an extremely powerful language. After all you can write even programs that can rewrite itself in it (like Cassette/Zygote, which is more than simply AST manipulation but full IR manipulation), there is no limit to how clever your program can be. But that level of introspection in the future could bring amazing linter capability to the language, that could detect all kind of bad practices as you type (for example type piracy, type instabilities, and possibly enforce customizable policies) which will become important tools as more people aim to use the language in production environments.

And until then the best practices for larger project using Julia's paradigm (Multiple Dispatch, which is by itself not as well understood as OOP or functional) will probably become more largely known. I feel like Julia projects should not grow large not because it's not suitable for large scale projects, but because even large scale Julia projects should emerge from the composition of many small and maintainable Julia projects.

Do you have any links to other material where I could read up on some of the problems folks have faced using Julia in production? Thanks!

I'd love to do a write up on the issues to guide the Julia team towards focusing on production and robustness, investing most of their efforts in a solid debugger support (Juno - totally incapable tool), and stopping all new features and "optimization" projects before getting these things sorted out. I am just too beaten, too bruised and injured to have the mental capacity to write an article. :( I am sorry if I am too harsh.

Harshness is not the problem, credibility is. I don't mean that you sound like you're lying, but that it's impossible for anyone to evaluate your assertion or how it applies to their use case. Anyone who reads your comment (including me) just files it away in their head as "somebody on HN made an unsubstantiated comment that Julia sucks in production in some vague way".

If I were in the position of making or advising someone making the decision of whether or not to use Julia, an unsubstantiated, untestable assertion like that makes virtually no difference in my decision or advice. I hope it's clear this is not an insult, it's just that if I have no way to evaluate your assertion or its applicability, there's no way for me to incorporate it into my decision or advice.

By contrast, if you could point to even one concrete example, I could file it away in my head as "someone on HN pointed out that Julia has X, Y, and Z problem". If I were in a position of making or advising a decision on Julia, I would be able to evaluate whether it applies the use case in question, whether it points to some deeper design problems with Julia, or whether I even think it's a problem at all.

You're right, I see the problem. I'll do an article and post it with detailed analysis. It will take some time because repos are like Tolstoy's quote - "All good repos are alike, every broken repo is broken in its own way". There are issues ranging from startup time, upstream packages breaking, constantly changing ecosystem (given for a new language), quirky behaviors and certain problems related to HTTP package and LibPQ packages that I couldn't even debug. So, I rewrote them in Python.

Until then - I concede, please take it with a grain of salt and use your own judgement.

Thanks. I would enjoy reading this too.

It's quite possible that certain area of the Julia ecosystem are not not mature enough. The situation continues to improve quickly as far as I can tell.

Rather than rewriting code in Python, we took an approach of using PyCall. It turns out that the overhead of calling Python is very small... at least not to an extent that I had to worry about. For my production system, we used this strategy to access Oracle and Apache Kafka. The code still looks very clean as the calls to the Python packages are just like normal Julia function calls.

Interesting, those all sound like serious issues I would very much like to learn more about.

I'd be interested in reading this as well. In the past legitimate critical blog posts have made a positive impact on the Julia community.

I can't fault you for switching to something more battle-tested, but did you file issues for the LibPQ.jl issues you encountered? I wrote and maintain the package and we (and others) use it in production so bug reports are much appreciated.

Ah okay. Bummer. I wasn't necessarily asking you to write it, but maybe others have documented similar problems. I don't use Julia, but would love to here more detailed production environment experiences.

I've thought about this problem, quite a bit, and I think that productionizing Julia would require at least a few additional things above your criticisms about error codes.

1. Get rid of global dependencies. Store all of your dependencies in a project local deps directory with a project lock file.

2. Opinionated file system structure. Maybe you don't need this in one-off scripts but definitely for some sort of "project" layout

3. Force all packages in the package manager to obey these constraint.

The ship may have already sailed on 3), sadly.

What's the issue about global dependencies that you're trying to solve? For my production system, we manage version changes by way of Manifest.toml so that we don't get any surprises.

I agree that an opinionated file system structure is nice. Julia already wants you to put source code inside `src` but then underneath you are free to do whatever you want. We generally manage this via coding convention since people can never agree on a standard.

You are describing every Julia package. These are the rules of the package manager.

1. Every package must declare it's dependencies (and to register must declare compat bounds on them, and good devs do that always anyway) 2. Every package must have Project.toml in the base directory, source code goesin `src`, test code goes in `test`, documentation goes in. `docs` (and more structure there if using Documenter.jl) What more could one want? If a package needs more folder structure within `src` then it should be multiple packages. 3. These are the rules, done.

Further you are describing every sensibly done julia application.

Basically no seriously Julia developer uses the global environment for anything but dev-tools, like BenchmarkTools or ProfileView. Certainly one does not depend on the content of them for any reused code -- that is what Project.toml and Manifest.toml is for.

> every sensibly done julia application.

Can you point me to some documentation on those best practices. My girlfriend - architectural acoustics consultant - is working on a project in Julia and is having a hell of a time managing dependency shift underneath her program (also she barely knows how to use git).

Sadly I can not point you to a single piece of documentation that covers everything. That's definitely an area Julia can improve. Writing down the things "everyone" does, so newcomers don't have to learn them again.

The Pkg manual doesn't have a tutorial on standard practice. But it's worth reading the compat section and making sure to always set your compats https://julialang.github.io/Pkg.jl/v1/compatibility/

And the section on creating one's own project. So as not to need to use the global environment. And alternative to using `activate` after starting Julia (and my preferred way) is to start Julia with the `--project=.`

https://julialang.github.io/Pkg.jl/v1/environments/#Creating...

And can go further and create projects via PkgTemplates which is was "everyone" does. Because a good project looks just like a package (one might as well consider them synonyms when looking for thus kind of advice) https://github.com/invenia/PkgTemplates.jl/tree/v0.6.3

It's got anaconda-like project spaces? While I can see why they did that because it's what people are used to... That's a terrible, terrible choice (I joke there's a reason why docker got invented and it's that conda is awful). Why not just put deps in the project directory instead of in ./julia/..

You probably cannot say which company this is? Or what production workflow? Or what you are using Julia for?

Unfortunately not because of obvious reasons, but we are a Biotech AI startup, about 180+ people.

Thank you. And: good luck!

Reply on news.ycombinator.com