Hacker News Reader

Comment by seanmcdirmid | Hacker News Reader

Is Julia really meant for CLI scripting?

It's not.

But people who like the language will invariably use beyond its core competency.

Hence it is important to ensure that julia is "kinda mediocre" for CLI scripting (big step up from "absolutely terrible"). Personally, my familiarity with julia and its features outweighs the fact that python/perl/bash would be the better tool for many CLI scripts.

I think it's disappointing that Julia is not meant for CLI scripting. IMO it has failed to live up to its manifesto (https://julialang.org/blog/2012/02/why-we-created-julia) of being a general-purpose language that can "have it all".

My own bioinformatics work involves parsing massive amounts of text data, and Julia is excellent for this -- it is very fast, yet high-level and easy to write. However, bash pipelines are also a huge part of bioinformatics, and Julia scripts are not very good for this due to the long startup time, which is a shame.

> I think it's disappointing that Julia is not meant for CLI scripting. IMO it has failed to live up to its manifesto (https://julialang.org/blog/2012/02/why-we-created-julia) of being a general-purpose language that can "have it all".

I think it’s still early in the languages development to say that. There isn’t any fundamental reasons Julia couldn’t be tuned to be better at CLI scripting. Currently it’s what, 400ms to startup, which isn’t too terrible but could be made faster. I could see using the Julia debugger as an interpreter for CLI scripts. Or if you run a script a lot it’s possible to have Julia compile an executable. Personally I use it in mainly via notebooks or a repl.

400ms to startup?

    > time julia -e "exit()"
 
    real 0m0.164s
    user 0m0.106s
    sys 0m0.047s

This is on a relatively old laptop:

    Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz

according to my lscpu

I helped work on getting that number from ~400ms to ~150ms maybe 18 month ago for the v1.0 release. FWIW, a big part of it was shedding a couple excess C libraries with bad load times (usually now lazy loading them when required instead). The next big jump will take more internal effort, but I don’t think there’s any serious showstopper. The bigger short-term interest though has been towards reducing the latency of loading external (user) code/libraries.

For an example of what I mean by the latter, python seems to be pretty fast initially, but then seems to take a huge hit from just trying to get numpy loaded.

    $ time python -c '0'
    real0m0.024s
    user0m0.016s
    sys0m0.008s

    $ time python -c 'import numpy'
    real0m0.165s
    user0m1.508s
    sys0m2.312s

    $ time ./julia -e 0
    real0m0.215s
    user0m0.240s
    sys0m0.144s

I believe you compared first (uncached) startup of Python against second startup of Julia:

    $ time python -c '0'
    python -c '0'  0,03s user 0,01s system 7% cpu 0,467 total
    $ time python -c '0'
    python -c '0'  0,03s user 0,00s system 98% cpu 0,030 total
    $ time python -c 'import numpy'
    python -c 'import numpy'  0,17s user 0,05s system 9% cpu 2,401 total
    $ time python -c 'import numpy'
    python -c 'import numpy'  0,11s user 0,01s system 99% cpu 0,118 total
    $ time julia -e 0
    julia -e 0  0,25s user 0,27s system 17% cpu 2,868 total
    $ time julia -e 0
    julia -e 0  0,09s user 0,05s system 92% cpu 0,155 total

The impact of actually calling something from numpy is also negligible in Python but not in Julia:

    $ time python -c 'import numpy; numpy.random.rand(10,10)'
    python -c 'import numpy; numpy.random.rand(10,10)'  0,10s user 0,01s system 99% cpu 0,116 total
    $ time julia -e 'rand(10,10)'
    julia -e 'rand(10,10)'  0,35s user 0,23s system 209% cpu 0,277 total
    $ time julia -e 'rand(10,10)'
    julia -e 'rand(10,10)'  0,36s user 0,22s system 209% cpu 0,278 total

We got approximately the same numbers (your clock speed is likely to be much higher). User/system time is quasi-bogus, since it’s a high core count system (although still a bit concerning). I accounted for possible cache effect by running each a number of times and reporting the last. I’m not really trying to make an absolute time comparison here, just pointing out that if 100ms is unacceptable, numpy would just miss that bar too. Once you’re past the bar of “this needs to be kept running”, I don’t think a constant factor of 100ms vs 1s makes much difference in QoL, and now we’re just comparing apples and oranges. A constant factor gain on the rest of the time can make a huge difference on the rate of results per second. But I actually hope both will improve!

Ah. I was trained by zsh's time reporting to focus on the last figure and noticed that "real" is at the top in your comment only after posting mine. And then still left scratching my head looking at "user" and "system" times an order of magnitude higher than "real".

Excellent! I was mainly going off the numbers given by another comment. Well ~150ms seems within useable CLI times for me. One note, I’d reckon to have a fair comparison to the Numpy timings the Julia example would need to use an array type to do something.

By "slow startup time" I also include the time to import libraries which is really the bigger problem. For example loading the DataFrames library takes quite a long time, and I've heard the libraries for plotting take even longer to import.

Are startup times really your problem?

Consider https://github.com/JuliaLang/julia/issues/30044

You can manually enable buffering of the pipe endpoints. Otherwise julia will be super-duper-slow when part of a pipe.

Julia has been excellent for shelling out since almost the beginning. See https://julialang.org/blog/2012/03/shelling-out-sucks

Nobody complains about julia for "shelling out". The imo justified complaint is that julia sucks at "shelling in".

(OK, the pipe buffering rules still annoy me)

Similar with FFI: Julia has awesome support for calling out to other languages; support for getting called by other languages is suboptimal.

Reply on news.ycombinator.com