Julia Package Ecosystem Pulse | Hacker News Reader

Julia Package Ecosystem Pulse

pkg.julialang.org

63 pointsidunning51 comments

As primarily an R / stats user, the progress Julia has been making in terms of libraries has been staggering. I have just enough historic experience with Python to be similarly impressed if I place myself in that mindset.

I recently attended a Software Carpentry instructor training session and witnessed a breakout between social scientists using R and natural sciences people using Python. Julia will have "made it" when orgs like SWC can start reasonably offering Julia sessions that draw both communities.

That's an interesting statement, because Software Carpentry intentionally positions itself as a follower rather than a leader in technology. That is, Python & R workshops are given because of the wide adoption of Python & R. Julia workshops would need to follow demand.

Agreed, that was my point if it wasn't clear. At the point where there are enough people from natural sciences and social sciences looking to learn Julia at an introductory SWC level, Julia will have made it.

I've been giving Julia a shot lately because I really like MATLAB syntax and IJulia is awesome for analysis reports. There are a few warts though:

1. I really don't like how the entire standard library is included in the default namespace. I know this is how MATLAB does it, but my function names are constantly colliding with built-in ones.

2. The documentation is hard to navigate. I'm used to MSDN. Compared to that, providing huge pages of functions with a sentence or two of prose and no examples is not that helpful.

3. Plotting tools need work. Gadfly produces beautiful output but interactivity is weak (you can only zoom in on Y-axis) and it chokes when you tell it to plot a lot of data. It should downsample based on display resolution... I should be able to tell it to graph 100,000 points without crashing my browser.

> but my function names are constantly colliding with built-in ones

I'm sure you know this, but the casual drive-by reader may not.

Functions with the same name aren't a collision. The parameter types of your function and a builtin of the same name all have to match in order for there to be a collision.

In julia "module" introduces a namespace: http://julia.readthedocs.org/en/latest/manual/modules/

Glad you're enjoying Julia :-)

> I really don't like how the entire standard library is included in the default namespace. I know this is how MATLAB does it, but my function names are constantly colliding with built-in ones.

This may not be obvious, but if you don't use a name from the standard library, then you are free to use it yourself since bindings imported by `using` are done lazily. For example:

    julia> e = "some message"
    "some message"

    julia> Base.e
    e = 2.7182818284590...

    julia> println(e)
    some message

    julia> Base.e
    e = 2.7182818284590...

On the other hand, if you don't assign to `e` then you can use the built-in definition of `e` like so (in a new session):

    julia> e
    e = 2.7182818284590...

If you want to use both your own `e` and `Base.e` then you do need to qualify `Base.e` – but this is exactly the same in any other language – if you use the same name from two different namespaces, you have to qualify them. Making the standard library namespace smaller or more granular wouldn't help in any way.

> The documentation is hard to navigate. I'm used to MSDN. Compared to that, providing huge pages of functions with a sentence or two of prose and no examples is not that helpful.

I definitely agree that our documentation isn't up to the standards of Microsoft. Pull requests are always welcomed and are a lovely way to give back. Over time the documentation will get better – I suspect we're already doing fairly well given the age of the language.

> Plotting tools need work. Gadfly produces beautiful output but interactivity is weak (you can only zoom in on Y-axis) and it chokes when you tell it to plot a lot of data. It should downsample based on display resolution... I should be able to tell it to graph 100,000 points without crashing my browser.

I'm curious what plotting system does actually let you plot 100,000 points without choking. Automatic downsampling would certainly be a nice advanced feature, but the mature plotting systems I'm familiar with do not do this. For example, neither R nor matplotlib do. Is there some Microsoft plotting library that does?

Bokeh for python does this

Definitely a nice feature.

>I'm curious what plotting system does actually let you plot 100,000 points without choking.

Datagraph for OS X handles ~1 million points without breaking a sweat, and it's an interactive GUI application.

I've been using Julia 0.3 for algorithmic trading in production for about 3 months now, and the language is truly, astoundingly productive. The only thing lacking for me at this point is true multithreading support (so I can fork off a python thread from within julia) - I currently get around this by using a socket interface to pass messages to python.

Edit: For anyone who's interested, an indicative problem is this: http://stackoverflow.com/questions/21113030/async-thread-dea...

How would you compare this to using python with an easy JIT like numba?

I haven't used numba or benchmarked it, so I'm not really qualified to comment. I do however vastly prefer the MATLAB-like syntax of julia to that of python.

I actually both in terms of speed, and in terms of writing code they are remarkably similar approaches.

numba forces you into certain ways of writing code to achieve significant speed ups, but usually that's not too restrictive.

If you are used to writing numerical python code, you'll feel write at home in Julia - especially with the amazing PyCall module.

Whenever I have tried to use Numba, I find it is slower than standard python. OTOH, Julia just works. Gives me the same performance as C.

There is a threads branch and we expect to have support in 0.4.

This is great news! How do I get this branch from github? And a big thank you for the tremendous work you guys are putting in.

The work can be found here: https://github.com/JuliaLang/julia/tree/threads. This is still pretty rough and experimental, however. It requires a patched LLVM to fix thread-local-storage support in MCJIT and the patch only works on Linux. And the threading API is also far from final. I can't say that I recommend using this yet.

Could you send me an email (see my profile)? I'd be interested in learning a lot more about how you have your system set up - training, testing, databases, hosting. I've had trouble finding a good resource for how to do so.

Sure, sent.

Could you refactor that email as a blog post?

In this case, you could either use the Python multiprocessing module, or start the python listener from another Julia worker as suggested in the SO answer. I just tried the latter and it works fine: you need to do `@everywhere using PyCall`, change the `@async` to `@spawnat 2 begin ...`, and put the `@pyimport` inside the spawnat block). See: https://gist.github.com/ihnorton/2d22639230d4bbe30639

(alternatively, you might be able to use Python's asynccore module to create an asynchronous listener and yield back to Julia)

Thanks, ihnorton. That worked!

I should point out the main Julia package listing is at http://pkg.julialang.org/, this is the "executive summary" page. At the bottom of this "Pulse" page are test results - we run the tests for every package every night on both Julia stable (0.3) and unstable (0.4) to make sure everything is still working nicely together. Although, breakages on 0.4 just mean cool stuff is happening on master.

You aren't kidding: Gadfly still takes 30 seconds to import on 0.3, but I hear there's some sort of binary caching in 0.4 that should fix it.

We should have static compilation in 0.4.

Of modules or entire programs?

I don't know exactly the current status or roadmap (if there is one), but the idea is to have both eventually. Packages and files will be cached automatically so that they load much more quickly, and it will be possible (eventually) to output standalone executables.

I was thinking the other day, before 2004, did we ever say "ecosystem" to refer to a bunch of software packages in a particular domain? I have a feeling that this book is what convinced us all to refer to our software collections as an ecosystem, and we have forgotten that it was this book's idea:

http://www.amazon.ca/Software-Ecosystem-Understanding-Indisp...

The word is ubiquitous now, but if you think about it, it's a bit of a funny way to refer to software, with allusions to predators, prey, producers, consumers, and population dynamics.

Looks like the growth curve starts before then: https://books.google.com/ngrams/graph?content=software+ecosy...

Interesting, although the smoothing can be a bit misleading. As I expected, "open source" does not exist before 1998, when Eric Raymond, Bruce Perens, et al coined the term:

https://books.google.com/ngrams/graph?content=open+source&ca...

However, with data smoothing, it makes it appear is if the term existed before 1998:

https://books.google.com/ngrams/graph?content=open+source&ca...

The metaphor is widely used in lots of other areas though. Education, Science, Research, Business and so on.

I've written a few small Julia programs and have found it to be a very productive language, in most respects. Unfortunately its module system is horrid. If you look at almost all Julia programs, they are essentially one big file. Any secondary files just get included into a main file wholeclothe. It's like working with archaic PHP all over again.

Don't people used Module in practice?

http://julia.readthedocs.org/en/latest/manual/modules/?highl...

I haven't read up on Julia yet, so I'm curious.

I've only played, but a few hours into it I wanted to partition up my project into separate places in order to separate concerns . I found modules to be a good way to do that once I realised that it's Pascal/modular 2 / ada again and not objects. I think that the problem may be that people are tripped up by the separation of type and modularity when they are used to that being coupled in c++ and java.

I agree with this. I think in large part this is because the developers decided to try as much as possible to 'decouple' the concept of module from the filesystem.

I think this is clever in the abstract, but in practice, it's a lot of headache for very, very little gain and a lot of added confusion.

The main Julia team is a bunch of incredibly smart and dedicated guys who know a lot more about designing language than I do, but I think this was a case of trying to be too clever for your own good.

So what are the module systems you like and what do you like about them?

I really like Python - I think forcing people into using a certain directory structure is nice, and it makes browsing code easy.

I think it's largely a matter of taste though! I just think that having a directory layout that reflects the code structure makes it easy to browse things at a glance, provided there's not too much magic in the __init__ files.

That structure makes sense in class-based single-dispatch o.o. languages like Python and Java. It does not make sense in a classless multiple dispatch language like Julia.

Hangon - I thought a module was a separate namespace; and a way of partitioning programs into files. Can you be more specific in your criticisms?

You and I might expect a module to be both of those things, probably based on experience with languages such as Python.

Julia's position so far has been that files and modules are entirely unrelated to one another. Modules are namespaces, and files are messy things on a disk. Hooray! You can organize your code however you want!

The problem is that I don't want to decide how to organize my code. I want to write the code to some best practices that have been thought up by someone who deeply understands their implications. Julia doesn't have these yet. I hope that it does one day.

The application I work on is fairly complex and I've had no trouble at all building and organizing my code into separate modules.

Is this going to hamper it from being the next python for both general and data science programming? Do you think I should invest in learning julia, or continue with python?

I often say this, but I think if you have doubts about whether you should learn Julia or Python, then you should certainly learn Python.

Personally I really like Julia's module system. It's certainly not dogmatic, which (as in any flexible system) means it's easy to write bad code, but there's nothing stopping you from being as organised as you want to be.

Then there's the pattern of having a few large modules as opposed to many small ones; I can see how that would seem strange if you're used to a more object-oriented style, but it's pretty common in more functional languages.

Well, that is just one way of doing it. As other commenters say below, julia decouples the concepts of namespaces, and files. Which means there are many ways of organising your code.

You're right in that most projects include files into one large modules. Not every body like a it, but it does work for many codebases.

If you want however, nothing prevents you from having separate modules per files.You could also create hierarchical modules, and store the files in a matching heirarchy of directories.

As a young language, some programming patterns are still being experimented on by the purveyors of the language. But I don't think the module system is fundamentally weak in this respect.

The stars seem to indicate popularity but not quality. They should have people rate it 1-5 and give a histogram like other places do, it's much more informative.

Any recommendations for books for a Matlab user looking to use Julia? Thanks

There's not much in the way of books, but a few will be coming out soon, I believe. "Learning Julia" by Leah Hanson, for example. See also [1] for more learning resources.

[1]: http://julialang.org/learning/

Not a book, but scipy had a two-part introduction to Julia talk that's worth watching:

part 1: https://www.youtube.com/watch?v=vWkgEddb4-A

part 2: https://www.youtube.com/watch?v=I3JH5Bg46yU

Calling julia functions from python would be nice

All of the basic pieces are available here:

https://github.com/JuliaLang/pyjulia

Reply on news.ycombinator.com