Metaprogramming – Julia Language | Hacker News Reader

Metaprogramming – Julia Language

docs.julialang.org

127 pointssusi2252 comments

Elixir has similar mechanisms. It's great to see language designers explore new ways of making a language homoiconic without pure s-expressions.

Rebol, Io, Smalltalk & Perl6 are also examples of non s-expression homoiconic languages.

c++ too ... just kidding

But more seriously, it seems like once one starts expanding the definition, homoiconicity gets to be in the eye of the beholder.

"In a homoiconic language the primary representation of programs is also a data structure in a primitive type of the language itself."

So you can take a monster language (at least any one with an eval command - Perl5 say), expand it's syntactical definition and get a "homoiconic language".

http://en.wikipedia.org/wiki/Homoiconicity

Fully agree – homoiconicity ends up being almost more of a standard library feature rather than an essential part of the core language. For example, how does the existence of Python's ast module [1] not make Python homoiconic? The main difference, it seems, is how much the language lets you leverage having a representation of code as data to generate code, e.g. with macros. There are projects like macropy [2], but the core Python language doesn't provide macros – although I can't really fathom why at this point since most of the features are there.

[1] http://docs.python.org/3/library/ast.html

[2] https://github.com/lihaoyi/macropy

I know what you mean.

In my examples I think only Perl6 is stretching the definition of homoiconicity, NB. are Perl6 Grammars == data structure/primitive ??

However Rebol (blocks & words) & Io/Smalltalk (objects and messages) do fully meet the definition you give above for being an homoiconic language.

The serious question I would have about non-s-expressions language is - while they might be homoiconic by whatever definition, what makes them easy to parse?

The thing that's appealing about s-expression languages is that code-as-data is easy to do because parse, creating and modifying code is dead-easy. Is there some still-very-easy algorithm for parsing things like infix operators or does code-as-data depend on using the facilities of the language.

Edit: I can see Julia does this by reflection and macros. But I'm still curious about the general issues and algorithms involved here.

> still-very-easy algorithm for parsing things like infix operators

You would be interested in the shunting-yard algorithm[1], to start with.

[1] https://en.wikipedia.org/wiki/Shunting_yard_algorithm

>The serious question I would have about non-s-expressions language is - while they might be homoiconic by whatever definition, what makes them easy to parse?

Rebol is very easy to parse. Here is a Rebol parser in 10 lines of Rebol:

    parse text blk-rule: [
        some [
            str:
            newline |
            #";" [thru newline | to end] new: (probe copy/part str new) |
            [#"[" | #"("] blk-rule |
            [#"]" | #")"] break |
            skip (set [value new] load/next str  probe :value) :new
        ]
    ]

ref: http://www.rebol.net/cookbook/recipes/0042.html

That is interesting and I'm getting back late on this.

But after mentally parsing that, it seems all this is doing separating blocks. What one would want to also do is to also scan operators, if one was evaluating code or transforming it into an abstract syntax tree.

I've been on holiday since I left my last reply so also getting back late on this :)

Everything in Rebol is just blocks of words, where (in general) expressions are evaluated from left to right; however, within each expression evaluation occurs from right to left [1].

So (infix) operators are also just words and have no precedence order so they just evaluate left to right (like Smalltalk) when parsed [2]...

  >> 1 + 2 * 3
  == 9

To enforce any precedence you use paren! blocks [3], which are blocks that get evaluated immediately.

  >> 1 + (2 * 3)
  == 7

refs:

[1] http://www.rebol.com/r3/docs/guide/code-eval.html

[2] http://www.rebol.com/r3/docs/concepts/math-precedence.html

[3] http://www.rebol.com/r3/docs/datatypes/paren.html

is the environment also available? are environments first class? it talks only about expressions - the environment in which they are evaluated seems to be implicit.

what could you do with first class environments? could you implement logic variables and backtracking? something like oz?

also, i think julia is optionally typed? what's the type system like for expressions? i saw an 'Any' in the link, which looked worryingly vague.

[having said all that, as a "scientific programmer", i am worrying more and more that this is a language i really should be using]

[edit: thanks for the informative replies]

Environments are intentionally not first class: that particular dynamic feature really sabotages optimization and can generally be worked around (e.g. instead of letting the full environment escape, expose a closure that mutates only what must be mutated).

Julia has optional type annotations, but this doesn't mean that code without types is dynamic while code with types is static – all code is dynamic, the type annotations just limit the set of types that values can have. It's a subtle but important distinction.

All toplevel expressions are evaluated within a module which is a first class object. Other than that you always need to place the expression into its environment (to construct top-level expressions). If you just place macros the environment is not accessible (i.e. macros operate solely on their arguments), however, you can distinguish whether a symbol will be evaluated within the defining scope (of the macro) or the scope in which it is used.

Regarding types, all expressions are of type Expr or Symbol (there's a few other things that can be in the AST, but that's mostly it). What kind of expression it is is determined by the head field. There as some discussion about making the Expr type parametric on its head field so you can dispatch on that, but nothing in that direction recently.

This is a copy from a comment I did in an earlier thread a couple of weeks ago, but I think it still applies:

I really like Julia. I'm currently playing around with it whenever I find some spare time. I don't see it as a competitor to R, I think it could well be something along the lines of Python or Go. It is LLVM based, and already really fast even though it is still a 0.2 and the JIT seems to have a lot of room for optimisation. Whats more, it seems to offer just the right blend of language features: - Easily include C libraries via a simple ffi [1] - It is homoiconic like Lisp and thus allows for fantastic macro facilities [2] - It has solid parallel programming support via a Coroutines implementation (Tasks) (similar to Goroutines as far as I can tell) - It is a non-pure functional language - In contrast to Go it has generics, so functional constructs like map, apply, drop, reduce, fold, partition & friends are already in there (or can easily be implemented) [3] - It has optional types, so that you can assign types and the compiler will check for it and mark errors and will be able to create optimised code, but you don't have to [4] - Running external programs is a joy [5] (Example: a=readall(`echo hello`)) The community seems to be very alive. There's a simple web framework called "Morsel" and I've recently set it up against a couple of contenders from the web framework benchmark (cpoll-cppsp, phreeze, and some others), and even though it is still a version 0.2, the performance for the json serialization benchmark would be pretty close to Scalatra (I yet have to publish these numbers, will do so soon). I really hope that Julia will grow, as I love the choices that went into the design of the language, and it would be a shame if it would be only a replacement for R instead of something much bigger, as it is such a nice language.

(original: https://news.ycombinator.com/item?id=6172751)

[1] http://docs.julialang.org/en/latest/manual/calling-c-and-for... [2] http://docs.julialang.org/en/latest/manual/metaprogramming/ [3] http://docs.julialang.org/en/latest/stdlib/base/#general-col... [4] http://docs.julialang.org/en/latest/manual/types/ [5] http://docs.julialang.org/en/latest/manual/running-external-...

There is a lot to like about Julia. It's obviously heavily influenced by Common Lisp. I see offhand only one thing I really don't like: they are using object semantics rather than value semantics for arrays. So for example:

  A = [0, 1, 2, 3]
  B = A
  A[1] = 2
  // now B[1] == 2

Their manual section on arrays says this:

In Julia, all arguments to functions are passed by reference. Some technical computing languages pass arrays by value, and this is convenient in many cases. In Julia, modifications made to input arrays within a function will be visible in the parent function.

but as the above example shows, it's really the semantics of assignment we're talking about here, not just that of function calls.

I believe that value semantics is easier to understand and less bug-prone, particularly for nonexpert programmers. It can also be implemented well enough that the small efficiency cost is worth paying in most cases. For one take on how to do this, see my my FSet functional collections library for Common Lisp[0]. For another, see the Clojure collection types.

[0] http://common-lisp.net/project/fset

The Julia team has made it a convention to end functions with a '!' that alter your input variables.

http://docs.julialang.org/en/latest/stdlib/base/ (4th point)

Using value semantics for arrays is lovely in concept but ends up being really annoying and impractical for writing really high-performance array code. This is one of the most common complaints of library authors attempting to write fast, efficient code in Matlab and is one of the many reasons why one ends up having to write all performance-critical code in C using Matlab data structures.

Clojure is reputed to get remarkable performance on a broad variety of data structures, but I've had a hard time finding benchmarks comparing Clojure's purely functional arrays with normal linearly indexed arrays – if you have some references to such comparisons, I'd love to see them. In my experience, nothing beats a plain old linearly indexed, mutable array when that's what you need. Making that basic structure unavailable in a language intended for high-performance numerical computing strikes me as utterly perverse.

Clojure gets great performance with its immutable data structures, but no one pretends they're as fast as plain native java arrays or using mutation. It's normal in the Clojure community to drop down to using arrays or transients (http://clojure.org/transients) in high performance functions and then convert the result back to an immutable data structure. As long as you still present a pure interface, no one knows or cares what kind of mutation you do internally.

Oh, I wouldn't suggest making plain old mutable arrays unavailable. I totally agree that library authors and sophisticated users need access to them (and preferably not by dropping into C).

I just think the array and other collection types presented to users for ordinary application-level programming should be functional. Mutable types should stay in an "advanced topics" section of the manual.

That's a bad an completely not intuitive trade-off for a language that is expected to be fast.

I've first studied Julia today, but just by knowing its niche I already expected it to use reference semantics, not value. It'd be quite confusing if the examples demonstrated otherwise.

As long as it's not exceedingly difficult to drop down to using mutation, having immutability by default is ridiculously nice. I don't think I appreciated it fully until I used Clojure a significant amount, but using languages with reference semantics feels disgusting now.

Iirc clojure lets you get your hands on a mutable array for performance via the transient keyword. The contract is such that you can do what you want (fast things) in the transient block, but it's a black box to outside callers who never see the array in its mutable form. Ie the function is still a pure function since no side effects leak.

http://clojure.org/transients

I'm going to nitpick horribly here: in Clojure, an array is just a Java array. But what most people use are vectors, which are immutable. 'transient' returns a mutable vector, not an array.

I teach scientific programming and this is always the point in the class where the students really start to appreciate the tradeoffs between speed/efficiency, and readability. Passing around large arrays and matrices by value is much more readable, but obviously slower and less efficient than passing by reference, which is in turn more difficult to keep track of, if you're used to languages like matlab and octave.

This is also the point in our introduction to python and scipy/numpy where they realize, if they have to contend with passing by reference in this "high level language" then why to just contend with it in C and be 400x as fast?

Python is typically more like 40x slower than C, not 400x.

Passing arrays by reference really just isn't that big of an issue in practice. We tend to write both non-mutating and mutating versions of most array functions, distinguished by whether they end in ! or not. Typical development procedure is to write an algorithm using non-mutating APIs and get that right; then you figure out where argument copying is slowing things down (using e.g. the built-in profiler) and modify the code to avoid copying, sometimes by simply adding a ! or sometimes by pre-allocating some work array and using a mutating function on it. Compare this with a similar scenario in Matlab where there simply is no straightforward way to avoid copying – if the JIT is clever enough to avoid copying you're in luck, but often it's not that smart and you're forced to live with the copying or drop into C.

I've never used Matlab and know little about it, but in general, there are much better ways to implement value semantics than by copying entire collections. Clojure's collections probably reflect the state of the art here. You might also find Chris Okasaki's book on functional data types of interest (I think Clojure's sequence type was been borrowed from Okasaki).

FSet does it with balanced binary trees, which gives log-time update and log-time indexing, which are acceptable for many applications but not the best you can do for sequences.

The problem is that for arrays, most non-trivial datastructures have a high cost because of indirection. Memory is already the main bottleneck in many numerical operations, and adding pointers is often terrible for memory performance.

For arrays, even large, reasoning in terms of O(N) for access patterns is useless, because the constants are what matters.

Representing vectors, matrices or general arrays as anything other than a single contiguous chunk of inline values is a complete non-starter for numerical work since you can't call BLAS, LAPACK, FFTW or any other high quality numerical library on e.g. Clojure's vectors. Without those libraries, you lose many orders of magnitude performance on crucial operations like matrix multiplication.

I love the Clojure collection types: basing a whole language around persistent data structures is an unbelievable win.

There is a library for Julia that adds collection types similar to Clojure's[1]. My understanding is it's not very thoroughly optimized right now, but it could be a promising place to look if lack of persistent data structures is keeping you away from Julia.

[1] https://github.com/zachallaun/FunctionalCollections.jl

so many sigils, is Julia channeling Perl?

The $ is only for splicing expressions into quoted code. Normal code that isn't doing metaprogramming doesn't have any sigils in it. So, no, not really channeling Perl.

Only Perl can channel Perl.

Once, deep in a meditative trance, I channeled Perl. My friends took me to the hospital after I refused to stop speaking in regular expressions and hashes of hashes.

Stephan, sorry for using this thread to ask you a question:

In the last ~5-10 year everything has moved more an more to the web. HTML5+JS have pretty much replaced Flash,Java and other proprietary stuff. ~10 years ago it was almost all PHP, ASP. Then came Python etc.

If you look at the speed that libraries were developed for node.js (and now Go), it seems like the web-dev community can really push forward a technology if it can be used to serve web content.

Q: Is there anything done in Julia, to actively attract web-devs for it? I think that if Julia were "promoted" to also be perfect use as a web-server it would attract much more developer and speed up library development.

Don't get me wrong, I'm not a web-dev, but would use Julia to replace MATLAB, but I think it might be a good strategy for Julia to speed up the development (especially for libraries).

Not every language has to have webdev functionality shoehorned into it. I hardly think that having more web developers involved would speed up the development of the high quality scientific packages that Julia contributors are already currently working on. The people the Julia community should focus on attracting are data scientists and researchers in quantitative fields. That being said, the guys at Hackerschool are working on a web framework for Julia https://github.com/hackerschool/Morsel.jl.

Oh and major kudos to the Julia team. This is a seriously awesome piece of software. The decision to write most of the runtime in Julia itself and rely on a solid JIT compiler and simple FFI to get the necessary performance was very well thought out.

I agree that web developers may not help building a scientific computation language. But...

> Not every language has to have webdev functionality shoehorned into it.

I'm still in doubt about this. Web is the basic human I/O nowadays.

Web is the low-hanging fruit. Take a nice-looking language, add a web framework, and developers can instantly do something useful in a language they like. And as you say, it's a widespread need, so even a percentage of that attention is quite a few people.

I think if a language really nailed a niche, especially a lucrative one like scientific/numeric computing, it could be successful on those merits alone, because it would get a large percentage of that niche.

> If you look at the speed that libraries were developed for node.js (and now Go), it seems like the web-dev community can really push forward a technology

Libraries aren't really being 'developed' for new languages. The best ones are being copied and ported from existing open source code bases, thus the time to deliver libraries has fallen dramatically over the last 10-15 years.

I like where you're going with this. My question is similar: instead of thinking about Julia serving web content, what are the possibilities for getting it to interact with JS in-browser? If Julia could drive something like D3, or even D3 itself, that would make me really think about moving away from the IPython/Numpy/Pandas setup. (IPython itself already works with Julia, I think.)

John Myles White's Vega.jl package [1] already supports inline D3 plots in IJulia notebooks using Vega [2]. I think it's still a bit beta (we only wrote the IJulia kernel a few weeks ago), but it's already quite capable. This may soon become my favorite way of doing data exploration in Julia (for other things I still love my terminal-based repl).

[1] https://github.com/johnmyleswhite/Vega.jl

[2] https://github.com/trifacta/vega

I have an entirely different take on the speed with which libraries have developed for Node and Go – I don't believe that it's because they are "web languages", but rather because both JavaScript and Go are languages where libraries are written in the high-level user language rather than being written in C via an API. JavaScript gestated in the browser where writing extensions in some other language simply wasn't possible; with the advent of V8, it suddenly became a fast language and all that JavaScript code became the core of an ecosystem of pure JavaScript libraries. Go is new and fast and therefore it has never made sense to write Go libraries in a lower level language.

That being said, writing web apps in Julia is great and very much supported and encouraged. Like Node, we use libuv for I/O and networking. Others have mentioned Morsel, Julia's Sinatra-like web framework, which unsurprisingly – since it's based on the same underlying event framework – gets similar performance to Node.js.

Nice to see!

I choose Mathematica over python for a lot of scientific/numerical computing due to how "lisp-y" Mathematica is.

I'll be keeping a keen eye on Julia as well!

I love that Julia does so many things in so Lispy a way. I wish that other languages like Lush received similar amounts of attention.

i looked at lush several times, but somehow never got into it. some of the things that put me off: the fact that sdl and opengl were not supported under windows (my primary interest was in developing desktop applications and small games), the dynamic scoping (i'm a scheme fan; not having lexical scoping throughout just feels offputting) and the (real or perceived) lack of a strong community. it did look like a great language, though, and i was rooting for it despite never finding a good use for it myself.

I agree with the lexical scoping part (explains more or less why I don't use Lush). Its a pity that there aren't any good numerical computing tools in Common Lisp/Scheme. Oh, well.

What about http://docs.racket-lang.org/math/ ?

I haven't looked into Racket-math, but this post: http://lists.racket-lang.org/users/archive/2013-January/0560... makes me wonder if Racket is really up for non-toy numerics.

In contrast, SBCL uses tagless float arrays, so that array operations tend to be cache friendly. With adequate declarations, numerical code in SBCL tends to very fast (~1-3x C speed). I assume one can do the same with Bigloo and Chicken scheme as well.

On 04/22/2013 01:21 PM, Richard Cleis wrote: > Recently I used it for estimating orientation parameters of > telescopes. Inputs are the locations of stars, outputs are telescope > coordinates, parameters describe the relationship of axes to each other > and to the earth.

http://lists.racket-lang.org/users/archive/2013-April/057487...

> In contrast, SBCL uses tagless float arrays, so that array operations tend to be cache friendly.

This is also the case in Racket.

http://docs.racket-lang.org/reference/flonums.html#%28part._...

I found a package that advertises being an R-like data frame just today, but haven't yet played with it: https://pkg.racket-lang.org/info/munger

If Clojure is acceptable, there is Incanter (http://incanter.org/).

With Quicklisp, the situation has improved a bit. I've used tpapp's libraries without much of a problem.

https://github.com/tpapp/

Reply on news.ycombinator.com