The Lily Programming Language

Actually, no, I've never used C# except for a few minutes and only one time. I have more experience with the syntax of Scala and Rust. Between the two, I found Rust's more appealing. I like that enums serve a single purpose and can't be inherited.

Currently, the best you can get for iteration of Lists and Hashes is to do each-ing the way you'd go about it in Ruby, except that the language currently lacks iterators.

That's not to say I don't want iterators. But I've spent a huge amount of time of this release in making sure that all of the built-in classes get dynaloaded so that memory usage stays low. Now that everything's getting dynaloaded properly, I've less of a problem with doing iterators.

Lack of escape analysis or manual alternatives really kills perf on .NET. It feels like they got "good enough" vs Java and left it there.

> I'd very much like to see zero-cost abstractions in a new language, so that there is no surprise differences between various ways of iterating over collections.

I never got around to writing in detail about it, but I have that goal with iteration in Wren, and came up with a little iteration protocol[1] that I think accomplishes it.

In most languages a loop structure like:

    for (i in sequence) {
      print(i);
    }

Desugars to something like:

    {
      var __iterator = sequence.getIterator()
      while (__iterator.moveNext()) {
        var i = __iterator.getValue()
        print(i)
      }
    }

You've got one fundamental piece of state: a cursor that tracks where you currently are in the sequence. This is separate from the sequence itself because you can iterate over the same sequence multiple times, sometimes even simultaneously.

And you have three fundamental operations you need to perform:

1. Produce a new cursor pointing to the beginning of the sequence.

2. Advance the cursor and determine if we've reached the end.

3. Get the value at the current cursor position.

In most languages, the last two operations are defined as methods on the cursor itself. That means for any given sequence, it needs to define a cursor type specific to that sequence (and storing a reference to it), and you have to allocate that object at runtime, mainly so that you can use it as a dispatch token to find its implementations of those two operations.

That's kind of lame. In Wren, it desugars a little differently:

    {
      var __iterator = null 
      while (__iterator = sequence.iterate(__iterator)) { 
        var i = sequence.iteratorValue(__iterator) 
        print(i) 
      }
    }

The first two operations are combined into a single iterate() method that uses a "null" parameter to mean "produce a new cursor" and any other parameter to mean "advance". If it returns null or false, we're done with the sequence. Otherwise, it returns a cursor pointing to the next element in the sequence.

To get the value at the current cursor position, we call iteratorValue() on the sequence passing the cursor in.

This means the cursor object is pure state—it's just a bit of data that gets passed repeatedly to the sequence. The sequence itself has the dynamic dispatch of the methods for its own iteration protocol.

What this means in practice is that for simple things like lists, you can use a dead simple cursor type: a number.

The implementation of the protocol for lists looks pretty much like this (except its actually in C in Wren):

    class List {
      iterate(iterator) {
        // Start the at first element.
        if (iterator == null) return 0

        // Stop at the end.
        if (iterator >= count) return null

        // Advance.
        return iterator + 1
      }

      iteratorValue(iterator) {
        return this[iterator]
      }
    }

Since numbers are unboxed in Wren, it means many sequences can be iterated without any dynamic allocation at all. It makes it easier to implement the iteration protocol because you don't always have to define a class that stores a reference back to your sequence.

[1]: http://wren.io/control-flow.html#the-iterator-protocol

I can’t tell if you’re referring to iterators (abstracted by `yield return`) or enumerators (abstracted by `foreach`).

In case it’s the latter, I’ll just point out that you can have zero-cost abstraction because, in C#, `foreach` is pattern-based: An object is enumerable with respect to `foreach` whenever:

1. It has a public method named “GetEnumerator” of type ‘E

2. 'E has a method named “MoveNext” of type () -> bool and a property named “Current” of type () -> ’T (where ’T can be any class (including those that inherit from System.ValueType.. i.e. a struct)).

While it is the case that `yield return` does result in allocation for a reference type, it’s also true that this is easily remedied should it become a problem: In my experience, it only takes about 5-10 minutes to implement a zero-cost iterator as I described above.

Many of the BSL collection classes (e.g. System.String (as an IEnumerable<char>) and System.Collections.Generic.List<T>) use the zero-cost abstraction.

Thank you!

As for me, I'm not so sure what to share. I'm not all that exciting. Most of the time I have I spend on Lily. Once in a while I take a break. But I don't get out that often. Lily's nice in that it's a place for me to do something and feel like I make a difference.

My day job? For most of the time that I worked on Lily, I was working part-time at [redacted] as a Maintenance Assistant. My favorite memory was talking shop with a guy there who had some computer knowledge. He asked what I had done and I told him that I had just written a garbage collector and that it was so awesome because it took just a week and it went well.

He answered me saying something to the effect of "Well of course you wrote a good garbage collector, you're Maintenance! That's what you do!" We both had a good laugh at it.

I've never had a job doing software development. I recently left [redacted] to pursue a job in software, but haven't had much luck with it. I'm still holding out hope though.

So, no, my job isn't programming related.

Resources? My strategy is that when I get up to something new, I do all of the research that I can both in the topic itself, as well as how other languages do it. One example would be Exceptions. I didn't know how to do that, so I started researching how Python did it, how Java did them, Ruby, etc. I started thinking of what I wanted, and also what the goal of having them in my language was. I also made sure to search through Haskell, Scala, Rust, and OCaml since they don't use exceptions as much. The point being that I wanted to know what languages do when they tend to not use exceptions.

Rather than starting off as "This is a good idea", I instead approach it as, "I think I would like X. How does that typically get done?"

For resources, the most helpful thing I've read has been ESR's guide on struct packing:

http://www.catb.org/esr/structure-packing/

I'm afraid not much else comes to mind though.

The idea was probably to show off the features; |> as a 'pipe operator' is cute. It was a bit jarring to see those different ways of constructing an object in one list.

As stevedonovan says, the 'Spike' |> Cat is probably contrived. The feature probably exists for the sake of things like

   cats.map{|c| c.name} |> print

Which is more readable to me than

   print(cats.map{|c| c.name})

It's a little jaring that it has both Cat(arg) and Cat.new(arg) syntax. But I wouldn't want to judge that without learning about the semantics that it emerges out of.

I've heard a few complaints about this now, so I'm thinking of changing it. I wanted to cram a lot into a small space, and it didn't do so well.

The piping is more an example of how piping works with everything and that, yes, the new method really really is a method instead of some magical keyword.

But I stand by piping: https://lorgonblog.wordpress.com/2008/03/30/pipelining-in-f/

I started working on an embeddable, byte-compiled dialect of Standard ML after coming to the realization that Lua with types (as I conceive of it) is extremely similar to ML, but I've had to put it on the backburner for now. But, I agree: a typed language that can be used as an embeddable extension/scripting language would be cool.

First of all, thank you! As for what motivates me, it's a few different things:

* The challenge of it. Lily's one of the few things that I've done where I feel really challenged, and where I can actually make a true difference in it. It gives me something to do, something that I can always be refining and thinking about.

* I -strongly- believe there is a market for this. I want to see this used in game development, used in server development, and used standalone. It's really, really fast scripting language with static typing that brings a unique blend of features that allow for terse code. I want to eventually port this to Arduino, just to say that I did it (Lily boots in 6K with 64-bit pointers, so I think it's possible).

* It's something that I can put at the top of my resume.

* I balance what I do. You might see me spend a week doing plumbing work in lexer, and the next I jump around and patch different places in emitter. I try to not stick in any one area too long. If I do a feature that's difficult and takes a long time, I balance that out with what I term "candy". Candy, in this case, being something that's more immediately visible. I added the constructor shorthand syntax shortly after doing type erasure. Type erasure took a month, and I was a little burned after that. So I balanced it out with a feature that was easier, and had a more visible effect on the language.

* Once a week, I'll go over what I've done, and sort of pat myself on the back, and also think of "Alright, X is done..." and it always goes to "So what's next week?" There's never been a "I'm done". Honestly? The first two years were the hardest, because I didn't have a product that I cared much for. These last three have been easier, because I'm excited about what I'm doing, and there's a more broad spectrum of what I can do.

* You all. No, seriously, I'm finding myself more often talking to different communities about what I've done. I talked to 4chan's /g/ last night. Sometimes I talk to different boards on Reddit, sometimes I try to talk here. I tried Slashdot once, but they didn't bother to even approve the article. Some people give crummy comments, but seeing the new stars, seeing people say they're willing to try the language, that means a lot to me. I really enjoy engaging people when it's over text about what I've done, since I'm comfortable when talking over text and I don't get RL opportunities to talk about what I do hardly ever since I don't know anyone personally RL who does coding for a living, and I'm a pretty stay-to-myself person.

This is a big problem in the language now. There's no solid apis for working with the frontend (invoking the parser and getting a result). The backend apis are acceptable, except that it's difficult to move values around.

I'm a couple of weeks away from the end of a three month release window, so I've (for the moment) given up on trying to finish the apis. But I'm not far from having them done. A month, maybe?

But after that, I'll write a guide on how to embed Lily, and how to extend it.

For the moment, your best bet if you're creating a standalone package like postgres is to refer to how postgres structures the dynaload table. If you're looking to embed the interpreter, it's awkward and hard.

Once this release cycle ends (July 10, Lily's true 5 year anniversary), it's the first thing on my list. I want to give people a solid api so that other people can start building stuff and not have to worry about me breaking it.

First the api, then the tooling, then documentation. You've got my word on that (and HN won't let me delete this, on top of that).

No, I'm just terrible at making up examples so cats! Cats everywhere!

Multiple return values I'm not very interested in. Lily has first-class tuples, and you can return a tuple which I might later add in desugaring for. But multiple return values themselves cause a lot of problems with the internals being pretty set on 0 or 1 return.

Macros are a hard no. DSLs, maybe. For the latter, I could see allowing a pluggable import handler through Lily's options. But I'm reluctant to commit to a yes or no on that at the moment, since I'd rather get over the hump of api not being fully fleshed out.

I owe ESR a lot for writing about struct packing, because that's a large part of how Lily uses so little memory:

http://www.catb.org/esr/structure-packing/

I don't like the colon too. Not in Python, not in Nim. It seems jarring to me, something someone fearful of programming might want!

Also, as Larry Wall said: everybody wants the colon, so it is an extremely useful symbol elsewhere! A shame to waste it.

That being said, this is by a long shot the most tasteful little language seen in a long time.

There's no operator overloading. I'm pretty reluctant to add it too, because I've seen it be abused far, far too many times. It has uses, yes, but I'm not very interested in it at the moment.

I won't say absolutely no to it, but if it were to be added, it wouldn't be right now. There's too much api that needs doing right now.

Thanks!

I started Lily to be an alternative to PHP. The first commit was adding templating support, and it was a while before the language could be standalone.

I had two requirements when creating and designing Lily:

One, that the language had to be interpreted. I didn't want to offer people an alternative to PHP but then tell them that they would need to repeatedly run compile passes over it. Facebook has HipHop, and it transforms PHP into C++. I wanted a language that could be used like PHP, but having a faster interpreter wasn't enough.

The other requirement was to have a statically-typed language. I believed that a statically-typed language would lend itself to better transformation into a lower-level language. In theory, an llvm frontend might also be writable that took Lily as input.

I wanted Lily to outperform PHP both when interpreted, and when compiled. Over time, the focus broadened once I realized Lily's low memory usage, and how fast the turnaround of the parser is.

I couldn't think of a good name. I had ideas of something like Meteor or Hurricane or Tornado. The first was taken, and I -very- quickly had second thoughts about the other two.

My mom, from time to time, mentioned that she really liked the name Lily. After giving it a lot of thought, I said, "You know, why not go with that?"

I've grown to like it.

That's a good question, and unfortunately one that I'm at a loss as to how to answer right now. I've been focused more on building it, and less on using it. However, I've been looking to change that lately now that the core is becoming more stable and usable.

No traits yet. A few other things need to be done first. Generics need to be name-able, System F, and core api is a more pressing need right now.

My plan is to tackle traits in a few months, so long as everything goes well. But I want the above stuff done first, so that when I do traits I can start thinking about higher kinded types, trait inference, and the like.

Already answered :) https://news.ycombinator.com/item?id=12017668

To my understanding, Swift wasn't public for a long time. The idea behind requiring a special syntax for interpolation is that it's supposed to be more clear as to do you really want interpolation, or do you not actually want it. I'm :/ about the interpolation syntax right now. One thing I wanted to avoid is telling people "oh no your string never terminated."

You're right about reference counting being able to pause. But that's something you're more able to control. If you're doing game dev, you can say "I won't assign to references that might be big" during frame critical code.

With garbage collection, you're more or less "Run the gc before, hope it doesn't trigger during that critical spot." or the alternative of fiddling with the gc/manually firing it off so it doesn't stop your world when it's a bad time to stop the world.

> print($"My name is ^(@name).") > > I don't understand why anyone would still require special notation for interpolated strings, when we have the awesome and unambiguous string interpolation syntax introduced by Swift (granted, this project was started in 2011, before Swift).

Why should the swift syntax be considered the default now? "My name is $blah." has been the de-facto standard for ages.

Personally I like interpolation sigils since in most cases you don't actually use it, and languages like Scala allow you to define custom sigils with custom behaviour. For example, Slick defines a sql"SELECT * FROM table WHERE username=$username" syntax that does not cause SQL injection issues.

Swift string interpolation syntax is not more awesome or any less ambiguous than other syntaxes used in other languages

In fact, it's the first time that I've seen interpolation with round parentheses, rather than curly braces (which are used in javascript with $ (es2015 template literals), python, bash with $, ruby with #, groovy with $, etc... )

> LoL. Refcounting can lead to just as long pauses as GC

He didn't say more performant, he said consistent

> LoL. Refcounting can lead to just as long pauses as GC, if your reference is the last reference to a huge object graph.

Laughing out loud is fine I guess but it adds nothing to this discussion as that paragraph would easily stand on its own or even be better without.