Hacker News Reader

Comment by Aardappel | Hacker News Reader

Hi, author of the language here.

Not a great time for HN exposure, as I just added a lot of new functionality, but none of that is documented yet :) But hey, all publicity is good publicity?

For those interested, some recent things that have been happening:

- Compile time reference counting. This does an analysis similar to the Rust borrow checker to infer lifetimes, but unlike Rust doesn't make the programmer jump through hoops.

- "Inline" structs: structs are allocated in their parent, and come at zero overhead compared to individual variables, which is particularly useful for Lobster's heavy use of 2d & 3d vector types.

- Multi-threading that uses separate memory spaces, meaning individual threads can run at max speed with no GIL overhead etc.

- A WebAssembly backend is being worked on.

- A lot of optimization work to make the language run faster.

For reference, it has been on HN before, as early as 2013 (https://news.ycombinator.com/item?id=5904430), but it has changed a TON since then (e.g. it was a dynamically typed language, now it is statically typed, with flow sensitive type inference).

View on HN

Hi! I might be an embarrassingly gushing fanboy about everything you do, so I'm probably biased here. (I wrote the FlatBuffers Wikipedia page, for example.) Lobster is pretty inspiring (despite not being nearly as radical as, e.g., Aardappel) with its lightweight lambdas and (like most of your work) a very minimal core language.

Is there a comparison of different versions of Lobster somewhere? I'd like to be able to make statements like "My new language XYZ, unlike Lobster, is dynamically typed, resulting in difficulties such as plicplocpluc in developing the IDE" but different versions of Lobster now cover such a large part of the language design space that this is becoming difficult. For example, that statement doesn't make sense with respect to old versions of Lobster, which were also dynamically typed.

Hah! Thanks a lot for the wikipedia article :)

Yes, Lobster is meant to be a much more pragmatic language, for people like myself to actually get the job of making a game done :) Other languages of mine are much more research-y. Though I will say I think the type system and lifetime analysis are pretty novel actually.

I'd like to think there's really only one Lobster, while older versions are still recoverable, they for practical purposes don't exist anymore :) Have you seen https://htmlpreview.github.io/?https://raw.githubusercontent... ? That actually contains some history.

Holy cow, are you the Quake Aardappel? That is a name I haven't heard in a long, long time...

Yup, the one and only :) Do I know you as well?

Nah, just a forum lurker who had a folder full of mods and maps and engine binaries much too large much too late.

Didn't know how to code back then, and probably can't rocket jump anymore now. Mostly just surprised at having that old memory surface out of nowhere.

Thanks for making that scene cooler way back when.

Hah, thanks!

You should check out Sauerbraten.

Let me take the opportunity to thank Wouter for the wonderful Amiga-E language I had the pleasure to use a long long long long time ago in a very different galaxy. Thanks!

Where's my "like" button.

Would you mind elaborating on the "compile time reference counting"? Is this new? How does it handle cycles?

A comparison to Swift's ARC would also be of interest :)

Yup, I will write up the algorithm some time soon.

I've seen Swift's intentions to add lifetime analysis to ARC, but from what I could see it is much more complicated in Swift, requires annotations, and with significant runtime overhead remaining.

Arc isn’t that stuff, Arc is what it already does today. “Automatic reference counting.”

I'm aware of that, I said "add lifetime analysis to ARC", as in, ARC is what I call regular runtime reference counting, and with lifetime analysis some of it can move to compile time.

Yep oops, sorry.

Anyway, psyched to see more people experimenting with this kind of thing!

Yes, I frankly feel this design space is the future of memory management. GC, begone!

http://concurrency.ch/Content/publications/Blaeser_Component...

This seems to scale out very well - it's kind of static GC analysis with RAII and erlang-style messaging.

http://aardappel.github.io/lobster/memory_management.html

It is so new I haven't even documented it yet.. hence why I said the timing for this HN post isn't great.

So imagine the current implementation does a lifetime analysis somewhat similar to Rust, but then where Rust would error out, this implementation simply inserts a runtime refc increase. This gets rid of most runtime refc overhead without the programmer needing to be clever about it.

Then, I intend to add an optional type annotation that gets you the Rust behavior, for those cases where you want maximum control. I don't think its the right default though.

The algorithm is more lenient than Rust, since I don't have any shared memory concurrency for example.

I intend to write up more thorough details about the algorithm, but again, haven't gotten to it.

It does not handle cycles. If you create cycles that you do not manually break, you'll get a cycle report at program exit that prints the kind of values that weren't reclaimed. You're then expected to fix your code accordingly :)

Any chance you could you share some references on this kind of type inference? It sounds very interesting!

It is not an implementation of an existing paper or anything.. I am going to write it up soon, promise!

Feel free to email or msg me if you want to be notified.

You could try: http://liu.diva-portal.org/smash/get/diva2:20899/FULLTEXT01....

That sounds more similar to Swift's approach, where the compiler does aggressive optimisation to elide reference counts were possible. Do you agree?

Yes, it is more similar to Swift than Rust. I guess people are more familiar with Rust.. since Swift hasn't actually implemented it yet?

https://github.com/apple/swift/blob/master/docs/OwnershipMan...

The compiler has implemented a lot of the infrastructure for reasoning about ownership, and has had optimisation for eliding reference counting operations for a long time (but the ownership stuff makes them simpler).

This at first glance actually looks similar to http://liu.diva-portal.org/smash/get/diva2:20899/FULLTEXT01....

And this style of static analysis has been taken even further by Luc Blaeser: http://concurrency.ch/Content/publications/Blaeser_Component...

The cycle checker I'll have to have a look at - I'd like to crib parts of it for an OS I'm trying to cobble together. Your project looks very interesting, thanks!

Thanks for those, I hadn't seen them. The Ritzau paper indeed appears to be doing something very similar, where in 5.2 he describes "inner references" whose lifetime is bounded by the outer one, I call that "borrowing", where the outer definition (the owner) gets locked from mutation during the borrow.

What I do is a bit more involved though, as for every language construct I indicate if it wants to own or borrow its children (and wether it wants the parent to own or borrow), to remove maximum refcounts, also for cases which are not simply variables. Again, I will describe this in detail.

The second paper I don't see how it relates to what I am doing.

http://aardappel.github.io/lobster/memory_management.html

> I don't think its the right default though.

I think Rust should allow this kind of relaxed ownership, with ARC behind the scenes, as lots of user level code should not require fussing over ownership details. A "get shit done" mode.

Rust supports proc macros, you know-- you can do literally anything with your code. So what you describe is already possible! Make a good proof of concept and submit a proper RFC for it, and it might even be adopted as part of Stable Rust!

Yes, that is exactly my intention, for the majority of code this compile time reference counting is good enough.

You can of course use Rc types in Rust, but that is verbose, and throws all ownership out the window, unlike Lobster.

More on the compile time reference counting here: http://aardappel.github.io/lobster/memory_management.html

Weird, just yesterday, there was a link from a different language that seemed very similar to yours or at least has similar goals: http://cone.jondgoodwin.com/

Not sure if I'd call these similar, Cone looks more low level, with more explicit typing etc. Looks cool though.

Are the coroutines copyable? This is useful if you want to use time-travel netcode (like ggpo) and coroutines attached to game objects in the same game.

I'm afraid they currently aren't, but there's no reason this couldn't be implemented. There's a "copy" function for reference types that currently errors on co-routines.

That sounds great :)

I found it a bit strange that the coroutines run until the first yield at the time they are created, but otherwise I quite like the system.

Do you plan to accommodate multi-line lambda? Python is hamstrung on this front thanks to the white-space oriented formatting.

> Python is hamstrung on this front thanks to the white-space oriented formatting.

You could have multiline lambdas with Python's signficant whitespace, the then-BDFL just didn't think it was important enough to accept any of the syntax options proposed for it or to bother finding another option, believing named functions were sufficient for the multiline case.

f# has white-space oriented formatting.

OCaml doesn't care about white space, and I don't think SML does either, although F# does. Maybe you are thinking of Haskell?

i am sorry. you are right. i misremembered about sml (haven't written in it much and in a while) and incorrectly lumped in ocaml. i should have been more careful. f# was what was really on my mind.

It already does multi-line lambdas (lambdas and built-in control structures share the same syntax), see e.g. the call to "fold" here: https://htmlpreview.github.io/?https://raw.githubusercontent...

  - Multi-threading that uses separate memory spaces, meaning individual threads can run at max speed with no GIL overhead etc.

Hmm? Does that mean that they are actual processes sharing no memory? EDIT: If so then that's a big downside. In Rust you get thread-safety for traditional, shared-memory threading/concurrency. Ideally you should be able to get as good as Rust for threading. Ultimately, I think it all depends on this:

  - Compile time reference counting. This does an analysis similar to the Rust borrow checker to infer lifetimes, but unlike Rust doesn't make the programmer jump through hoops.

I mean, that would be a huge win, but there must be a trade-off. Can you elaborate?

Even if it does mean separate OS processes (which I doubt), keep in mind that on Linux basic POSIX threads are basically just processes with some shared memory mapped in. There's no real performance difference.

This does not mean separate processes, it simply spawns one "VM" per thread. Lobster VMs are pretty light weight (just a self contained object that has everything for a program to run). The advantage is that everything inside one VM can be single threaded, down to memory allocator. The threads then communicate by sending each other lobster values, similar to a "tuple space". I personally think this explicit communication style of concurrency is safer and easier to use than shared memory concurrency.

As for the lifetime analysis, see some sibling comments: I need to do a good write-up of the algorithm. I wasn't anticipating this HN post :)

Ah, OK, sure. And I assume that Lobster compiles to bytecode then, that gets interpreted?

It looks nice! Do you have a gallery of games made with Lobster available somewhere?

There's a lot of great technical info on the homepage, but it's tough for me to see the forest for the trees here. Could you make a short (probably oversimplified; that's ok) case for what Lobster is especially good at and why it's worth using rather than other programming languages? Or direct me to where one is already written?

At a glance, one thing that makes it stand out is that it's minimal like a framework, but allows you to code in something with similarities to python, but with more design towards that task. Most python-esque languages are poorly suited for games in terms of performance/features.

Seems to nail most of the desires of mobile/indie scale development.

I was hoping the current homepage had that function, with a list of special features followed by some examples.

You want it shorter? Something like "Get Python-like ease of use together with static typing and speed and built-in game programming functionality" ? That really doesn't do it justice. In the end, like most languages, it is a particular cocktail of trade-offs, and to see whether this suits you, you need see the full list of features.

Maybe you want a list of features with an example for each? It's hard to make it short though, especially the type system.

Not features. Benefits.

Benefits show an increase in something good or a decrease in something bad.

Better speed. Less code. Easier distribution. Lower cost.

A lot of the benefits are hard to express in numbers, and I am not a marketing person.

Does this particular kind of type system help you in your software engineering? Very hard to quantify, and depends on circumstances and even personal preference.

I don't think I have any hard-number benefits that are meaningful.

Heyy @Aardappel Interesting project!

I wanted to share your language with a game programmer I know but the links to documentation from your homepage look broken.

Essentially https://htmlpreview.github.io/ is dysfunctional. Try accessing the sample links in that project's Github page.

Weird, those links work fine for me.

I guess I'll have to change them, though not sure what to.. annoying github doesn't have a way to view markdown pages.

I guess for now, to read the docs online you could read the raw markdown here: https://github.com/aardappel/lobster/tree/master/lobster/doc... Better yet, just download the repo and read the html files in docs/

Now all docs served from http://aardappel.github.io/lobster/lobster/docs/README_FIRST... which works much better

Looks really interesting. Have you done any write ups on the type inference bits?

Flow sensitive type inference looks like exactly what I need.

The standard work on flow-sensitive higher-order type inference proceeds from Olin Shivers's dissertation, but Oortmerssen has written a bit about his ideas on lambda-the-ultimate.

There's an informal description here: https://htmlpreview.github.io/?https://raw.githubusercontent...

But yes, I should really write up a more technical description some day.

Do you have any plans to support building statically linked binaries like Go?

Currently the only way to do that is via output to C++. This will give you static binaries, but it is of course a more complex build than Go's direct executable output. Not sure if I'll ever do native codegen myself (as in, direct x86 or ARM), currently working on WebAssembly support, and LLVM IR is also a possibility.

Pretty cool.

How does it compare to Dyon?

https://github.com/PistonDevelopers/dyon

Hadn't seen this, thanks! At first glance, it appears that the difference is that Dyon is dynamically typed and requires the programmer to get lifetimes right manually (though annotations, ordering, or clone), whereas Lobster is statically typed and does all lifetime analysis automatically (and falls back to a runtime reference count increase when analysis fails). You'll be able to force the Rust/Dyon behavior manually with a type annotation in the future.

> all publicity is good publicity

... unless it's an obituary :)

(an old joke)

On first look, the language seems superficially similar to Nim. Which, per se, is certainly a plus in my book! ...but makes it somewhat difficult to understand when should I pick one or the other. Could you point out some important differentiators? (Apart from the current set of available libraries, which certainly make it appear more useful for the intended target audience at this very moment AFAIU, but in longer perspective are potentially somewhat secondary.)

It does have some similarities, mostly in the sense that both are using a Python-esque syntax, but that doesn't mean they have similar semantics.

Some difference off the top of my head:

- Nim so far has been a GC-ed language (I know manual allocation is possible, and an ownership system is planned, but this was the default for most code so far). Lobster has been reference counted (and now compile time reference counted) from the start. Nims proposed ownership system is pretty cool, but still requires a fair bit of programmer cooperation, in that sense it will sit somewhere between Rust and Lobster in terms of programmer friendlyness.

- Lobster has a type inference algorithm that is able to go further than most languages in terms of allowing code without types to "just work". Nim has nothing even remotely like this.

- Nim is faster. Nim was designed for near native speeds from the start, Lobster started life a very high level language that slowly become more low level. There's nothing that prevents Lobster being as fast as Nim, it just will take a bit more time.

- Nim is more mature, in the sense that it has been around longer, has more libraries and tooling etc, and a larger community.

- Lobster has great support for programming with higher order functions: terse syntax, best possible type inference, zero overhead (compared to manual loops), non-local returns etc.

So in summary I'd choose Lobster over Nim if you care about Lobsters type and memory management or higher order function features, or if you like the idea of a language that makes games/graphics out of the box easier. If you just want to "get the job done", Nim probably is ahead of Lobster at the moment :)

Lightweight syntax and zero overhead for higher order functions sound fantastic. I'm very interested.

Btw, you have a bunch of dead links to Google+ (readme, the linked page, etc).

On http://strlen.com/lobster/ I only see a dead link to Google+ (which I'll remove, thanks!), but no other dead links?

A very cool domain name, by the way.

Sorry, I meant you link to google+ a bunch, not you link to many different dead things.

That I've noticed: Twice on that page, once in the readme, again in dev/readme and lobster/readme.

Is there a list somewhere of games written in Lobster?

No, most of them are unreleased. Here's one though: https://store.steampowered.com/app/687560/Avoid_The_Monsters...

can you drop the need for the : at the end of the line if there's a newline? i.e. it seems like unnecessary syntax

That would make it much harder to parse in certain cases, both for the compiler and the human, given that it uses this same syntax to parse not just control structures, but also higher order functions / lambdas.

I like how ruby does it

I have never understood why people create entire programming language when just creating a nice C++ library would been more than sufficient, desirable, integratable and adaptable. You can always export bindings for any target language and users would be saved from learning yet another beast that would be unmaintained in few years. I'm not saying never developing new language... If you have radically new thought, go ahead do it but just keep combining feature1 from this language and feature2 from that language doesn't make a lot of sense (especially the part of taking completely unrelated unsearchable word to name your language).

Because the language affects how you think about things, and how productive you are. It matters.

And in this case, this is someone with a 20+ year background of experimenting with new languages, ranging from production-ready (e.g. I used AmigaE a lot back in the 90's) to crazily experimental ones.

C++ is a complex beast and while I quite like working in it, I'm finding that the latest versions (17 and 20) are filled with features that I'm having trouble understanding. The surface for errors is also huge. I've been programming in C++ since about 2002 and I feel quite proficient in it, but, if given the option, I'll use something that is both safer/less error prone and more productive (and I say that as someone whose argued right here that C++11 and beyond is a very productive language).

Beyond that and what sibling comment said, the author is known for creating various languages, so he clearly enjoys doing it and is making this language for himself.

I have never understood why there's always somebody complaining that new languages are unnecessary when a new language is shown on HN. I personally love seeing new languages. I also recently went on a language hunt for something that met a specific need for a side project I'm working on and I came away empty handed because I couldn't find something that met my criteria[1], so I'm always on the lookout for something that might.

[1] In case somebody has any suggestions for me, I want something with Clojure-like immutable data structures as a default (or something similar, at least), is native compiled to reasonably efficient and low footprint binaries (but is not as complex as Rust or Haskell), is a functional language (but with prettier syntax than OCaml, which was almost my choice, but when I started reading example code, its a complex mess of symbols <insert perl joke>). And... it needs reasonable library support... (at a minimum I need to be able to respond to HTTP requests, make HTTP requests, parse and generate JSON and talk to postgres, ideally using SQL and not an ORM). Haskell is probably my best bet, but I don't have the energy to learn all of its complex features.

I'm not against creating new languages. In fact the Lobster language is something I would want (although current syntax is bit repulsive). Python with static types but without getting in the way is pure gold. The problem is that creating interpreter/compiler is one tiny little part of creating language. Much bigger task is creating vast number of libraries that every other popular languages have. Think about everything from numpy to scipy to graphics to networking to web frameworks to OS APIs... Another bigger part is creating tooling and interoperatibility - everything from debugger, refactoring, linting, IDEs, CI, docgen etc. Now put on the top of it all the reusable code that millions of people have already written spending billions of hours in existing languages.

Without large teams working for years, its very hard for a new languages to break in to general use. There are ofcourse outliers and you can get lucky - but then you have to be terribly lucky. So the end result is that 99.9% of languages just die or become hobby project for a person regardless of their features, benefits, beauty or aesthetics. Again, nothing wrong with becoming personal hobby project but if my goal was to create game engines that becomes force in the industry, I would probably not start by creating entirely new language.

I'm not experienced as much as the author, so this is my personal opinion and likely not very well informed. I writing this just because I see several people spending years in creating new languages while their goals have been something else. They dream about repeating success of Java while not realizing how rare it is. After they finish their labor of love, I wonder if they find it depressing to become just line item in the list of dead programming languages. What if they would have actually focused on their original goal instead with emphasis on adding new powerful features that integrates and plays well with existing stuff? It's sure not as sexy and respectable as being father (or mother) of new language but has much higher chance of generating desired impact.

You are not wrong. But you are assuming every language author must be wanting to become the next mainstream language. I am already aware Lobster will unlikely be that, and I am fine with it. I am having a ton of fun developing it, and I have used it for many projects. I do not want to have more impact by working on something more boring :)

C++ is a deeply, deeply flawed programming language carrying decades of baggage to stay backwards compatible. Let's not pretend that mere inertia is enough reason to keep using the same tools forever.

Reply on news.ycombinator.com