The Neat Programming Language | Hacker News Reader

The Neat Programming Language

neat-lang.github.io

> I’ve been in the D community for a decade and a half. People keep telling us they don’t like the GC. They keep saying GC is a dealbreaker. They want predictable memory usage and cleanup times. Somehow, none of this is ever an issue with C# and Java. Somehow, the fact that any memory allocator, including glibc’s, can incur arbitrary delays never matters. Well, fine! Fine. Whatever. I disagree with the choice, but as an offshoot of an offshoot, I really can’t afford alienating folks. I’m tired of arguing this point. Nobody has ever said that reference counting was a dealbreaker. So reference counting it is.

I think this is a bit off. It is an issue with Java and C#, but they don't position themselves as a C++ replacement in the same way D does. Even so you still get lots of complaints about missing RAII and GC pauses (witness how well received Go's low latency pauses have been received).

And yes glibc could introduce arbitrary delays, but it generally doesn't. Allocation is way faster than GC.

Maybe he was talking about long pauses when you destroy a big C++ object (e.g. a big `std::map`)? That can definitely cause annoying big pauses due to deallocation. But he already identified the critical factor - it's predictable. You can fix it deterministically.

Anyway reference counting is a decent choice. It can be very fast (especially if you are only referencing counting big objects and not little integers etc.)

I've come around more to "yeah GC is a problem actually" since I wrote that snippet. It really depends on your usecase though.

Yeah I agree. There are plenty of use cases where it isn't a problem, but in those cases you would probably not pick D/C++/Rust/Zig anyway. Or you don't have to at least (I still use Rust in cases where a GC is fine because it's such a great language).

> glibc could introduce arbitrary delays, but it generally doesn't.

Yeah it does, that's why gamedevs and hfts don't use any malloc in the fast path, glibc or otherwise.

Incorrect. They don't use malloc in the fast path because allocation is slow, not because it introduces arbitrary delays like GC pauses do. To be clear:

* Not allocating (or stack/bump allocation): extremely fast, completely deterministic.

* malloc: pretty fast, in theory arbitrarily slow but in practice it's well bounded

* GC pauses: very slow (until Go anyway)

If you can avoid allocations where speed matters, then the GC won't slow you down either, as (at least in D) it cannot be triggered if you don't allocate.

The difference is that it's deterministic. If you avoid allocation in C/C++/Zig/Rust then you know nothing is going to slow you down, and at worst you have a couple of allocations that won't cause a big spike.

With GC you can try to avoid allocations, but you might miss a couple and get occasional frame stutters anyway every now and then. Also avoiding allocation is a whole lot harder in languages that use a GC for everything than languages that provide an alternative.

The real solution for games is explicit control over GC runs - tell it not to GC under any circumstances until the frame is finished rendering, then it can GC until the next frame starts. I assume Unity does this for example. Still, games are only one application where you don't want big pauses - one that happens to have convenient regular times when you probably aren't doing anything.

> The difference is that it's deterministic. If you avoid allocation in C/C++/Zig/Rust then you know nothing is going to slow you down, and at worst you have a couple of allocations that won't cause a big spike.

> With GC you can try to avoid allocations, but you might miss a couple and get occasional frame stutters anyway every now and then. Also avoiding allocation is a whole lot harder in languages that use a GC for everything than languages that provide an alternative

Something that I think doesn't get the attention it deserves in Rust is how explicit it makes allocations at the type level. You don't have a pointer that maybe points to the heap and maybe to the stack (or in the case of GC maybe even is on the stack right now but won't be when you change how it's used later) or a slice that you need to track down whether it originated as a reference to a fixed-size array or if it it's dynamically allocated; you have either type that you know is a reference like &T or &str or a slice, or you have a type you know was allocated on the heap like a Box or an Arc or a String or a Vec. I've seen people need to spend a lot of time profiling projects in other languages to track down where they can optimize their memory usage. I wonder if people who are skeptical of languages with expressive type systems might see the benefits more if they were presented less in terms of what the type system provides you directly, but in terms of what it makes available for tooling to take advantage of; in the case of using the type system to track allocations, the advantage might seem limited to your own code and not helpful when handling dependencies without being willing to dive into their code, but I think it's easy to overlook that having the information available statically makes it possible for tooling to utilize it, and that applies just as much to dependencies.

In D you have 2 options to deal with that:

- As you described, you can disable the GC to do your thing. That's not the solution I would recommend, but if you call GC.disable then the GC won't collect anything until you enable it back (or if the program is running out of memory).

- You can mark a function as @nogc, and then the compiler will prevent you from allocating anything that could trigger GC allocation.

There is some level of support for @nogc in the language and few libraries to help, the issue is more in the standard library which relies a lot on the GC.

> GC pauses: very slow (until Go anyway)

If by "slow" you mean "high latency", then this isn't true. Plenty of GC algorithms provide guaranteed sub-millisecond pauses.

It wasn't the norm until Go focused on it though, and "sub milliseconds" is not a high bar. Malloc is much much faster than that.

> It wasn't the norm until Go focused on it though

It's still not the norm, but the point is that those were available

> and "sub milliseconds" is not a high bar. Malloc is much much faster than that.

Malloc is not guaranteed to return in that time frame. The sub-millisecond latency of these GCs is a real-time guarantee.

> Malloc is not guaranteed to return in that time frame

No but in practice it always does.

It doesn't though, which is why malloc is rarely used in embedded and realtime systems.

Go's GC throttles on allocation if it can't keep up and causes bad tail latencies just as much as GC in .NET or Java or even worse if you have high memory traffic (because Go's GC has far lower throughput).

> The most important thing to me is that you should have fun using Neat. The biggest innovative new language in C-likes, Rust, is primarily built on not letting you do things. Nobody understands what the hell C++ is doing anymore, it’s approaching string theory levels of opaqueness. Go is specifically built on the premise that Google developers are incompetent and must be kept away from any semblance of language power.

I can tell I'm gonna enjoy Neat. The dev understands perfectly what I like and dislike in a language.

I got a chuckle. I'm amazingly productive in Go but I would love something with less training wheels that's not C++ complex or Rust.

> Some languages solve this problem by allowing every value to be null, or nil, or None. Those languages are bad and after this paragraph I will speak no more of them

It keeps getting better

I too have made a language! Hope you find Neat interesting. It's heavily inspired by D, except with reference counting instead of garbage collector. Also, it has macros.

My DConf talk on the language is now up too! https://www.youtube.com/watch?v=nDqlYnS-K2c

Looks fun ... and neat.

But the mere fact that the primary ascribed goal is "fun" suggests it is not be taken seriously for actual work projects.

Oh no good god, Neat is definitely beta. This language has one user at the moment. Do not use it for actual work projects.

(Though I will be proper chuffed if you do. I just can not recommend it.)

To be honest, my goal is to have maybe 10 to 50 users right now. That'd be quite enough for this christmas.

RoR was optimized for developer happiness. Maybe a language optimized for fun could be used for serious work too?

https://rubyonrails.org/doctrine

There's a few languages that claim to be optimized for fun (among other things). One example is Raku. I think -o Fun, is written somewhere.

Haven't had the time to watch that talk yet... but what's your motivation to create a new language? Any drawbacks in D that you're trying to fix (besides the GC being replaced with RC - which many consider to also be GC by the way)?

Yep, here's the big ones:

- proper macros

- proper sumtypes with implicit conversion

- more normal lambdas (seriously, D's lambdas are wild)

- faster compiler with more caching and (maybe) eventually live reload?

- proper packages instead of include path.

And a bunch of small fry like format strings and named parameters.

And yeah, um, GC can be good but the D GC kind of isn't. We use D at work, and we run into GC issues frequently. Neat's RC is a much thinner wrapper around C memory allocation, which is just better imo, much as GC and RC are logically equivalent in some ways.

For the uninitiated what are wild about D lambdas? I'm familiar with C++ lambdas and Python lambdas.

Okay so if you're calling a D std.algorithm function, for instance

    int factor = 2;
    assert([2, 3].map!(a => a * factor).array == [4, 6]);

Then the `!` indicates that you're actually passing the lambda as a compiletime parameter to `map`. But the lambda can access the surrounding context! How does it pass a runtime value at compile time?

So what you're passing is actually purely a function symbol. The way that it gets the stack reference to the surrounding function is that it's actually a nested function. And the way that `map` gets the stack reference to pass to the lambda is that, effectively, that instance of `map` is also a nested function of the calling function.

That's also why you cannot pass a lambda as a template parameter to a class method in D: it already has a context parameter, ie. the class reference.

In Neat, the value of the lambda is the stackframe reference, and it's just passed as a regular parameter:

    int factor = 2;
    assert([2, 3].map(a => a * factor).array == [4, 6]);

Which avoids this whole issue at the cost of requiring some cleverness with refcounting.

This "Just pass it as a name" pattern has a been a complete disaster for D IMO. It was before my time but I think the explanation for why seems to be annoyingly along the lines of "dmd optimizer likes it".

It also encourages people not to think about what the structure of their templates, so you can end up with truly massive amounts of duplication.

Well, D lambdas and Neat lambdas cash out the same at the backend level. There shouldn't be a performance difference. If you're passing the context as an explicit parameter, that should turn out exactly the same as passing it as an implicit stackframe parameter. The difference is that instead of instantiating the template with the lambda, we're instantiating it with a type that uniquely corresponds to the lambda - it's pretty similar in the end.

The original justification was probably that an alias works as a form of specialization.

> proper packages instead of include path

Please elaborate on this. It's something I'm trying to get right in my own language since day one. I see in your language's manual that you have chosen to implement modules as files and packages as directories. I came up with something very similar but special cased the main module file so that the entire module could be contained in its directory.

  my-module/
    my-module.ln

  (import (my-module)); reads my-module/my-module.ln

How do you represent modules in your implementation? How do you handle module loading? What paths do you search? Do you support Linux distribution packaging? This last feature is something I'm interested in supporting, I added system directories to my search path for this reason but I wonder if there's anything else that I need to do.

Yes-ish. Basically, it's sort of like D in the sense that a `module a.b.c;` always corresponds to a file `a/b/c.nt` somewhere. But the places where it looks follow a recursive hierarchy, not an include path: you can define paths to search, but you have to explicitly define import relations between those paths. And that's because it's not really a search path, it's a full-on package graph, where each package is only allowed to see imports from packages that it explicitly lists as a dependency. In D, if you're importing a Dub package, because it's just using a search path, the files from that Dub package can actually import modules from your main project. In Neat, the lookup order is "current package", then "dependency packages", then stop.

So Neat's actual hierarchy is "file [> folder]* > package", where `package` itself is optional in the import declaration: you can write `import package(gtk).gtk`, but you don't have to. This is occasionally useful during bootstrap builds when you want to clarify which version of the compiler an import is coming from: the current running compiler is always `package(compiler)`.

This is all because I've been writing it with something like a package manager in mind from essentially day one.

edit: I'm not looking at distribution packaging right now, because I'm not looking at non-source libs at all. That's something that can come later if it's needed at all.

What do you mean by implicit conversion in the context of sum types? I don't see it under https://neat-lang.github.io/manual.html#types

Oh yeah I didn't actually write this down in the manual: if you expect `(A | B)`, then both A and B implicitly convert to that type.

Type A|B is implicitly constructible from A or from B sounds reasonable. A|A from A presumably an error, or possibly A|A itself is an error. Thanks

Did you attempt to submit PRs to D for the features you have listed?

It would be indeed quite neat (pun intended) to see D adopt them as well, specially sumtypes, Rust went ahead and made enums better, it's quite an expected feature for a language at this point

We're in a DIP freeze right now. But Walter doesn't like macros and implicit conversion. Half the point of Neat is showing D devs there's no reason to be afraid of those powerful features.

> DIP freeze right now

So much good has come of it...

I'm getting quite interested in D... can you explain what's exactly going on with DIP1000 and whatever else you may be referring to?

Are the changes to the language going to make it much better? Break existing code? How long until they finish that, is it nearing completion or just beginning now?? So many questions as someone new to D.

I have no idea honestly, it's all sort of in a state of confusion rn. The big thing coming up is editions, but that's not even slightly codified or implemented yet. I'm curious where it's going as well, I'm very hype for that approach. It could end up really good for the language. Or it could all sort of come to nothing. We'll see over the next years.

Just to clarify for anyone who may misunderstand this: D has sum types: https://dlang.org/phobos/std_sumtype.html

The problem the OP was talking about is that in D, you cannot implicitly use an `int`, say, where `SumType!(int, string)` is expected.

You need something like this:

    alias StrOrInt = SumType!(int, string)

    void takeStrOrInt(StrOrInt s)
    {
      writeln(s);
    }

    StrOrInt value;
    value = 10; // ok
    value = "foo"; // ok

    takeStrOrInt(value); // ok
    //takeStrOrInt(10); // not ok
    //takeStrOrInt("foo"); // not ok

    takeStrOrInt(StrOrInt(10)); // ok
    takeStrOrInt(StrOrInt("foo")); // ok

Even though this is not perfect, it works quite well (I believe it's zero cost to do `StrOrInt(10)` for example, but I'm a D newbie). It's a bit crazy for me to see people creating new languages instead of help improving existing ones because of minor stuff like this. The effort to create a language and a stdlib and a package manager etc. is ridiculously high compared to improving existing languages.

I think you're underestimating the effort cost of D sumtypes at scale. Every individual instance of StrOrInt(10) is cheap, but once you're actually constructing structs in the same line, you end up writing a lot of terrible code like

    publish(TrackingEvent(UpdateTrackerEvent(trackerId, position)));

that just adds visual noise. And the fact that you cannot return out of sumtype apply expressions just makes so many neat idioms impossible. Something like

    Object obj = nullableObj.case(null: return false);

is just fundamentally impossible in D.

So it's not one thing, it's a lot of things coming together. :) Mostly I just realized one day that D was never going to be the perfect language for me, because it wasn't even interested in being that language.

> The effort to create a language and a stdlib and a package manager etc. is ridiculously high compared to improving existing languages.

Have you seen the DMD source code? Genuinely, writing my own compiler was easier than improving DMD.

Hey, I've been trying to not interact on social media much anymore but I just wanted to let you know that this looks like a pretty great (neat :)) little language! I'm very comfortable using Rust so I'm probably not going to end up using it, but my girlfriend is a data scientist who only knows Python and recently tried to start programming a game involving nbody simulation in Python, only to find that for the type of games she was trying to make Python was far too slow, and she's been really struggling to find a language that's relatively close to systems programming level (since she doesn't want to use an established engine) that also uses concepts and paradigms she understands and has reasonable syntax and stuff, and this looks like exactly what she's been looking for! When I sent her the link she got extremely excited, even more so when she saw that you had examples that were making games and using sdl. I hope you keep working on this language because I think it really does hit a nice (if niche) sweet spot!

As a funny side note, I read the part on your website where you talk about why you chose reference counting over garbage collection, and despite your light-hearted resentment of the fact that garbage collection is such a turn off for people, the fact that your language doesn't use garbage collection is one of the major factors in why she might end up using it lol. We decided not to do C# (my first suggestion actually) because since she doesn't want to use an engine, it wouldn't be used just as a scripting language, but for the entire object model and update loop and so on, so GC would be a dealbreaker. Sorry-

(tiny rant mode as a hobbyist gamedev myself) there is actually a good reason for this -- it is much easier to do multithreading when you don't have a separate thread going over all shared memory, and it's much easier to do a soft real time thing like game development if your memory allocation and deallocation stays relatively consistent and predictable, even if it is slower overall. Also, yes malloc may cause variable slowdowns too, but definitely to a smaller degree/varience than the average garbage collector and more predictably since you can control when allocations happen; otherwise there wouldn't be a noticeable difference between using garbage collected and non garbage collected languages for writing large-scale games. Plus, in any case, that's the reason why game developers tend to try to limit dynamic allocation at runtime in the first place.

Did she find the built-in vector types yet? :) Small-scale gamedev is one place where I think this language could shine. Anyway, please also tell her to hit up the Discord (or IRC) for any questions!

> Did she find the built-in vector types yet? :)

Not yet, but I'm sure she'll be overjoyed to! She's so excited about this language, it'll finally make low level small scale gamedev accessible to her! :D

> Anyway, please also tell her to hit up the Discord (or IRC) for any questions!

I will! I'm sure she'll be excited there's an IRC

I’m very curious, did you consider Swift? It’s ARC, has C interop and is easy to read/write.

I'll be real honest I'm personally just not a fan of it (or Apple's development tools in general) so it didn't really cross my mind. But if Neat doesn't work out I'll keep this suggestion in my back pocket just in case bc tbh it does make a lot of sense!

Ask your girlfriend to checkout Nim

Actually after we dismissed C# it was the very next one I suggested! In theory it would've been a quite good fit, so it's a solid suggestion. It just didn't really click for her for some reason, possibly partly because when I was looking through its forums for some information on it I came across some really annoying rants from the language's BDFL, and when I showed them to her it really made her feel like it would be difficult and annoying to interact with the community. But that's just a guess idk

I think the BDFL is okay.

He doesn't have much time for the agents of corporations who through their foundations and charities aim to influence and seize control of such groups with their demands for CoCs, DEI in return for some funding and their choice of board appointees.

Nim has been getting along fine without them and will continue to.

A pleasant surprise seeing it on the frontpage.

Yeah I think I'dve been crushed if this had ended up at like three upvotes, lol.

Thank you for sharing!

I didn't notice any mention of tail calls. Does it support proper tail calls?

Not as an enforced thing. To be honest, I'm personally not too into recursion as a "must have" feature.

Is it ready for benchmarking? D currently sits at the top of https://github.com/jinyus/related_post_gen and it would be interesting to see how neat stacks up.

Ah, I remember I looked at that. https://gist.github.com/FeepingCreature/cd935a209fa2eebd5928... Here's a Neat port I had lying around, I think it's pretty much a copypaste from D? I think it was a bit slower, can't remember why though. Something like 80ms vs 60ms with `-O -release`.

Could a Rust like borrow checker automatically and transparently insert the refcount logic only when it's needed, slowing down only access to variables with multiple owners, which would be a minority?

And perhaps declare some special variables as "resource" to inform the compiler to really enforce the single write owner rule, for "special" external things like file descriptors etc.

IMHO, memory as a tightly managed resource is adequate for system programming, but a hindrance in most other general programming tasks. But the approach has its point for actual resources any program needs to manage, allowing powerful compile time checks.

So make it optional, with the default being automatic memory management.

I'm coming from a D perspective, where you really don't want to enforce anything single-owner-like, because it makes algorithms awkward. Neat does some commonsense optimizations to avoid pointless refcount changes, for instance it internally assumes that all function parameters are managed by the caller and doesn't take an additional reference for them. But at the end of the day, it mostly just refcounts and calls it good enough.

You can define types in Neat that are noncopyable, but it really limits what you can do with them. For instance, nested function references are noncopyable by default to avoid forcing closure allocations.

You’ve basically described Nim’s memory management.

I think lobster does this.

"Compile time reference counting / lifetime analysis / borrow checker."[1]

"Reference Counting with cycle detection at exit, 95% of reference count ops removed at compile time thanks to lifetime analysis."[1]

[1] https://strlen.com/lobster/

Once upon a time, NCR's mainframe language was called Neat (the version I briefly used was called Neat/3 which was a low-to-mid-level language if memory serves. I was writing compilers.) My memories of a language with that name are unpleasant.

This is probably the least useful of all comments, but please think of another name. My quibble with the language has to do with the use of '[' .. ']' pairs. I'm not confident that refactoring will be straight-forward. I could be wrong.

I tried, I failed, now it's too late. Just gonna hope not many people remember that other language.

Not sure what you mean with `[]`?

This construct: string longestLine = [ argmax(line.strip.length) line for line in text.split("\n")]; print(longestLine);

looks problematic to me -- are the brackets indicating scope? an array? something else?

As far as the naming is concerned, you'll probably have to put up with remarks such as "neat code is messy" because that's the way people are with something new. Don't let that dishearten you!

It looks like what python calls a list comprehension. An expression which builds up each element of a sequence, potentially from another sequence.

Yeah that's a macro (`std.macro.listcomprehension`), you can copy that file and change it to be whatever syntax you like. It's indeed inspired by Python's syntax.

Breakelse: When Compiler Developers Get Bored - https://news.ycombinator.com/item?id=37887426 - Oct 2023 (78 comments)

I've been working myself up to this post for a while.

feep’s been working on this for like a decade. Happy to see it get some recognition.

Ah! Technically, this version of the compiler has no continuity with the first one. It's just named Neat too, because I spent like a day trying to think of another name after I got sniped on C*, and couldn't come up with anything, and went "hey, I already have a name."

Neat as in the current iteration of the language, started in 2020.

Is it fair to discount all the false starts and early work prior to 2020? A bit like saying Picasso has been working on a painting for a month instead of his whole life.

I guess it’s more accurate to say that you’ve been an inveterate language designer for a very long time, and it’s nice to see your latest one go public.

Yay!

I perked up at "like D but with built in sum types". is C interop intended to be as good as D's?

It runs on plain C ABI, so you can just define C functions as `extern(C)`, just as you would in D. But you can also use `std.macro.cimport` to import C headers directly. Check out the Dragon demo, https://github.com/Neat-Lang/neat/blob/master/demos/dragon.n... :

    macro import std.macro.cimport;

    import c_header("raylib.h");

And then you can just use (most of) the Raylib functions and types.

cimport is a massive hack: it runs gcc on the header in preprocessor expansion mode, then parses the result. Its tactic for C syntax it doesn't understand is "just skip it and hope for the best." :) Works surprisingly well.

On x86, there was a lot to be gained from having a language specific ABI, because the cdecl ABI was so slow. But the x86-64 ABI, much as I may dislike its complexity, is genuinely plenty fast already.

Neat : Nice :: D : Java ~1.3

Well, the names (which I enjoy) are similar anyway, that's what made me think of the no doubt overstated analogy.

How is it implemented? Via transpilation to C?

Two backends:

- LLVM is a proper SSA backend, it just uses the llvm-c API to generate bc modules, like any other LLVM-based compiler. It technically links with clang, but that's just cause I didn't want to set up my own link driver and optimization pipeline.

- GCC is technically "transpiling" to C, but the generated C files are unreadable, because it's still SSA.

The GCC backend is also used to build the releases. I just build the compiler with the GCC backend, then zip up all the .c files that it produced.

That's why the release build script starts by building a bunch of C files.

Reply on news.ycombinator.com