Notes on the Crystal Language | Hacker News Reader

Notes on the Crystal Language

wiki.alopex.li

167 pointstodsacerdoti67 comments

> I still think the world really needs a solid, immutable-first and functional-first scripting/glue language, and Crystal isn’t that.

The author might be interested in Roc [1] by Richard Feldman et al, which seems to check all the boxes, including scripting with type inference and glue with good FFI to other languages. It is still in early stage, though, since it is a very ambitious project.

[1] https://www.roc-lang.org/

It seems that Clojure ticks all this boxes

The JVM is a pretty heavy dependency for scripting/glue, if just for startup times.

Babashka it when its done if it depends on libraries not in Babashka?

JVM is a hard dealbreaker for a scripting/glue language.

See Babashka.

... or most Schemes/Racket

I got to work on a crystal project years ago and loved it. I was exposed to a bunch of people who were really into crystal and it made me come to the completely incorrect conclusion that crystal was up and coming and I’d be able to find more work using it.

I was wrong. I’ve never encountered it again. It was a great language, super productive. I helped some people develop a pet insurance service. So weird. I’d work with crystal again for sure.

Did you have to work around some rough edges of the language in that job?

To me it feels like it never quite reached maturity, although in a better world it would have gotten more support and conquered all territory that is now ruled by the Gophers.

I guess to make a successful programming language, you either need corporate backing or extend an existing one nowadays, otherwise it's really hard to build a stdlib of modern proportions.

As I recall the rough edges were a lack of tutorials and docs at the time. There are some “weird” things about the language which are totally fine if you know what they are, but it’s brutal if you don’t have a quick reference to make sense of things.

It has been a while so I can’t recall very clearly, and I have a feeling the community and docs around it are all much better now.

> it's really hard to build a stdlib of modern proportions.

it would surprise me if the stdlib was the issue. you can clone the ruby, python, or go standard libraries, and that would get you into a good state pretty fast; those languages have already done the work to assemble a standard library that people seem satisfied with by and large, and translating code into a new language is far easier than coming up with what should go into a library and then designing and implementing it from scratch. still takes time and effort but there's a clear roadmap if you go that route.

I suspect the real weakness is the thousands of third party packages for stuff that is too specialised or niche to be in the stdlib, but when you need it you really need it. (e.g. I have recently felt the need for an implementation of the blossom algorithm for graph matching in elixir; in python or ruby it's already done for me and available as a package)

I loved writing straight Ruby. It is by far my favorite scripting language. It makes me sad that Ruby on rails was the one big thing done in Ruby and that that gave it somewhat of a bad rap.

Were it not for Rails, most people wouldn't have heard of Ruby. It is unfortunate that Ruby doesn't get more general computing use, but Rails is really just the tip of the iceberg in a big project that uses Ruby. (You still get to do a lot of nice work that isn't Rails specific, at least if you structure your system well.)

We use Crystal extensively at https://heiioncall.com/ for the high-performance stuff, and it’s terrific. :)

As the author notes: the typing system is very pleasant. And the standard library is great!

Generics work fine.

The testing story is very good (almost as good as rspec).

Have some experience with the Threads but won’t comment extensively as that’s about to get some big changes in the next minor release or two.

Overall: highly productive for high-performance backend API servers and background task daemons. Way more fast and pleasant to write than Rust or Go (IMHO).

> As the author notes: the typing system is very pleasant. And the standard library is great!

The author complained about the standard library, specifically the Dir module using inconsistent types and the implementation details of Temp files potentially leading to security flaws, leading me to think there are more warts he didn't cover.

Temp file issue is debunked here: https://news.ycombinator.com/item?id=41684848

Dir module works fine, though maybe the docs could be improved. https://crystal-lang.org/api/1.13.3/Dir.html . The stdlib code is also highly readable: https://github.com/crystal-lang/crystal/blob/d14d04562/src/d... shows that #each_child just calls #read and yields it to the block, so I don't think this is really a wart :)

I think the Dir issue wasn't that documentation was inadequate or stdlib code was hard to read, but he expected paths, and received strings instead.

How bad are the compile times? My understanding is that the compile times are currently bad, and they can't be improved without fundamental changes to the lang/type system.

It depends a lot on compiler flags, especially "--release". See https://crystal-lang.org/reference/1.13/man/crystal/index.ht... : "Compiling without release mode is much faster and the resulting binaries still offer pretty good performance. Building in release mode should be used for production-ready executables and when performing benchmarks. For simple development builds, there is usually no reason to do so."

For our API server https://api.heiioncall.com/ , on my Ryzen 5950X, with a warm CRYSTAL_CACHE_DIR ("where Crystal caches partial compilation results for faster subsequent builds"), making a one-line change:

Without --release:

    6.11user 2.37system 0:08.19elapsed 103%CPU (0avgtext+0avgdata 590844maxresident)k

With --release:

    76.54user 5.49system 1:25.16elapsed 96%CPU (0avgtext+0avgdata 696876maxresident)k

So let's call it about 9 seconds while developing, or 90 seconds for a CI build with full release optimizations.

Than you so so much!

> Can you magically return/break out of iterator functions like you can in Ruby?

Sure can.

    $ cat break.cr
    (1..).each do |x|
      break if x > 2
      puts x
    end
    
    $ crystal break.cr
    1
    2
    
    $

The answer to Crystal questions that end in "like you can in Ruby" is usually "yes." The language really is just Ruby with static typing, warts and all. Most random snippets of Ruby will run just fine when compiled as Crystal.

I agree this is weird, but I think it's largely an analogy issue: Ruby and Crystal both explicitly document blocks as not being closures, even when they can be converted into closures. They're their own weird thing.

Crystal also documents that it will fail if the block contains return or break statements[1].

[1]: https://crystal-lang.org/reference/1.13/syntax_and_semantics...

I'm a core developer of Crystal. Note that there are two different flavours of blocks in Crystal: inlined and captured ones. `return` and `break` are supported in inlined blocks.

The bit about iterating over a directory struck me as a bit of a misunderstanding and therefore the rant following it seemed a bit misplaced. The code example makes no sense at all, so you can’t really make sense of the argument being made.

Without that, it just looks like the author has an insatiable love affair with rust and their arguments lose credibility. Of course if they’re trying to smash Rust semantics into any other language they’re going to be disappointed.

nothing against the language but it's a bit suspicious that the top sponsors include "buy Google reviews" and "buy Instagram followers" (if you navigate to the actual Crystal language site)

Oh... and only for 750$ a month. The other sponsors pay more, but have to appear alongside those.

That's an interesting form of guerrilla advertising. Find a project with decent website traffic that displays sponsors, and commit just enough to be displayed in the list of top contributors (which includes some delicious SEO-friendly backlinking juice too). $750/mo and it's already getting the companies mentioned in the currently-top comment of a thread on the front page of HN!

I'm not sure there's a lot of SEO-friendly back linking juice. They're marked as rel="nofollow sponsored"

I'm a core developer of Crystal.

We've been wondering about why it's worth the money for these sponsors. The links ideally shouldn't have much effect because they're marked as `sponsored`. Anybody got any ideas?

Either way, it's money that helps the project so we're fine with accepting it.

That's a bit of an indictment on how poorly projects are supported. If those sponsors feel like the eyeballs they get from supporting this project is worth it, probably at least some of the owners of those eyeballs are getting enough value from the project to overbid.

Is it an artifact of the Individual/Organisation split of OpenCollective? I see the top 10 individuals that have put in $1000 or more.

I'm a core developer of Crystal.

All sponsorship contributions pay some development hours, so they're valuable to improve the project.

Sponsors presentation is only based on their _current_ contribution, not the accumulated total.

How does the team feel about selling Google reviews and Instagram followers? How do 84codes and Manas.Tech feel about being placed alongside these sketchy sponsors?

I get that money is hard to come by, but I hope you realize that you might be doing more than $1500/mo in damage to the project by accepting these ad placements.

This is a problem we have with the current page layout. We agree that these need to be clearly marked as ad, instead of legitimate sponsors.

Quite frankly, this is something new to us (ads paying enough to be there), so we're still figuring out.

But you don't see a problem with accepting an ad placement that is legally and morally dubious? The FTC is moving to ban these outright, and their justification is 100% valid:

> Fake reviews not only waste people’s time and money, but also pollute the marketplace and divert business away from honest competitors.

Labeled as an ad or not, you may well lose sponsors over this who don't want to be associated with that kind of behavior, and you're certainly losing potential users and contributors here on HN. My perspective on Crystal has changed from positive (I always love new programming languages) to negative over the course of this conversation, and I'm sure I'm not the only one.

This damage can't be measured as easily as money flowing in can, but it's very real and you shouldn't discount it. Some of us don't believe that receiving money for something is sufficient justification to do it.

https://www.ftc.gov/news-events/news/press-releases/2024/08/...

I've come across some legitimate open source projects where all the listed sponsors are dodgy gambling sites, I doubt they actually use the tech, it's just a way to get relatively cheap backlinks from a legitimate website.

Not sure what you're supposed to do as the project maintainer, might also be a bit strange to selectively only display some sponsors that you think are "trustworthy"?

I know it's easy to say from the outside, but a project is under no obligation to accept a sketchy sponsor. If someone offered money to post a link to something obviously illegal, it would be illegal to accept that deal. If they offer money to post a link to something dubiously legal, it is likewise dubious to accept that deal.

That they're starving open source developers doesn't remove the obligation to vet their sponsors before posting their links.

In my opinion, if it gets interesting projects funded I'm all for it. Otherwise, who knows if this language might get the support to thrive.

What's the performance like?

The comment at the end

OO language with a static type system that feels as low-friction as a dynamic one

Sounds nice, but does the performance act like a dynamic language or is it super speedy? I'd be happy with half the speed of C.

I'm still on the lookout for a language that feels as low-friction as a dynamic language but could do some form of iterable.map in parallel on all cores transparently.

There's a blog post on paralellism in Crystal https://crystal-lang.org/2019/09/06/parallelism-in-crystal/ but that's a fair few years ago now.

Super speedy. It is in fact about half the speed of C. It has a GC so that’s probably the main slow down, but hey, no memory errors or raking your brain over a barrow checker.

The GC in particular is excessively slow. It's not a production implementation. I don't need JVM level GC engineering everywhere but it really is painful.

https://en.m.wikipedia.org/wiki/Boehm_garbage_collector

What use cases do you see that in? And do you mean throughput or latency is slow?

Web server with DB backend.

Pause times can exceed 2000ms. No doubt some code changes could help but the fact remains the GC is not an optimised one.

I'm a core developer of Crystal. Looks like something went very wrong there. The GC may not be super optimised, but it's still practical. I have never heard about such drastic performance issues. And I'm aware of quite a few companies who use Crystal in heavy production loads for exactly the web server + db use case without such issue reports. So I'd suggest the root cause might be something else then the GC implementation.

What is roughly the gc algorithm?

It is using Boehm/libgc. Just a simple webserver should not have the described behavior. The GC is not incremental though, so having a big heap would trigger it. But that is typically not the case for the described use case. Likely the issue is with doing something that involves more allocations than necessary.

There are works in libgc to allow incremental collection, but it is not yet ready for the needs of crystal (or at least it wasn't the last time I investigated).

Can crystal use a moving GC or does it suffer from the same issues python has with C FFI?

Also, is it possible to use RC with crystal?

Yeah moving objects would invalidate pointers passed to external code that's not controlled by Crystal.

Conceptually, I think the correct time to do garbage collection is when your web server process is idle.

My Crystal implementation of idle-time garbage collection is here: https://github.com/compumike/idle-gc though please note that its idle detection mechanism only works for single-threaded Crystal programs.

An analogy is to imagine a single employee (thread) operating a convenience store. If there are customers waiting in the checkout line (latency-sensitive requests), the employee should priorities serving the customers ASAP! But once the line is empty (thread is idle), that might be a good time to start sweeping.

Right now, with automatic garbage collection, the employee only decides to start sweeping the entire store while in the middle of serving a customer! (Because that's when mallocs are happening, which may trigger automatic GC.) Pretty ridiculous!

With idle-time GC, the sweeping happens entirely or mostly while there are no customers waiting. This may not show latency improvements in an artificial benchmark where the system is running flat-out with a full request queue, but in the real world, it changes GC from something that happens 100% of the time in the middle of a request is being served (because that's when mallocs happen and trigger automatic GC), to something that only rarely or never happens while a request is being served.

Even better would be to combine idle-time GC with incremental GC, so that the employee could put down the broom when a new customer arrives without finishing sweeping the entire store. :)

See also "Idle Time Garbage Collection Scheduling" in Google Chrome (2016): PDF at https://static.googleusercontent.com/media/research.google.c...

10x compared to Ruby for my use cases.

> I'm still on the lookout for a language that feels as low-friction as a dynamic language but could do some form of iterable.map in parallel on all cores transparently.

If you can get past the typical "ew parenthesis instead of curly-brackets" and prefix/Polish notation, Clojure is great for this. You'd normally do `(map a-func my-elements)` but if you want the map processing to be parallel, you'd change `map` to `pmap` without any other changes needed.

I'm a core developer of Crystal. Crystal's dynamic nature is only in the source code. It generates highly optimized code thanks to LLVM codegen. So performance is generally comparable to other compiled languages like Go, Rust or C. Of course there are differences depending on specific use cases. A workload with lots of memory allocations means a high workload for GC which could be less efficient than an implementation that uses no garbage collection. But even that depends.

It's possible that the language you're looking for is Julia. https://docs.julialang.org/en/v1/manual/parallel-computing/

I know almost nothing about Crystal, so I can't make a fair comparison between them. Julia is used in scientific computing, where parallel clustered processing is essential.

I don't like subthreads which hijack a Language A post to talk about Language B, so I'll resist making an elaborate pitch, but Julia does fit the bill for what you're on the lookout for.

As far as a REPL for Crystal goes, the interpreter is available via "crystal i"

Also for finding shards/libraries/documentation, I'd recommend checking out https://shards.info/, https://shardbox.org/, and https://crystaldoc.info/

The awesome crystal repo is also a good: https://github.com/veelenga/awesome-crystal

I work with Ruby every day at work & have always been curious about Crystal. This was a very helpful dive into some overarching concepts & small details. Thanks for sharing!

> Oops, looks like its tempfile lib is written in library code in the style of mktemp(3), instead of calling the OS’s mkstemp(3) or equivalent. So it just finds a filename that doesn’t exist and then opens it separately, letting an attacker create a new file which they can read at that location between those two steps. Uh, have fun with your temporary files leading to exploitable race conditions guys.

Good catch. The API should be opening the tempfile atomically (the best option), or at least should use a CSPRNG for its random suffix. It doesn’t, currently.

I don't think that complaint is even true. A quick glance at the Crystal stdlib source[1] shows that the mktmp implementation uses the O_EXCL mode, which will cause the open call to error out if the file already exists, and the implementation here will handle that and try again with a different filename.

I think the worst thing an attacker could do here if they could somehow predict the generated names before the file is created is DoS the mktmp call by racing ahead of it 100 times until it gives up.

[1] https://github.com/crystal-lang/crystal/blob/release/1.13/sr...

Also, good catch! :)

Crystal is beautiful. I wish I had a use case for it.

Seconded. My use case might be running my own Ruby scripts faster, but they don't need to run any faster so it's not worth the trouble of setting up a development environment, especially one that requires a compilation step.

It would be a totally different thing if it could run Rails, which is one of the things I do in my job. The speedup would probably justify using it, especially when running tests on my laptop. My customers are perfectly happy with the performance of Rails on their servers. Same with the ones using Python with Django.

> It would be a totally different thing if it could run Rails, which is one of the things I do in my job.

At https://heiioncall.com/ we combine the two and use Rails for the human-facing parts, and Crystal for the machine-facing parts (inbound API server calls, and outbound HTTP probes). For us, the number of machine-facing requests per second is so many orders of magnitude higher than human-facing requests that Crystal's performance and lower footprint is valuable there. While on the other hand, the conveniences of Ruby on Rails are still great for a conventional human-facing web and mobile application.

Yeah,I've found that for the vast majority of Rails apps, the bottleneck is the database, not Ruby. Even a 100x speed up in the application code would have very little end user impact on performance in most cases.

I would see plenty of use cases as a Micropython substitute if at least a subset of it was ported to microcontrollers.

I spent some time experimenting with the Crystal language, and it seemed very promising. But the absence of capabilities for parallel programming (as of 2022) made me abandon the attempts to learn better Crystal, as it felt pointless.

I've tried Crystal half a dozen times but always ran into some inconsistency or bug that became a show-stopper. Sigh. I hope maybe the next time it will be reliable and usable.

“Oh it’s nice but it’s not Rust, why everything can’t be just like Rust”

The audacity of Rustaceans is so thick.

Reply on news.ycombinator.com