Hacker News Reader

Comment by losvedir | Hacker News Reader

Oh, hey, didn't expect to see this here. Thanks for submitting! I tried a few days ago but didn't really get any traction.[0]

Since I submitted it, though, I posted it to /r/rust and got a lot of feedback. At the time, rust and dotnet were comparable and at the top of the list, with ~10k req/sec. Now rust is far and away the most performant at ~20k req/sec! I also was able to improve Go's performance 30% or so. Still, I want to let the other communities chip in and see if I can improve them.

I was actually just in the midst of exploring how to improve Elixir's results. I'm finding I can almost double my requests per second from simply switching the JSON encoder from Jason to Jiffy. That sort of surprised me since Jason is the de-facto standard, and I thought was super fast.

[0] https://news.ycombinator.com/item?id=33291604

View on HN

Few tips to improve C# performance:

- enable source generation for JSON serialization

https://learn.microsoft.com/en-us/dotnet/standard/serializat...

- use `TryGetValue` to do single dictionary lookup instead of two.

  if (TripsIxByRoute.TryGetValue(route, out var tripIxs))

- specify List's capacity, if it's known when creating it, to avoid resizing operations while adding elements to the List

  var schedules = new List<StopTimeResponse>(stopTimeIxs.Count);

> I'm finding I can almost double my requests per second from simply switching the JSON encoder from Jason to Jiffy

Personal plug, but have you tried Jsonrs[0]? I typically get much better performance out of it (especially in lean mode, which seems to be the mode you'd want to use for this benchmark) than Jiffy for large JSON encoding workloads.

[0] https://hexdocs.pm/jsonrs/readme.html

This is great! Just pushed up a commit that uses it and updated the benchmarks[0]. I'm seeing a 1.6X - 2X improvement in overall performance. Not bad for a drop-in replacement. And since it's based on serde, I trust it, and I feel like trying out a different JSON library is within scope for me of not just "gaming the benchmarks", as this is actually something I'd now consider using at work.

It's not quite as high as I was seeing with `jiffy` (3,800 req/sec here vs 4,000+ with jiffy), but I'm not confident that was a totally fair comparison. `jiffy` doesn't integrate as nicely with Phoenix, so I was just calling `:jiffy.encode(...)` in the controller and then doing a `text(...)` response. I need to double-check if `json(...)` is doing more work here.

[0] https://github.com/losvedir/transit-lang-cmp/commit/140d693b...

One major difference you'll run into here is support for encoding protocols. Jason, Poison, and Jsonrs (by default) will have protocol impls for Elixir structs that define special logic for how to serialize those structs into JSON strings. ie, you probably want a DateTime struct to serialize as a date string rather than a map containing the year, month, etc. Jiffy doesn't support any of that, so you end up with differing behavior between it and other libraries as soon as you start encoding structs that aren't meant to serialize as maps.

Jsonrs supports encoding protocols by default, but lets you turn it off for a speed boost if you're okay with more Jiffy-like behavior. Plugging it in as Phoenix's JSON library won't turn off that protocol handling, so you're getting the slower operational mode of Jsonrs (which is fine and probably what you want in most cases for correctness, but will definitely eat a few ops/sec in a benchmark)

For anyone interested about the “posted it to /r/rust and got a lot of feedback”: https://old.reddit.com/r/rust/comments/ya4xfw/why_is_cdotnet...

The elixir implementation seems super slow. The first thing that jumps out to me is that we should be using streams instead of just reading the file. The second would be the `schedule_for_route` which should just be a GenServer lookup instead of reading from ets.

Agreed. I've seen significant performance improvements by using streaming with binary pattern-matching to parse files rather than loading them into memory and using String functions to get at the data.

These Elixir numbers seem suspect to me, based on my company's extensive benchmarking on many of these same languages. Elixir is somewhat slow at some tasks, but serving up web requests isn't one of them.

Yeah, I should benchmark streams. In my experience streams are rarely faster, though, and are instead something you should reach for if 1) you can't fit the file in memory or 2) you might stop the enumeration early.

Regarding GenServer, I'm not so sure about that. I suppose I should benchmark it, but intuitively I expect ETS to be better here. There's more overhead in getting the data, but it allows you to concurrently read it from different processes. A GenServer, meanwhile, could respond to a given message faster, but now you're serializing the (e.g.) 50 concurrent virtual users through a single bottleneck. I could have multiple copies of the data in several GenServers, I suppose, at the expense of much more memory use.

I would guess that there's too much work being done at runtime. I would expect all of the data to be cached in such a way that it is a read operation without any transformation. It's doing multiple lookups and unnecessary maps when the data should just be formatted a single time and cached. You could even skip the JSON transformation by just doing it once if you want to get super fast.

Using stream can be faster here. Because you don't have to rebuild the same huge list again and again.

Regarding C#:

> The "billion dollar mistake" is important to me, and while C# has non-nullability sugar in its typesystem (i.e. with ? after a number of types), the type system wasn't as rigorous as I was maybe hoping.

I'm genuinely curious if you have any examples? Nullability checks may be bolted on, but once you enable them, they should consistently prevent you from dealing with any null values that you haven't explicitly allowed. And other than that, the only hole in the type system that I can think of is array covariance, which was unfortunately common at the time (Java had it also) as a way to skimp on generics, but which doesn't occur often today because using generic collections is much more common.

> At one point I had a bug because I did a stopWatch.Elapsed / 1000 by accident instead of stopWatch.ElapsedTicks / 1000. The former is a TimeSpan struct instead of a long like ElapsedTicks, so intuitively it feels like I shouldn't be able to divide it, though it did a best effort and did something to it, though I'm not quite sure what.

This particular one doesn't have anything to do with the type system per se, it's just the way TimeSpan itself works. I'm not sure why "intuitively it feels like I shouldn't be able to divide it", since there are fairly obvious definitions of arithmetic operations on spans and numbers - if T is 10 seconds, then surely T*2 is 20 seconds, and T/2 is 5 seconds? Which is exactly what TimeSpan does by overloading the corresponding operators:

https://learn.microsoft.com/en-us/dotnet/api/system.timespan...

Note that the type of (TimeSpan/double) is still TimeSpan, so the type system is still enforcing proper use - it wouldn't have let you assign it to an int or a double. But it looks like you were just printing it, which is legal for any type:

   Console.WriteLine($"loaded stop_times.txt in {watch.ElapsedMilliseconds} ms");

> I'm genuinely curious if you have any examples? Nullability checks may be bolted on, but once you enable them, they should consistently prevent you from dealing with any null values that you haven't explicitly allowed.

I’m not the one you asked, but there’s a difference between you no longer having to deal with null checks and your program no longer having to deal with null checks.

The CLR will still do the null checks and fail your program fast when needed. Other languages will require you to properly handle them (or at least, make it obvious where you chose to bluntly abort the program when they occur)

I also don’t think C# will prevent you from forgetting to do a if null check in cases where you expect values to sometimes be null.

You're describing C# as it was before nullability tracking. For several years now, it does have a mode in which every reference that can be null must be explicitly checked to be non-null first before you can use it, very similar to Kotlin.

https://learn.microsoft.com/en-us/dotnet/csharp/nullable-ref...

So yes, "will require you to properly handle them" is exactly what I meant.

FTA, emphasis added: “Nullable reference types refers to a group of features enabled in a nullable aware context that _minimize_ the likelihood that your code causes the runtime to throw System.NullReferenceException.”

That approximates it, and likely gives you most of the benefits, but it isn’t the same as strict enforcement of necessary null checks.

There will be cases where the programmer can prove a value isn’t null, but the compiler can’t, and thus, the programmer can choose to ignore the warning.

That’s similar to how Java tries to prove that a variable is initialized when being read. (https://docs.oracle.com/javase/specs/jls/se9/html/jls-16.htm...).

Java makes the safe choice, though. It refuses to compile such programs, thus guaranteeing that every local variable and blank final field will be initialized before first use.

That's exactly why I asked for specific examples of when nullable enforcement does not catch a potential null reference at compile time. I can't think of any such case so long as all the code has it enabled (if not, it's like safe Rust calling into unsafe).

And if you ignore the warnings, that's a deliberate opt-out; why even bother with nullability checks then?

> I also don’t think C# will prevent you from forgetting to do a if null check in cases where you expect values to sometimes be null.

It should throw a warning in situations where a non-nullable value is not clearly non-null when it’s dereferenced. So you won’t get a warning in the body of a method declaration with a non-billable parameter, for example, but you will get a warning if you attempt to pass null into it.

Are you on OTP 25 ? With the JIT, in theory Jason is faster than Jiffy

Another thing to add, do you log things in the other languages ? If not, what you are probably benchmarking there is how fast your console is in receiving the text (with a lock).

Also use a release, it will help some things here, especially on "short" benchmark, and make sure that the JIT is run.

Yep, Elixir 1.14, OTP 25. I set the log level to warn so Phoenix isn't logging anything that I see. Do you think that still impacts performance? I thought Logger compiled out log calls below the level threshold.

Good call on running as a release! I'll try that next.

edit: Ah well, good thoughts but didn't pan out. I updated the logger config to use `compile_time_purge_matching` just to make sure there wouldn't be any logging impact, and I ran the app as a release, but didn't really make any difference in the numbers that I saw.

Did you post on the elixir forum? Those results seem very low, I feel like there has to be something amiss.

I posted to the Elixir Forum

https://elixirforum.com/t/an-informal-comparison-of-several-...

----

It's not surprising that Elixir is not good at large json handling and complicated data structure manipulation. This reminds me of https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...

At the same time I see that Elixir gave the most consistent response time. Max response time is less than 6x of median compared to ~x10 (Deno / C#) or x23 (Go / Rust) / x58 (Scala), probably thanks to preemtive scheduling of per-request process in Erlang VM

Reply on news.ycombinator.com