C# .NET Core versus Java fastest programs | Hacker News Reader

C# .NET Core versus Java fastest programs

benchmarksgame-team.pages.debian.net

100 pointsthrowaway1333779 comments

.NET Team member here. We don't look at these specific benchmarks. Our belief is that they are wild west w/limited moderation of what is reasonable/real-world and not. Not necessarily a reasonable comparison, but we focus on TechEmpower[1]. You can check out the latest .NET performance improvements in our latest blog post on the topic [2].

[1] https://www.techempower.com/benchmarks/

[2] https://devblogs.microsoft.com/dotnet/performance-improvemen...

I am thoroughly impressed with you what your team has accomplished. Roughly 7-8 years ago, the company I work with switched away from .NET (to Go) largely because our performance with IIS for HTTP traffic running .NET was...poor by comparison to other available options. I wrote a blog post[1] on the subject.

It's been exciting to watch the hard work you and your team are investing to make .NET (Core/Framework/etc) faster. In fact, virtually every single issue or point I raised in my blog post has been addressed to my sanctification and then some. The Techempower benchmarks show the massive strides you've all made and now I'm delighted to see .NET consistently performing at or near the top of the benchmark even without heavy customization/modification. To this day, I still really love C# as a language and there are a number of significant things in C# that I wish the Go ecosystem would adopt. If/when bootstrap a technology stack again, you can bet that C#/.NET will be the top contender on the list of options.

Thanks again for all of your effort. It is very much appreciated and admired, even if I'm not using that tech stack at this point.

[1] https://blog.jonathanoliver.com/why-i-left-dot-net/

Are Go vs C# results there too https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Go being only faster in one benchmark

> Roughly 7-8 years ago, the company I work with switched away from .NET (to Go) largely because our performance with IIS for HTTP traffic running .NET was...poor by comparison to other available options. I wrote a blog post[1] on the subject.

After reading this quote, and as a former .NET developer, I was really interested in the performance issues/story your blog would go in depth about. Instead, I found it strange, as I skimmed your blog, that I didn't see any specific scenario or perf numbers/bottlenecks discussed (seeing as you specifically state your company moved away from .NET due to IIS perf issues) - it was very generalized/hand-wavey. Do you happen to remember anything specific? Or were the reasons your company moved away from .NET literally the contents of the blog post?

It seemed to me that your frustration was more surrounding the MS bloated tooling &/or windows.

Some quotes from your blog:

"Once I used TFS on a small job and it was a monster--just like everything else from Redmond. It infected my project files and contaminated my source directories. Yuck. No thank you. Give me command line git any day...or perhaps SourceTree if you need a little love from a GUI."

You could definitely have used git back then.

"As a .NET developer, when you start a new project there are a few things you'll typically do (by hand, of course):

    - Create a new solution
    - Add it to Team Foundation Server
    - Create a corresponding website entry in IIS
    - Create a new SQL Server database
    - Associate the Entity Framework with the solution (typically for projects created after 2010)
    Start designing your database and your ActiveRecord entities."

You absolutely did not have to use EF or TFS, even back in 2014.

The post ends with:

"And that's why I left the .NET Framework because it kept reasserting itself and wanting to be more than it was: an implementation detail."

In my blog post from 2014 I was mostly illustrating a typical workflow that I've seen over and over and over at numerous Microsoft shops in 2000s all the way through 2014. I was always interested in messaging queuing systems and NoSQL and memcached but when I'd interact with devs from various organizations the answer was "We use SQL Server." I was intrigued because the use cases they were bending SQL Server to handle were more akin to map reduce kinds of scenarios. In other words, the mantra of 10+ years ago was "stay fully in the Microsoft ecosystem, it'll solve all your problems."

Also at that point as we started to use EC2 heavily and the workflow was so straightforward for Linux. The AWS docs would always have caveats for Windows, e.g. "This is how it works...but in Windows you also have to do X and you can't do Y." So that's why we jumped ship.

As mentioned in my earlier comment, I think they've completely tackled all of the perceived shortcomings to the point where the .NET stack is highly attractive and very competitive compared to other stacks.

It's usually the company policy that use TFS for C#, not developers. Usually it's the Microsoft marketing scheme to include .NET solutions with TFS.

> It's usually the company policy that use TFS for C#, not developers.

I agree, but remember we are specifically talking about a company that switched it's whole business off .NET. A company that makes such a decision is probably one that could also make the call to switch version control systems without changing stacks.

Is there a difference in performance between .NET on Windows and .NET on Linux?

Only a few seems to be plain code vs plain code. For the C# code there's use of unsafe, SSE/AVX intrinsics and calling C code to do the job (PCRE for the regex).

Certainly it's nice that you have these capabilities in C#, but at least for me it limits the usefulness of the results.

Handicapping one language to the featureset of another is unreasonable. You'd also have to disable a host of JVM features (such as hotspot) for a "fair" comparison, which is ridiculous.

Calling C is unfair, though. JVM can do the same and it should be implemented the same way.

> Calling C is unfair, though. JVM can do the same and it should be implemented the same way.

If you want to benchmark C code you can just benchmark a C application. You run stuff in a VM like the JVM or CLR because you want things like memory safety. If you allow native code in something like a regex benchmark then all your benchmarks are basically about: How fast can the JVM/CLR call native code. And say absolutely nothing about how fast the VM itself is.

So using native code in these benchmarks is just silly and basically makes the whole thing meaningless.

How fast your managed code can call native code is an extremely relevant performance detail sometimes. It's far from the only thing you should benchmark, but it is a useful thing to benchmark.

Absolutely, but then that should be a separate benchmark, designed to test exactly that.

Same with intrinsics. Have a benchmark that tests how good are the intrinsics, are there any significant overhead etc.

When I look at generic names like "n-body sim" I'm thinking ok, let's see how generic mixed floating-point code compares between the languages.

There's a big difference between "C# is fast" and "C# might be fast if you're extremely reckless". For me only the former is useful.

SIMD is not reckless.

If it can be used safely in production, great, but I'm not seeing that here.

Referring to Hotspot as a “feature” of the JVM is a gross mischaracterization.

A JVM is not Hotspot, and Hotspot is not a part of “the” JVM.

BS! The JVM from Oracle/Sun is called "Java HotSpot Performance Engine" in short HotSpot:

https://en.wikipedia.org/wiki/HotSpot

One JVM is, yes. That’s exactly my point.

Note the `C# .NET Core #5` and `Java #3` programs.

Note the `C# .NET Core #4` and `Java #2` programs.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Yes, C# is currently a more expressive language for performance than Java is, and can score much higher with tighter code. I wouldn't trust the opinion of someone who claims otherwise by ignoring the facts (as demonstrated by your link).

The specific benchmarks in the OP are not objective and fair. I personally want nothing to do with Java, its glacial progress and Oracle's lawyers. But fair is fair.

Using facts to prove a point (as you have) means that quality facts are needed. The OP is not quality facts.

The pidigits benchmark in the OP shows exactly the same facts that I posted.

This reminds me of the canonical scheme benchmarks: https://ecraven.github.io/r7rs-benchmarks/ . I don't think comparing schemes that guarantees safety to schemes that will happily run (car "hello") is very meaningful.

It is of course nice to be able to opt out of safety for critical code, but I don't think most code should run in a way where a type error might corrupt memory. Especially not in a dynamic language.

Look around the benchmarks game website and you might find this reference —

http://www.larcenists.org/Twobit/bmcrock.temp.html

And for C# there are programs that don't use unsafe, don't use SSE/AVX intrinsics and don't call PCRE for the regex.

"all C# .NET Core programs & measurements"

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Just looking at some .NET programs, they all use unsafe code, which is always faster than using garbage collected memory management, so the results aren't surprising.

It's using features of the language. Is that not what this is meant to measure/compare? .NET has unsafe, SIMD, and value types which opens it to a whole world of programming that Java is currently unsuited for.

One would think that a benchmark described in reference to the performance of the virtual machines themselves, would be measuring the performance of pure bytecode (or a JITing thereof), not the performance of native code linked into the runtime.

Using a native regex extension lib is the norm for "scripting" languages so totally reasonable to benchmark like that. How do you define when it's OK to use a native extension and when it's cheating?

You can call C with Java.

For example

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Benchmarking both C# [Managed] and C# [Unsafe] is preferred.

Like this ?

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

If you want performance, you need to refactor your code into a small performance-critical unsafe core (that has higher test coverage and audit standards), while keeping the rest of the business logic in the safe domain, and relying on the CLR to do the heavy lifting and guarantee memory consistency.

Bonus point: you can write the core in C++ and use COM-like marshaling for absolutely unbeatable performance (no actual COM runtime is needed).

Maybe look at the .NET programs that do not use unsafe code?

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Gaming the benchmarks for fun and profit.

Seeing the debian.net domain name, I'm a bit saddened to be reminded that DotNet Core is open source, yet still not available on Debian without installing packages from external sources (like the deb packages MS provides):

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779970

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779969

Right now the official install instructions want you to trust Microsoft keys and I'm not really a fan of that. Further, it installs a custom package repo and in the past I've had very negative experiences with PPAs so I tend to avoid them.

https://docs.microsoft.com/en-us/dotnet/core/install/linux-d...

> Right now the official install instructions want you to trust Microsoft keys and I'm not really a fan of that.

Why?

Because I don't want to trust Microsoft more than I have to. It'd be the only non-distro provided keys on my computer.

I would subject myself to them. Maybe in 5 years they decide that to download, you need a Microsoft account login?

This weakens one of the core strengths of GNU/Linux, that every software installation is only an apt get away. Also, Debian packaging does a level of vetting that the software complies by their packaging policies, which ensures that it is truly libre.

I know that Microsoft has their UEFI signing keys in all the IBM computers around the world and has signed the desktop Linux bootloaders as well, and I respect that they do this, but that's a different part of the stack. I understand that some entity needs to sign the bootloaders, and Microsoft has since shown itself to be a reasonable guardian.

Debian will probably require someone making dotnetcorium that removes the telemetry.

Telemetry removal won't be too hard I hope? For smaller changes, Debian has ways to patch packages already.

    export DOTNET_CLI_TELEMETRY_OPTOUT=1

Everyone just games these benchmarks by calling out to C libraries. It's better to actually compare the slowest programs for each language instead of the fastest ones, because those are usually idiomatic.

> Everyone just games these benchmarks by calling out to C libraries.

Really?

Please say which of the n-body programs do that.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Meanwhile, sort by program source code size —

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

I took a quick look at the code, it's nothing alike.

It seems they have people submit solutions to the problem and compare the fastest implementation for each.

And the fastest implementations aren't the fastest either, I was able to knock about 10% off one of the Java examples from 2010. I suspect the C# folks were a lot more motivated to "prove" that C# was no longer slow.

As others have pointed out, if you want to test C#'s ability to call C-functions in an unsafe way, then fair enough. Otherwise, with memory safety enabled, they are roughly comparable.

That said, I did port the C# version to Java and it added about 20% extra time, even with some of the cheats (algorithm optimizations) it has implemented.

> I suspect the C# folks were a lot more motivated to "prove" that C# was no longer slow.

One person became curious and was brave enough to contribute their programs.

Take with a grain of salt, JVM tuning is an entire art and science, probably same with CLR.

Presumably, these are comparably tuned.

The .net benchmarks use architecture-specific SIMD intrinsics. There's no equivalent on the JVM, though JEP 338 aims to add a vector API.

Maybe I missed something, but there doesn't appear to be a benchmark between JNI, JNA, and pInvoke. Most my experience with .NET/Java has shown it is fast enough in most cases, but then run into areas that need to be optimized into Native Code. This calling of Native "C" code can be slow with JNI and generally just fast enough for .NET pInvoke. I haven't ever had a reason to test JNA.

Unity also shows how you can use C# for high performance game code with their custom burst compiler. They even switch some of their C++ code to C#.

They’re moving away from the managed runtimes and compiling straight to native code — it’s not really comparable to what is being discussed here.

You can feel the sluggishness of every Unity game instantly, not saing it's C# fault, but its not a good engine for fast games.

I thought I remember a post about them shutting it down from a few years back. Did they reopen it, or did I just fabricate that memory?

Who are "they" and what is "it" here?

Oops, I reworded my question and removed the antecedents.

"It" is the benchmarks game and "they" are the maintainers of the benchmarks game.

I remember that too, then suddenly it reappeared. My guess it caused too many flamewars and anger at the maintainer.

https://lists.debian.org/debian-devel-announce/2018/04/msg00...

Shutting what down?

The debian benchmarks website. It was called benchmarkgame.alioth.com earlier if I recall correctly.

I didn’t even realise the domain was different. Been so many years since I’ve seen this site I had no idea it was gone.

Alioth was the name of a machine running FusionForge. Debian replaced it with "Salsa" that's running GitLab. The benchmarks game is hosted in this repository: https://salsa.debian.org/benchmarksgame-team/benchmarksgame

But (without intending to spread unsubstantiated rumor) it seems like it's temporarily unmaintained at the moment:

- in the README.md there is some complaint about being difficult to reboot into a Linux dual-boot, maybe the measurements+website were being published manually

- CI yml seems incomplete

- Some very strange issues like https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...

That issue appears to be someone who was upset with the methodology of the benchmarks and the delay in accepting his/her programs so they resorted to repeatedly changing the title of one their issues.

> … without intending to spread unsubstantiated rumor…

So don't :-)

> … difficult to reboot into a Linux dual-boot…

That's likely to be a problem soon after 1st September.

> CI yml seems incomplete

In what way? "This GitLab CI configuration is valid."

> Some very strange issues like…

What else specifically?

This is really ghetto. Why does Debian allow it on their domain?

You just described most subsites under the Debian domain.

.NET core was slower than mono which was slower than JVM for running real time multiplayer games (3d math, serialization, reading and writing from multiple streams) as of a couple of years ago, did something big change With the runtimes very recently or is this just coding to benchmarks?

The regex-redux one looks like more of an endorsement for the PCRE2 jit.

Like compare `regex-redux #8` and `regex-redux #5` ?

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

These are pretty hilarious.

Did you guys actually read the code in the benchmarks before downvoting me? Both sides are terrible.

Please be specific.

So do Go, C, and many others.

JVM clone edges out JVM. Fantastic.

Reply on news.ycombinator.com