Compiling Rust for .NET, using only tea and stubbornness
fractalfir.github.io
fractalfir.github.io
I'd love to One Day™ Rewrite It In Rust, so that we could write .apk-s purely using the Rust toolchain, just using a JNI library as appropriate, sprinkling the code with some proc-macro annotations where needed by the assembler (for stubs), and possibly adding some lines in a build.rs (for .apk packaging).
The .dex assembler itself is at: https://github.com/akavel/dali — you may like to check out the tests at: https://github.com/akavel/dali/tree/master/tests to see how using it looks like.
An example project with a simple .apk written purely in Nim (NO JDK) is at: https://github.com/akavel/hellomello/tree/flappy (unfortunately, given Nim's poor packaging story, it's most probably already bitrotten to the extent that it can't be quickly and easily built & used out of the box). I recorded a presentation about this for an online Nim conference — see: https://www.youtube.com/watch?v=wr9X5NCwPlI&list=PLxLdEZg8DR...
One can use Uniffi with the C# generator to get fairly automatic bindings. You still need to package it up, which is a bit of a pain.
Uniffi really is an awesome idea. I expect more and more Rust code for foundational shared libraries as a result.
I wish I had this level of dedication at that age...
(I apologise for this being vague, but this isn't really my area so if I tried to get more detailed I'd rapidly go from "quite possibly wrong" to "definitely wrong")
It sounds like Rust's internal MIR intermediate language is a somewhat closer level match to CIL.
Adding a .NET backend to Rust could give you high-level two way interop.
Adding a similar backend to LLVM would let you use .NET target similarly to WASM in that you could compile pretty much any software (C/C++, Go, Rust, etc) and run it on any supported platform without recompiling (well, it would JIT). But you'd have to stick to C-level APIs.
https://ericsink.com/entries/dotnet_rust.html
It seems like it should have higher adoption given the performance boost over JS on Node while being syntactically similar to TypeScript (not hard to adopt for teams already familiar with JS/TS).
Combined with pretty good tooling these days and DX (hot reload is a thing), I'm always surprised by its seemingly lackluster reception.
It's continuously gotten better, but I feel that many folks still have an image of C# and .NET from back in 2003 rather than 2023.
I have some workloads that I'm developing on my M1 Mac that I'm containerizing and pushing into an x64 Linux runtime environment in Google Cloud.
I guess this is my point: a lot of misunderstanding and misconception about the state of C# and .NET.
(If only we could first-class FreeBSD support…)
- EF Core works great. We connected it to RDS Postgres in AWS with container instances running on T4g ARM instances.
- Our pipeline used the `dotnet` CLI to do all the builds on our local M1 MacBook Pros and in GitHub Actions on Linux runners. Want to build a .NET codebase? `dotnet build` from the CLI. Want to run it? `dotnet run`. Want to watch for file changes and hot-reload? `dotnet watch`. .NET is really, really streamlined right now. Better than Node at the moment (as a dev that works fullstack and has workloads on .NET and Node).
- Our team used a mix of VS Code and JetBrains Rider. Two devs started on VS Mac, but eventually we all ended up using VSC or Rider on Mac.
- Bloat is pretty minimal; really easy to build a console app or a web API that more or less looks like an Express web app.
Here's a 7 minute video demonstrating building a .NET web API on an M1 Mac and then pushing it into an x64 Alpine Linux container in Google Cloud Run: https://youtu.be/GlnEm7JyvyY
I suggested C# because of the similarity to TS syntactically while being multi-threaded and more performant for the use case but was amazed at the backlash.
It's interesting to me that many teams that are already comfortable working full stack with TS reach for Go or Rust instead of C#. I can understand Rust if a team is doing systems level programming, but C# is an easier lift than Go (IMO) for teams building APIs and apps who are already versed in TS because the paradigms are very similar (async/await, try-catch-finally, generics, inheritance, tuples, deconstruction, anonymous types, general syntax, etc.)
Noticeable things such as employing hybrid monolithic-microkernel design/kernel modules/SMP/cross-platform capability existed in Windows kernel long before Linux is mainstream. An honorable mention would be the Job Object since Windows 2000, and it is the nephew of Cgroups that is a key component in the Linux Container scene. NTFS is also one of the most successful B-tree based transactional filesystem that has many hidden gems behind.
But Windows userspace things and most infamously Microsoft Office, and Microsoft's hostile and manipulative approach towards end-users makes the whole Windows thing very shitty, while I believe Linux is the other way round, the whole ecosystem is built around free and openness. In the end the tech for building the foundation of the Windows is great but the execution by the upper management is disastrous.
For C#, this is the first language that provided the now ubiquitous async/await asynchronous programming syntactic sugar to the masses. Before that, we have to use something called Begin/End pattern called APM (Asynchronous Programming Model) which is already foreseeing a CPS (Continuation-passing style) based async programming. The async/await pattern, or the state machine + futures approach quickly spread to ECMAScript, Rust and recently C++20.
Don't get me started with LINQ/PLINQ, expression tree (which is a cousin of LISP macro) and extension methods (Rust and D people should know it as mixin/traits and UFCS), which is bringing a life-changing SQL like data operation pattern, which not only runs on SQL (LINQ to SQL) but also for daily life programming. I'm very confident that LINQ predates and heavily inspired Java Stream and Rust iterator design, with Java Stream being a half-assed copy of LINQ and Rust a saner one.
Not only that, but C# lambda is also far better than Java where you can only use final variables in Java lambda, which is based on anonymous class generation (you will understand it once you turn on a Java decompiler), while C# lambda is implemented using something called "delegate", which is an analogy of "function pointer" to C/C++ and is even interoperable as a native code thunk with API marshalling. So, you can indeed store C# delegates into native function pointers and callback to C# from C++ later.
You also have first of its own FFI in C# with DllImport, and with the magic of JIT almost all C/C++/native method can be imported efficiently and directly without so much preprocessing unlike JNI/JNA in Java. No need to manage Java state and stuff in C++, just figure out the right ABI and structs, but those are up to the user to implement which is still closed under C#.
C# also have the right kind of reified generic that does take types into consideration (hence reified) unlike Java which does Type Erasure instead, and eventually making the "generic" type into nothing but a glorified Object and losing a lot of optimizations opportunity due to this loss of information.
Java only recently started to work on value type with Project Valhalla. C# already has it from the beginning.
But, Microsoft is so shitty that it binds .NET Framework to integrate with Windows heavily. They want you to stick with WPF/WCF, IIS, SQL Server and ASP.NET, which is cool in the beginning but barely evolves and is missing what others have in the long run. Plus, the hostile attitude of Microsoft towards open competitor such as Linux from the early days (remember that Microsoft once called Linux "cancer" but now tried to fully embrace it, while I'm not sure when will Microsoft extend and extinguish Linux), with a big contrast to Sun's Java (until the acquisition from Oracle, which today turned to become a hostile and threatening old Microsoft monster). As such, the development and ecosystem of .NET and C# in extension, is often being laughed at as Microsoft shut-in and the developers as Microsoft fanboys. The trend is even more pessimistic when Android adoption exploded, which uses Java as the foundation for software development. So this is why C# development staggered since 2014. There is no competition between Java and C# anymore, because Java already killed it. Barely anyone uses .NET at that point on.
.NET missed a lot of growing opportunities especially in China which is being ridiculed to this day despite many improvements over the years, although this is more of a political choice over technical choice since there are tons of Java developers everywhere in China, most of them are often hostile to C# developers because they are taught not to embrace Microsoft.
While Mono is an exception, and it once exploded with Unity Engine (but now replaced by IL2CPP), it is still a pretty niche market (well, .NET itself is already being marginalized as a niche), and I assert they must have been in the brink of getting sued by Microsoft a couple of times in the past 20 years by copying some key components of the .NET Framework. Still, the relative success of Mono in the cross-platform market is a good attempt into making Microsoft rethink their hostile strategy and is a key catalyst to gradually make .NET open.
So, Microsoft is essentially burying diamonds and gold and jewels upon piles and piles of shit. They totally have the right minds and the capability to pull off the most stunning, impressive and revolutionary tech, but there are simply too many idiotic managers that does little to no coding, with a heavier side on greedy marketing just for the quick bucks, to cockblock and kill off those innovative projects, much like the Google today.
> But, Microsoft is so shitty that it binds .NET Framework to integrate with Windows heavily
However, this is my point: C# and .NET are sorely misunderstood. Yes, the .NET Framework is Windows-only, but .NET Core and beyond are not. For the last 2 years or so, I've been doing .NET exclusively on a Mac M1 and deploying to a mix of Arm64 and x64 Linux servers.
The runtime changes needed to improve the language could not be made easily due to the need to support it for 10+ years.
Mono has been there for a very long time, and even gave life to Unity 3D.
The guy who invented C# went on to invent TypeScript.
I've been working with C# for over 20 years. The big problem with C# in the early days was the ecosystem: an awesome language, awesome IDE, but limited libraries and bloated frameworks.
.Net Core fixed a lot of this: ASP Core works more like Express, where routing is declarative, and has no surprises. In contrast, the original ASP for .Net used reflection to infer routes at runtime, which was extremely fragile.
The way that NuGet works now is more like cargo / npm; but before that package management was either all DIY or unpredictable.
IE: Until .Net Core, you could argue that C# was a better language, but end up getting trapped in the shortcomings of the ecosystem. Now that the ecosystem is on par with Rust / Node, it's much better.
I do think there needs to be a way to smooth over the difference between sync and async APIs. Far too often the difference between an async and sync method is just copy/paste. There needs to be a way for async/sync to just JIT out to call either the sync or async method of the stream/socket/lowest level API.
I've worked at one YC startup and interviewed with others. C# seems like the natural step after TS. But several were more interested in Go and Rust. Strange to me when C# seems more logical.
Rust's biggest advantage over C# is that it's framework-free. IE, you compile a working executable; nothing more is required.
What's harder to understand is: Will I be more productive using Rust compared to C#? (After dealing with the learning curve.) Will my code be "faster enough" to lower the cost of operations? (IE, a cluster of 5 servers instead of 15?)
If I could annotate a parameter with some kind of “move” keyword that would prevent the caller from using it again, that would be great.
“Frozen collections” and ImmutableArray<T> can solve this issue, but the latter is essentially just a defensive copy of the array, but in a special type. I'm not holding my breath that such a thing would ever be implemented; Analyzers will probably be the best we get.
> “Frozen collections” and ImmutableArray<T> can solve this issue, but the latter is essentially just a defensive copy of the array, but in a special type
My thoughts involved constructors. In those, sure, I could take IEnumerable<T> or whatever else there is, but if I want to store an array (or list) as a field in my class, I'd have to make a copy with ToArray() (and friends). Being able to "move" an array from the caller into the constructor (callee) would be nice.
ReadOnlyCollection<T> isn't actually read only, but just a wrapper around an array/list. IReadOnlyList<T> also isn't read only, but just restricts me from editing it. I can't edit the parameters, but the caller possibly could, and that's my issue.
> I can't edit the parameters, but the caller possibly could, and that's my issue.
// in one function of the caller's type
_data = new int[] { 1, 2, 3, 4, 5, 6, 7 }; // could come from some data collection engine
_dataTable = new(_data);
// then, sometime later, in another function of the caller's type
Array.Clear(_data); // reset for next iteration or something
// data table now has an array of zeros if it didn't copy the data
This is something Rust gets right, IMO. If I want the callee and the caller to share mutable ownership, RefCell<T> or Mutex<T> can be used, and the usage of such a type makes that clear. If I want thread safety, I can wrap the mutex in an Arc<T>. And, if I don't want shared ownership, a plain T can be used, and ownership passes into the callee. The issue is: in C#, without defensive measures, all objects are just T* with no ownership or thread-safety.
It does not work that way, because it is an `in` parameter, thus you can still accept any mutable type (including an array) which implements the essentially immutable IEnumerable interface you require.
Now if it was about a return value, it would be a different story.
edit:
Still what you suggest has its merits in the other direction, as it gives the caller a guarantee that the passed in collection will not be modified inside. Had to untangle broken code (without unit test when i got it) that was called like something `CalculateCost(SomegraphData input)` and while its name (and xmldoc) did not suggest, it did subtle modification to the data it got handed in... I was very upset about that legacy codebase I just inherited when I found that...
Working in a fully const-correct codebase in C++ is a joy. Working in a partially const-correct codebase in C++ is a nightmare because everything has to work around the semantic expectations of everything else.
Anders Hejlsberg, and those who took over the C# language design after he moved on to TypeScript, were certainly aware of how to make the language fully constable. The fact that they didn't leads me to believe they viewed it as a trade off between how much benefit do you get from using it vs how much cost do you take on from having that markup need to be everywhere in your entire language ecosystem. It's not enough for one application architect to say they want to take it on. Every engineer working on every library and every potentially reused function would have to take it on, because constness done right either tendrils everywhere or is a lie, and there are big costs to extending those constipation tendrils, just as there are big costs to not having them.
It probably would have been easier and cleaner to implement some form of `restrict` and `const` semantics (or maybe move, like you mentioned) than to work as hard as they did to come up with a still sub-optimal solution, but the performance they’ve managed to eek out of the frozen and immutable variants is to be commended.
But proper support for discriminated unions (and perhaps something better than the emasculated match blocks known as switch expressions) cannot come soon enough for C# to enter the big leagues.
We’re also using rust for cli utilities and non-client-facing backend server stuff.
I’d have been less surprised if there were a llvm backend for C#, but that wouldn’t exist without an IL LLVM target (because you’d be limited to the language without any (standard) library support.
The first-pass Roslyn compiler is really naive when it comes to optimizations; I constantly marvel at how little optimizations are performed in its first stages compared to what llvm does (the jit is amazingly well-tuned, however). An LLVM backend for C# would make for very interesting learning and research opportunities!
> As for the heap allocated objects, they will be allocated from unmanged(non-GC) memory, and will be allocated/freed exactly like in Rust.
I understand this decision, but it would also be interesting to see a version of this that hijacks the global allocator and the alloc types to use the GC instead (while still allowing you to opt-out and use unmanaged memory).
Good work nonetheless!
1. unmanaged pointers (C# syntax: T*, C++/CLI syntax: T*): these are the same as C pointers: can be converted to/from integers, cast arbitrarily, pointer arithmetic can be used. The garbage collector ignores these. These pointers can point to the stack, to native allocations. They can also point to the GC heap, but the GC won't adjust the pointer if it moves the underlying allocation (but allocations can be temporarily pinned).
2. object references (C# syntax: "T" (where T:class), C++/CLI syntax: T^): these are references pointing to the start of an object on the GC heap. They cannot point to the stack or to unmanaged memory. The garbage collector will update these as allocations are moved. Pointer arithmetic is not supported.
3. interior pointers (C# syntax: "ref T", C++/CLI syntax: T%): these references can point to the GC heap (including into the interior of objects), or to the stack or unmanaged memory. If pointing into the GC heap, the garbage collector will update these as allocations are moved. However, managed references can only live on the stack. It is not possible to store these on the GC heap; and certainly not possible to store them on the unmanaged heap (the GC wouldn't know to update them). Pointer arithmetic is supported, but conversions to integers are not.
It's not possible to translate Rust references to object references, because Rust references can point to stack or to the interior of objects. It's not possible to translate Rust references to interior pointers, because Rust references can occur on the heap, not just on the stack.
So a garbage collected version of rust is not possible without significant restrictions to the language (or using a GC more flexible than .NET's).
In addition to the limitations of the pointer types above, there's also an issue with enum types: .NET doesn't have discriminated unions. But the GC needs to be able to read the discriminator to tell if the enum contains pointers that need to be tracked by the GC.
Class hierarchies are discriminated unions. .NET doesn't feature general unboxed discriminated unions (although it can do this for simple blitted types using [FieldOffset]):
https://sourceforge.net/p/sasa/code/ci/default/tree/Sasa.Bin...
> Besides, we don't want to be worse than the C++ people, do we ;)?