Making Win32 APIs More Accessible to More Languages
blogs.windows.com
blogs.windows.com
E.g.: Rust has the UUID crate: https://crates.io/crates/uuid
Similarly, Java has had a UUID type since v1.5: https://docs.oracle.com/javase/8/docs/api/java/util/UUID.htm...
It's not just concrete types, but core language concepts often "don't translate". E.g.: Win32 is essentially a C-like set of bindings. In many languages such as C++, D, Zig, etc... this is not an issue. However the type system of C is much less strict that Rust, so the metadata has no concept of ownership and lifetimes. This means that if you want to consume Win32 APIs from Rust, you're still forced to wrap everything manually at least once to escape the "unsafe" sandpit.
A thought I had is that C is used for API definitions because "it is lowest common denominator". But that's the wrong thing to do, the metadata doesn't have to "sink to the lowest level"! An ideal metadata definition would be something akin to Rust's type system, or even stronger than that! "Degrading" a binding from a strong type system to a weak one is trivial and can be performed completely automatically. The converse is difficult and error-prone even for humans, and impossible for code generators. You can't synthesise this missing data!
Even the read-only guarantee is too weak. E.g.:
// Rust could treat 'buf' as read-only borrow.
void SendData( [readonly] void* buf );
// But here the borrow outlives the function call lifetime!
void SendDataAsync( [readonly]void* buf );
There are many such abstract API concepts missing from C, such as thread safety, process integrity safety, non-returning calls, blocking vs async, etc...
This doesn't have to do with C, it has to do with the nature of systems programming - not everything can be baked into the one type system to rule them all that is checked by all programs in all languages at compile time. Metadata is insufficient, you need runtime type checking, which is not free.
And to be honest I don't think this is an attractive thing to strive for. OS APIs are not that hard to program against (well, maybe Windows APIs are). Differences between OS's and versions are, however. We don't really need safer systems interfaces, what we need are standard ones that actually work to solve hard systems problems like asynchronous i/o, real time scheduling on non-realtime kernels, and arbitrary control flow. These are very unsafe things that do not have good solutions in the OS APIs, adding strict types where they are not needed creates little value for the systems programmers that want to make cool new systems.
If step out the realm of sets and into programing languages types are very similar to sets. Heck there is type theory which also is very similar to set theory, but has a few difference.
Incidentally, there is no reason why the metadata can't be improved with more information that can be used to create safer wrappers. The winmd format isn't restricted to only that which can be represented in C's type system.
I suppose you could hack it by adding new Attributes...
It seems at least one of their hacks is not valid in the CLR, so that bridge has been crossed. Plus they do make use of attributes.
And I think, they have thought about that before they startet :)
As for CC specifically, I'd love to use it with .NET Core - I gather it doesn't need that much work to get going again. It's a shame Microsoft hasn't invested more into it, especially as there was talk of making CC an integral part of the C# syntax (as in, first-class support for `pure` functions in C# and a succinct syntax for attaching them as invariants/etc to C# types and methods). If this was done it would solve so many problems (e.g. compiler-enforced guarantees about which navigation-properties on Entity Framework entities are loaded) and maybe even elevate C# to compete with Ada and Eiffel for safety-critical systems!
Here is a nice overview of the whole story and how interrelated COM and .NET are.
https://arstechnica.com/features/2012/10/windows-8-and-winrt...
This shouldn't actually be a huge problem in Rust, because of the traits system. It should be possible to implement all the expected behaviors (Into/From, Add/Sub/Mul/Div, Ord/Eq, etc) for these special types and if library code is properly generic, it shouldn't care whether or not you're using the native type. If it does require a specific concrete type, From/Into should still make things reasonably ergonomic.
Re: safety, the typical approach has just been to keep one library that's a direct mapping to the C API, which is publicly unsafe, and then create a safe library abstracted on top of that which codifies the documented behaviors. That layer relies on documentation, and on humans to follow it, but then so does all C code ever written :)
A typical method call is:
Allocate some struct.
Zero it out.
Set a "length" field to the length of the struct.
Populate the "parameter" fields.
Call the API with the struct and 0 for the return buffer size.
You get back an expected error code for "buffer too small".
You allocate the required space in your buffer.
Call the method again. The amount of data changed! Loop and try again.[1]
You get your data! Now to check the error codes.
Look up error code using a Win32 method.
Free the buffer.
PS: Win32 strings and Rust strings aren't compatible. I've already noticed that the current version of the Microsoft code-generator seems to use an awful lot of [u16] arrays, which are not the system OsString struct.
For comparison, the C# "String" type isn't directly compatible with Win32 strings either, but the FFI engine handles this conversion back-and-forth for you.
Similarly, few (any?) languages have good built-in support for the various kinds of "struct-followed-by-dynamic-data" typically returned by system APIs. Notably, C does not have any way to encode this in the type system, so API metadata tends to just leave this out or describe it in the documentation. If the metadata was extented to cover this kind of thing, then the generators could do all sorts of clever things with this scenario...
[1] A lot of people don't do this, they assume that the second call will always succeed. Programming in this style leads to rare random errors throughout the entire application. A loop is required for correctness for most (but not all) Win32 calls that return dynamic data...
https://github.com/floooh/sokol-samples/blob/89f5825ab5d3690...
The Win32 window system functions and D3D11 are actually very convenient to work with in C99, better than C++ actually.
> A loop is required for correctness for most (but not all) Win32 calls that return dynamic data...
A very good point, one very rarely mentioned.
Of course, the win32 windowing and common controls and single threaded, so if it's the same application it won't have changed underneath you.
std::string nicerGetComputerName() {
char buf[1024];
DWORD len = sizeof(buf);
memset(buf, 0, sizeof(buf));
GetComputerNameA(buf, &len);
return buf;
}#2) You probably want GetComputerNameW. GetComputerNameA uses an unspecified encoding that you'd have to handle.
#3) A 1KB buffer is excessive! Over-sized buffers can blow through the L1 cache all too easily, slowing down applications. It's also stack allocated, and this can be a problem in some scenarios.
#4) You're forcing the computer name to be heap allocated. This can also be problematic in some scenarios, and is slower. GetComputerName might be used in logging scenarios that are already trying to handle cases such as "out of memory"! Allocating more memory may be unacceptable.
You get the idea.
Wouldn't it be safer to use GetComputerNameW to get a UTF-16 string?
char buf[1024] = {0};(1) Change the C-API to make it more "binding-generator-friendly", for instance by adding a range/slice-struct to th C-API which bundles a pointer and associated size, or specially named typedefs that only exist to give the binding generator hints for special case handling. By adding a C-type which represents a range/slice, it's easier to connect high level languages to the C API, and as a nice side effect, the C-API also becomes safer and more "correct", because pointers and associated sizes can be passed around as one item.
(2) Make the bindings-generator configurable on a per-language and per-API basis, this can be as simple as a map which overrides type- and function-names, or injects manually written code into the generated bindings.
The goal is to make the generated bindings more idiomatic to the target language.
This mostly works if you have control over the underlying C-API of course, e.g. the language bindings are created by the original C-library project, not as an external project to convert a fixed C-API.
I wrote a blog post about this whole topic:
https://floooh.github.io/2020/08/23/sokol-bindgen.html
...and here's an example of such semi-auto-generated Zig bindings:
https://github.com/floooh/sokol-zig/tree/master/src/sokol
...for instance note the "injected" helper functions here:
https://github.com/floooh/sokol-zig/blob/1c93f60ad178869b84d...
I mean, if you need to use the raw win32 api, there must be a reason you need raw access, and in such a case, conversion to language types is more likely to be a hindrance than help.
Also, if you need to call multiple native apis, it is easier to pass the native types around if the is no conversion to language types.
It is up to the generation tool to provide idiomatic bindings based on the information provided by win32metadata.
This is nothing new, WinRT works exactly like this across .NET, C++, JS, Rust and whatever language projection one feels like adding.
I guess there is also the possibility to use the JNI replacement when it comes out of preview, although that would be constrained to Java 17 or later.
Are they planning to change this?
Waiting patiently for Microsoft to actually put some effort into the obscure TPM documentation along with making it easier for developers to implement.
Not claiming it's perfect, saying it exists on nearly every cpu running Windows today. There's serious value for plenty of people out there to have access to these interfaces that are only internally documented and available to a small group of people from one company alone.
Only have to look at the history of adding randomness to CryptographyAPI-NextGen to understand where I'm coming from.
No one has been able to add an additional RNG source to the Windows entropy pool for ~15 years now, highly venerable Microsoft developers say this is a good thing in public for all their enterprise clients when basic maths proves it wrong in about 2 minutes.
They do not have your best interests at heart, no matter how big you are, in fact it's less likely the bigger you are.
It contains definitions that can be used to automatically generate bindings to any language. So, we have C# and Rust as examples, but the very same definitions could be used with languages like Zig.
Very cool initiative.
I usually use Vanara for Win32 PInvoke wrappera, or hand-code them if they are undocumented or I only need a couple. Rarely I've come across memory issues with the wrappers provided by Vanara and PInvoke.net, where they've forgotten to return a buffer - these kind of problems are absolute bastards to track down, and the root cause is of course manually building wrapper code. Having an automated means to do this will be a huge boon.
I wonder though - quite often if I'm digging into Win32 stuff, it means I'm working on performance sensitive code, where I want to keep allocations low - I haven't looked I to it yet, but I hope C#/Win32 has an option to generate wrappers that take pointers instead of only using managed types!