BlueHat IL 2023: Microsoft rewriting core Windows libraries in Rust | Hacker News Reader

BlueHat IL 2023: Microsoft rewriting core Windows libraries in Rust

youtube.com

Rust discussion starts at: https://youtu.be/8T6ClX-y2AE?t=2610

Highlights:

    * 2 projects: dwrite and Win32k
    * 2 devs, 6 months to rewrite 152KLOC C++ to 90KLOC Rust re dwrite
    * Performance increases of 5-15% re dwrite
    * Only about 10% of Win32k code is unsafe
    * No perf regressions re Win32k
    * Syscall in the Windows kernel now written in Rust

Is it really C++ or just C saved in .cpp files? (which is a norm in Win32 API world)

Only if the only thing one has read was Petzold's book.

Since MFC and COM, that outside of the kernel most code is C++, even MSVC libc is actually written in C++ with extern "C" declarations.

Additionally starting with Windows Vista, C++ code started to be allowed on the kernel (VC++ got /kernel flag), and nowadays there is a template library called WIL.

Win32k is the kernel side of GDI, that's the core GUI drawing code for those unfamiliar with Windows.

It's a common source of privilege escalation bugs because it runs in kernel mode and accessible via a large surface of user mode apis.

So code optimized for 80486 machines in the 90's is not less perfomace after the rewrite in Rust.

What a miracle!

Rust marketing is the best.

You're getting downvoted but i think it's a fair point that revisiting win32k in any language might come up with some opportunities for optimization along the way. Performance best practices have changed a lot since that component was written.

I suspect there reason the parent is being downvoted is that "So code optimized for 80486 machines in the 90's" isn't something we actually know. Do you really think Microsoft hasn't touched this code since the 90s? It comes across as sour grapes.

My understanding of what the presenter meant here was "this is gnarly code that's been around for a very long time." Not that it literally remains solely optimized for the 486.

I seem to recall hearing from MS affiliated people in my network that someone revisited GDI circa 2015? It also wouldn't shock me if there's more that can be done.

Also worth noting that various parts of MS have declared GDI as legacy at various points in time. Yet it remains a critically important component.

Generally codebases tend towards performing worse as people work in it, rather than better.

51:13 in the video: "old (late 80, early 90) and perf crtical, designed for 286/386."

Am I the only one who watched the video?

The API dates from those days. That is true. That doesn't mean the code is untouched since then. That's you extrapolating.

I watched the video yesterday, before it was even posted to Hacker News.

Rust marketing says it is on par with performance with C/C++, not better. So I don’t really get your point.

A rewrite of an old code base with today's hardware characteristics in mind should yield better performance, in Rust, C/C++ or any other system language.

Depends on the task.

> Rust marketing is the best.

This is not Rust marketing. This is some MS systems programmers detailing their experience with Rust.

That's because the bugs are also implemented in rust.

The slide at 48m26s says 152 KLOC of Rust and 96 KLOC of C++.

I didn't watch the whole video though. Is that the same number you're quoting?

You are correct, and I was incorrect. It's actually the reverse of what I said, 152 KLOC of Rust compared to 96KLOC of C++.

Overlooked is that it's one of two incredible successes that have come out of Mozilla:

* The open web, David defeating Goliath Microsoft (arguably like a small open source retailer defeating Amazon today).

* Rust: Not entirely Mozilla, but they played an essential, major role nurturing it, and were among the first to deploy it.

Any others that I'm overlooking?

By the time of IE 5 came to be there was little left from David, hence why IE 6 was hardly having any updates, and most of the team went on to work on WPF.

Let's Encrypt was also originally from Mozilla.

In addition, I seem to recall that the Alliance For Open Media and its AV1 codec also derives from Mozilla sponsoring Xiph.org's Monty to come up with an unencumbered, next-gen media codec.

Vorbis and later Opus contained a lot of contributions from Mozilla.

Those memory bugs are costly after all. I think the best part of this presentation is they managed to improve performance while adding memory safety. Two people rebuilding a 90kloc c++ into rust in 6 months is also pretty impressive.

One the subject of porting code from one language to another (ignoring that the headline says "rewrite", which implies more than a dumb port).

Are there any tools to help map out the work that needs to be done? I found myself doing this by hand recently, doing graph search on a piece of paper. I needed to port one function, so I identified all the functions that function called, wrote them down, and then looked into the next layer of functions etc. I manually searched the potential call graphs by reading the code and wrote down all possible functions involved. I then crossed of functions that seemed obvious, such as a function to retrieve the value of a dictionary with a default value, such simple functions are trivial and need no special effort. In the end I had a list of functions that needed to be ported one-by-one and I began working from the bottom up.

I was wondering if there are any tools that could automate this process? It seems to be the beginning of a source-to-source compiler, but such a compiler could be kept simple by embracing that most of the work can be presented to a human to perform manually. The "compilers" only job is to structure the work and break it down into small chunks.

https://en.wikipedia.org/wiki/Call_graph#Free_software_call_...

Lots of software tools for generating call graphs. I've used doxygen (free) for it in the past and LDRA (not free) at an early job. This won't generate the new code for you, but will be a lot easier than manually constructing a call graph.

I have used SourceTrail for that in the past, sadly its development stopped.

Writing kernel-mode code in Windows with rust is very cool, but I am curious if there are code samples out there for this kind of thing.

Windows kernel drivers are usually fully asynchronous so I'm curious how rust handles IRPs and completion callbacks and such in the kernel.

I haven't tried them out myself, but https://github.com/StephanvanSchaik/windows-kernel-rs exists. Has a bunch of code, though it hasn't been updated in a long time, so it may have bitrotted.

Slides: https://msrndcdn360.blob.core.windows.net/bluehat/bluehatil/...

Would it be a good idea to rewrite glibc in Rust?

Rewrite the C standard library in Rust? I'm not sure I understand how that even makes sense. Are there problems this would solve?

A C standard library rewritten in Rust already exists. [0]

glibc has its share of CVEs, and many of the bugs are of the double free, use-after-free, buffer overflow variety. [1]

[0]: https://github.com/redox-os/relibc [1]: https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=glibc

Many C standard libraries are actually re-written in C++ nowadays, e.g. Microsoft and clang's, so doing it in Rust is hardly any different, it would be safer than C++, minus the usual C gotchas that are anyway exposed on the public APIs and the implementations can't do much about it (like bad pointers or size).

> Rewrite the C standard library in Rust? I'm not sure I understand how that even makes sense. Are there problems this would solve?

The C standard library isn't necessarily a standard library of C code, it's more of a standard library for C code. There's plenty of languages these days where the standard library is built in other languages, somewhere between partially and fully; some portion of the C standard library can probably benefit greatly from the memory lifetime structures of Rust; although the constant need to interface with C code may reduce the effectiveness.

Already exists: https://github.com/redox-os/relibc

Can’t say whether it would be a good idea but it probably wouldn’t be a bad one.

Since Rust doesn't yet support shared libraries, would each binary end up statically linking its own copy of glibc?

#![crate_type = "cydlib"]

Rust supports shared libraries.

Rust supports dynamic libraries with C ABI very well.

Technically, it also supports Rust ABI dynamic libraries, but their portability is quite fragile due to changes in the compiler. That's why Rust parts of the code are usually statically linked.

MS and Amazon own and run Rust, this isn’t surprising.

They contribute to it; I don’t think it’s fair to say they own it? It’s certainly nothing like Swift or Go for example.

I guess they pay the salaries of key contributors, and certainly have some influence via their work.

Were they paying those salaries before or after those people became key contributors?

I do not know the answer (in this case) and believe this makes a notable difference.

Mostly later, as they got laid off from Mozzilla.

However it is naive to think Big corps don't have an influence on it.

Reply on news.ycombinator.com