Hacker News Reader

We're all waiting for your anti-malware Rust Win32 kernel module...

Ok, but seriously I don't believe this will ever happen and I don't really think this is a language debate nor do I want to engage in one.

This is about putting critical infrastructure connected to the internet that's running an operating system that you can't trust out of the box. Since the Windows OS is susceptible to so much malware you need all these third party services (which you also can't trust or audit, but it's absolutely better than not having anything) on top of the OS.

There was a whole host of companies that had zero problems, not because they're using Rust, but because they have much better security practices and quality infosec employees.

> This is about putting critical infrastructure connected to the internet that's running an operating system that you can't trust out of the box. Since the Windows OS is susceptible to so much malware you need all these third party services (which you also can't trust or audit, but it's absolutely better than not having anything) on top of the OS.

Agreed, they should not be using Windows in the first place. That should have been the first line of defense.

> There was a whole host of companies that had zero problems, not because they're using Rust, but because they have much better security practices and quality infosec employees.

Fair enough, I only commented on one layer of the security stack -- so your remark that expands the scope is valid and welcome.

> We're all waiting for your anti-malware Rust Win32 kernel module...

I am done working for free. If I am paid to do it I am sure I would have done better than this poor confused soul who allows NULL pointer dereferencing which is a mistake that most C/C++ interns quickly learn to avoid.

>Agreed, they should not be using Windows in the first place

Crowdstrike borked RHEL 1 month ago https://access.redhat.com/solutions/7068083 Literally the same situation, unbootable machines.

The reality is that shitty software broke everything. Why do we have to drag the OS into this?

Dunno, I guess I naively thought the quality of Linux drivers is higher but on the other hand, if the same confused randos are writing them then you're right that it would not make a difference.

I didn't know that. So that makes this two strikes?

My understanding is this was not a case of null pointer access that could be caught by a compiler really... but of a corrupt data file making a mess all over the place... running in kernel space, where no segfault is safe.

The root issue is giving privileged access to a business entity you think you can trust, but clearly can't.

I'm a fulltime Rust developer, but I don't think Rust saves you here.

Wouldn't a strongly typed language like rust sill catch a bad datafile?

E.g. loading it would require you to setup a maximal size and a valid configuration struct?

It could.

We haven't seen the code but it could be something like:

  char *ptr = parsefile(file_we_released_without_testing);
  if(ptr[0]=='A') { } // BSOD loop

parsefile returns NULL unexpectedly.

So this style of error can be addressed by using a safe language. Or static analysis. Or code reviews. Or not doing this stuff in the kernel. Or formal methods. Or fuzzing.

As someone else said you likely can't easily use Rust for Windows kernel modules/drivers. I'm sure a strong enough engineering team could do it (e.g. transpile Rust to C) but I'm not sure it's the biggest engineering problem CrowdStrike has. Microsoft has a complete tool-chain for developing these and it's usually C/C++ or assembly.

unhandled null in rust will still cause panic. still cause the bootloop.

I'm not a Rust expert but wouldn't you pick some ("null-safe") type that can't be null in Rust? A reference?

i dont think it matters, if you have any exception in the critical boot part, you will end up with this. Rust cannot fix this. Microkernels might.

Something like this Go snippet:

  func parsefile(string) string {
  }

  func thatfunctionthatcrashedinC() {
      defer func() {
      if err := recover(); err != nil {
        log.Println("panic occurred:", err)
      }
    }()
    result := parsefile(badfilethatcrashesC);
    if result[0] == 'A' {
    }
  }

so... using a type that can't be nil. recovering from runtime panics (you have to do that but this can be enforced by standards and also it can happen up the stack for all code, e.g. like http handlers do by default in the Go standard library). More importantly these errors are not segfaults in Go, i.e. there's "exceptions" you can and should catch and there are exceptions you can't.

You have all that in C++ too. Exceptions are near zero cost and used everywhere, sometimes even in embedded stuff too.

Sure. I speak C++ ;) You can do this in C++ but I think it's generally more crash prone than Go. Based on personal experience of ~20 years of C++ and ~10 of Go I've debugged many a core dump in C++ and I think zero in Go. You can restrict yourself to the somewhat safer parts of C++ for sure.

If what you say is true -- OK. Then would you say that the tweet posted earlier that showed the NULL pointer access was incorrect or misleading?

Null pointer access caused by bad data is entirely conceivable... esp when you overwrite parts of kernel memory with nulls.

I see. That doesn't make it better though, and OK let's forget about other languages.

I mean that the least you can do before pointer dereference is just check for several bad sentinel values, NULL being one of them.

Seems like a rather amateur mistake to me.

From that point of view, there are several million amateurs, some of them quite highly paid, out there writing terrible code.

Well, that statement is 100% true.

I've been an idiot as well in the past. Happily some of us actually learn though!

A safer language like RUST won't help you against bad practices and poor QA processes. This is a kind of error that you should catch with automating testing, even before pushing the change to main branch.

Not just QA; security assurance, code reviews, static and dynamic testing, threat surface analysis, unit testing, and pentesting either didn’t exist or weren’t sufficiently applied.

I have to imagine that this bug has existed for quite some time and I’d be curious to know what other input validation errors they have, considering the amount of untrusted input they evaluate at ring 0 originating from userland.

Again, there are safe ways of doing this. For example, Wuffs exists: https://github.com/google/wuffs

At the very least, big money security software companies should be parsing untrusted content with some kind of rigorouly safe approach, not just squirting it through a big pile of C/C++.

And don't get me started on the whole concept of undefined behavior in those languages. To quote I. I. Rabi, "Who ordered that?"

>At the very least, big money security software companies should be parsing untrusted content with some kind of rigorouly safe approach

the malformed files were updates from crowdstrike itself. It's not exactly "untrusted content".

It is untrusted data in the sense of files being read from disk that are not part of the signed kernel driver code.

The files in question reside within C:\windows, which requires admin privileges to write to. If untrusted data can end up there, you're already on the other side of the airtight hatchway[1].

[1] https://devblogs.microsoft.com/oldnewthing/20220907-00/?p=10...

Fuzzing...

I'd love to hear from an engineer on the project but unfortunately we're likely not to.

Highly unlikely anyone except governments or top-paying corporations with custom-negotiated T&Cs will see a detailed post-portem, unless someone blows the whistle. Would love to read an AmA.

I agree. My point was that using a language whose compiler will not allow you to build your production binary if you make a certain mistake could have been one extra line of defense and who knows, that might have prevented this problem this one particular time.

But I am in full agreement with you that sloppy programmers cannot truly be helped. They just screw up and move on like nothing happened. Sigh.

You can still do unsafe, and you do need unsafe in some cases.

And human beings can indeed write safe "unsafe" code. But to do so consistently, you have to be very smart, very cautious, and somewhat lucky.

Indeed, ensuring that unsafe is isolated and obeys certain semantics is a superpower that few languages have rust+kani is a good and modern way to achieve this.

OK, sure, but don't you think that being a world-wide antivirus vendor should have warranted a better process?

Oh absolutely. This is utterly unacceptable. The ease with which CS pushed willy nilly a bad build to prod in what seems to be a monophasic release is absurd.

Something of this nature would have had our entire team fired. The number of phases and the thoroughness and exhaustiveness of the protocols we have to ensure we don't push bad builds would have most engineers taken aback... but we have to. With great power comes great responsibility.

Memory-safe lanuages (for goodness' sake, even the crap I write in Python qualifies!) are the very minimum that is needed; not to use them for anything critical is simply crazy. Yes, do all the other things, but at least put out the blazing fire in your basement while you are implementing your fire-safety strategy.

Even with memory-safe languages you can shoot in your foots and on Windows, AFAIK, you need to stick with c/c++ for this kind of low level programming.

BTW, using your metaphor, until 2 days ago they didn't even know that there was a fire in the basement, nor a basement.

How do you write a driver in Python?

You can do bugs in any language. The problem here was monoculture and critical dependence on one supplier, not a programming language choice

You can indeed write buggy/unsafe code in any language. But it's a lot easier to do in notoriously unsafe languages like C/C++, which for some maniacal reason we seem to have based the world's digital infrastructure on.

C++ was a terrible, terrible mistake.

I'd argue it's several things, these two things included.

To screw up so legendarily requires a concert of bad decisions.

Yes. The term "normalization of deviance" comes to mind. Even just a phased rollout would have caught this one with just a tiny fraction of the damage observed.

1. It wasn't a null pointer: https://xcancel.com/patrickwardle/status/1814343502886477857

2. If the driver crashed due to a Rust panic, the result (boot loop) would be the same.

Your plan is to replace all the software written in C/C++ with... software that doesn't exist.

It's good to criticize the current state of things, but don't pretend you have a solution.

Also, do we know if rust would have helped here? Rust doesn't guarantee no crashes -- in fact, panicking (aka crashing) is the default.

From what I've seen, it was a NULL pointer dereferencing. Dynamic, not static, so still requires diligence even in Rust.

RE: panic default, don't get fooled by hobby projects, professional Rust code always does pattern matching and does not defer to panics.

The "software that doesn't exist" point is somewhat valid, though it's also the chicken and the egg problem, as in that not many people are working to make it happen because the current state of affairs is wrongly deemed as good enough. And it really is not.

> RE: panic default, don't get fooled by hobby projects, professional Rust code always does pattern matching and does not defer to panics.

That's a "programmers who know what they're doing don't make that mistake" argument. If that were tenable there'd be no need for rust in the first place.

Yeah I don't disagree, you do have a point.

But this is easier to scan for compared to all the potential memory unsafeties in C/C++. It's an improvement.

Windows does not support drivers based on rust language, so perhaps may be another 5 years

You can write drivers in Rust - it's just quite hard at the moment. Microsoft published metadata packages for WDK APIs and started creating samples: https://github.com/microsoft/Windows-rust-driver-samples

Good. Microsoft doing something sensible.

How can that be possible? Are drivers not binary files?

Rust can't compile DLLs? AFAIK it can?

Look at Linux. Getting Rust to work with the kernel is a long story of defining APIs, cleaning up the C-side API to make it tenable, coding test filesystems and whatnot to make sure it all works, and getting buy-in and maintenance for all of the above.

Doing the same with zero control over the non-Rust side of the kernel seems completely untenable.

I am not saying there are no challenges. I am saying that CrowdStrike does not seem to have even tried to have a better process. Rust would be only a small part of the picture; just one more layer in the security posture (a small one at that, admittedly).

Exactly. It's like the brown M&Ms in the Van Halen rider; it's not that the M&M's were the problem, but that it was a test of diligence. People who don't care about detail are likely to screw up the big things just as badly as they screw up the little things.

Being a multi-million dollar company and using unsafe languages today is not a good look. But everyone gets away with it because everyone else is doing it.

This is a kernel driver. Runs in kernel space. Intercepts syscalls. You'd definitely be fighting uphill to write it in Rust. And your code would be riddled with `unsafe` by necessity anyways.

Fair enough, still Rust's unsafe is not dropping all of its guarantees. Quite a lot of them remain in place.

Not saying you can't write bugs in Rust, of course -- that would be crazily delusional. I am saying they needed a better process. And I am saying that a stricter language could have improved the process a bit as well.