DARPA project for automated translation from C to Rust (2024)

The key to this is using an LLM to recognize idioms, then formal methods to verify that the LLM guessed right. Which is what the current project is trying to do.

For example, consider C code with a function:

    void foo(int* tab, size_t len) { ... }

- Look at foo and its callers. Determine whether "tab" is an array or an indirect pass of a single integer.

- Look at foo and see if tab is ever modified. That determines whether it is mutable. It may be necessary to go down the call chain. This analysis may not be able to determine if it is mutable, but if determines it is immutable, tab can be passed immutably.

- Ask the LLM to figure out the size of tab from context in both foo and its callers. Is len the length of tab? That's a common idiom in C an LLM can recognize, but it might be a wrong guess. Is it the length in bytes or in ints? Remember, the programmer had to be able to figure this out, so the LLM has a good chance of doing it.

- If tab is immutable, hypothesize that the correct translation is

    fn foo(tab: &[int]) { ... }

Now attempt to use "lifting" formal analysis, as mentioned in the paper, to see if that changes the semantics of the program. Does "foo" subscript out of range? Can you prove that it doesn't?

This problem needs both LLMs and formal methods. LLMs to infer intent, and formal methods to check that inference.

Fil-C is so compatible that I have a Linux userland compiled with it https://fil-c.org/pizlix

Think of Fil-C as Java-ifying C an C++

Fil-C largely gets rid of UB.

There are exceptions left but they’re all fixable.

The only reasons why C has UB are:

- lack of memory safety (Fil-C fixes this)

- policy (Fil-C mostly fixes this but there are still a small number of remaining issues, which I intend to fix).

Fil-C is super comprehensive in how it fixes C. I bet it’s more comprehensive than you’d expect.

The reason not to use Fil-C in safety critical systems is that some operations have varying cost and there’s a GC. The GC is concurrent so you might be able to make it suitable for safety critical hard realtime stuff; it would just be a lot of work and probably not worth it. Rust is better than Fil-C for hard real time stuff for that reason.

Rust’s most important safety mechanism is panicking on out of bounds access. OOBA’s are the thing that attackers most want to do, and Rust, Fil-C, and almost all of the other memory safe languages solve that with runtime checking.

In short, I’d say what Rust gives you is static checking of a bunch of stuff and also memory safety. Fil-C gives you memory safety (and Fil-C’s memory safety goes further than Rust’s).

For such projects, a GC runtime goes against the reason why they used C in the first place. Rust can replace C where Fil-C can't.

A technically memory-safe C with overhead is not that groundbreaking. It has already been possible with sandboxing and interpreters/VMs.

We've had the tradeoff between zero-overhead-but-unsafe and safer-but-slower languages forever, even before C existed. Moving C from one category to the other is a feat of engineering, but doesn't remove the trade-off. It's a better ASAN, but not miraculously fixing C.

Most projects that didn't mind having a GC and runtime overhead are already using Java, C#, Go, etc. Many languages are memory-safe while claiming to have almost C-like performance if used right.

The whole point of Rust is getting close to removing the fast-or-safe tradeoff, not merely moving to the other side of it.

Rust doesn’t rely only on compile-time checks, so I find your claim dubious. Actually, the dynamic checks done by Rust are presented as useful features in themselves.

The memory safety audience and most people seem like arbitrary groups defined to support claims which aren’t really supported by strong evidence.

The biggest improvement in memory safety in the past decades wasn’t thanks to Rust, but languages like Python, Java or C# which rely heavily on dynamic checks. The goal is to catch bugs and avoid security vulnerabilities.

Taking standard C code and achieving the above would be amazing and much more useful than rewriting something in Rust.

Like, say you’re writing a port daemon. Then your argument holds! And that’s admittedly a big category since it includes databases, application servers, and a lot of other stuff.

But say you’re writing an app that talks to systemd and your desktop shell and renders things and has a UI. If you do that in Go or Rust, you’re pulling in a ton of unsafe dependencies. But the Fil-C story is that you just compile all of that stuff with Fil-C and so there are no unsafe dependencies.

By the way, that’s not a fantasy but like mostly a reality. I have a Linux distro where all of userland is compiled with Fil-C and it works great!

Now you might say, why not rewrite all the deps in Go or Rust? The funny thing about that is you sort of can’t do that today since neither of those languages has dynamic linking. Fil-C supports dynamic linking just fine.

Here's a struct that maintains an invariant - say, that field a is less than field b. That invariant should be set when it is created.

You find a bug where a is greater than b. Where is the bug? With a struct, it can be anywhere in the code - any line that touches the struct. But with a (well designed) class, a and b are private, and so you only have to look at lines of code within the class. The surface area where the bug can be is much smaller.

The bigger the code base and the more well-used the class is, the more this matters.

My experience with random rust projects is that they usually "just build", with limited exceptions like weird embedded stuff.

Can you mix Zig libraries written X years ago with modern Zig? Will today's code work in 5-10 years if I want to use it in a project then?

I've definitely cloned down my fair share of C projects, mashed the make command into my terminal, and watched the gcc/clang logs fly by and never batted an eye beyond checking the sha256sum on any downloaded tarballs.

There's a valid argument to be made about supply chain attacks, but there does exist tooling to lock that down, and I would argue that any serious software development firm should be auditing every third party dependency they take on.

For reference: https://news.ycombinator.com/item?id=40172033

It is the native language of every major operating system.

It is the language most closely tied to mainstream ISA assembly, and—symbiotically—which mainstream archs have tried to optimize for.

It's one of few languages that are completely public, in the sense that it is not de jure or de facto owned by a single market participant.

And it is the only language that both has standardized ABIs (per platform) but API and ABI compatibility implemented in a huge number of other languages.

There is no extern "Rust" {} in C++. There is no with Convention => Swift in Ada.

There is extern "C" and with Convention => C exactly because because C is the de facto standard that everything has to interface with.

That's not fair. C++ 26 gets from the uninitialized primitives by default being a massive footgun to only slightly worse than using Rust's unsafe core::mem::uninitialized function - which was deprecated in 2019.

Now, sure, in a good language this mistake shouldn't even compile, so C++ moved from "Reprehensibly awful, do not use" to "Very bad". But that's more than a femtometer.

Barne has talked about it, but one of the primary reasons he created C++ was to address safety issues in C. And he's talked about this since the early 90s.

Like access modifiers and private by default. Something we take for granted now, but something which is impossible in C.

Or std::vector. Unlike array in C, it won't blow up if you add too much stuff to it. Again, obvious, but at the time, revolutionary.

Or exceptions! We take them for granted to!

No more forgetting to check errno! No, if you don't catch an exception your program terminates, so you have to handle errors.

Or, the biggest one of all: RAII and ownership. Might sound familiar to rust devs!

Point is, C++ is not C. That's why C++ exists. Its not safe, but it's much, much safer.

The issue is that fundamentally interior mutability is hard to reason about. It requires a lot more temporal reasoning to make sense of and ensure the safety of. Leaning into the type system Rust has does make interior mutability harder when you have "temporally based" safety arguments. It's a lot easier to reason about correctness in the model Rust uses.

Have you looked at the godot-rust bindings? They've gone very far to make the Godot object system work well going between C++ Godot code and Rust. I would love to see that model adapted to Gtk.

C# overlaps C++ in all these “performance-sensitive high-level code” domains.

Is C# perfect for every task, no. But it's probably suitable for most of the tasks that DARPA is targeting to support. The reality is that low-level languages were used in the past because generally our computers were less efficient, and the higher-level languages didn't exist. Most things today written in C++ would probably be written in a higher-level language if designed today.

Infact, I know of a specific case where a vendor is currently in hot water because a GUI tool running on a standard windows machine is not memory safe when in this case some agencies think it should be. This is probably similar to the vast majority of C++ code DARPA is concerned with.

The US should definitely keep doing R&D, the tech that comes out of DARPA has lots of downstream benefits beyond military applications. We should continue producing the best tech in the world, and explore all sorts of skunkworks and moonshots and crazy tinkering. I agree with "peace through strength" - be too dangerous to abuse, and if shit goes down, be the one who decides who wins and loses.

We don't need a corrupt military industrial complex, endless wars, private contractors, NGOs, nepo-grifters, and all the rabble that have abused the lack of accountability to make themselves rich, or inflict unrest on other countries, or manipulate bureaucracy for political gain, and all that jazz. Basic things like "department of war" naming fall in that direction, from my perspective. Simplify, cut, use blunt truthful language, and figure out how to hold them accountable going forward, at the very least, and at best, figure out how to claw back the trillions, or figure out where it went. You don't need to alter the underlying mission at all, really.

See, there's this notion that the US is a bad guy, and they point out all the casualties and conflicts and collateral damage and say "see, we're the baddies!" They never account for the context. Some of it is as simple as pointing out that if we hadn't killed all those german nazis, and sought pacifism and peace at any price, we'd all be speaking German. Some legitimate atrocities have occurred, as recently as Obama's unnecessary drone murders, but ranging from improper harms done to US troops, exposure to toxins, Abu Ghraib abuses of civil liberties, some of what's happened at Gitmo, etc. Each and every time we recognize a harm done, we should at the very least hold ourselves, collectively, responsible, and seek to be better. Accountability is a civic duty, regardless of political affiliation or philosophy. We should also be honest about necessary evils, not allow ourselves to be manipulated by bad faith actors, back up positions and words with actions and force, and hold ourselves to the highest possible standards. If we find ourselves continuously presented with no alternatives except between two evils, we should choose the lesser evil which allows us to preserve ourselves (otherwise, what are we even doing?)

The US body politic has allowed assumptions of good faith by our bureaucrats and military brass and politicians to fester into unimaginable corruption and vice, proving Eisenhower a prophet. We should simply be better - little things, important things, ground up discipline, principles, and a persisted social construct upon which to stably build a nation's strength is 100% necessary.

A lot of what Trump and Hegseth are doing pushes in that direction. Some aspirational, some performative, some well intentioned but wrong, but I think it's mostly in good faith, and I can get behind that.

I wish more on the left would at a bare minimum wish everyone to be responsible and to do well, for everyone to be happy, healthy, prosperous, and free. It has to be permitted to hope that whoever is in power achieves victories and success in those basic things, or we've lost the plot.

I agree that most of the awkwardness of reading comes from explicit declarations, but really, even if it's more verbose, I would prefer that that explicit nature is defined elsewhere rather than reading like a compact run-on sentence. (I'm hypocritically writing here, since I do this too often myself.)

  let x: Option<Result<Vec<i32>, std::num::ParseIntError>> = Some(Ok(vec![1, 2, 3]));
  let flattened = x
 .map(|r| r.unwrap_or_default())
 .unwrap_or_else(|| Vec::<i32>::new());

I have no experience with Rust so it makes sense i dont understand, but i also have no experience with Python or JS.

I mostly expose myself to it, at the moment, through benchmark work which highlights how competitive of a good language it is.

But there's seemingly nowhere else to go, but maybe small Discord servers where you can meet people and share honest opinions that are real and agree to disagree without penalty.

Everyone should feel free to express harmless opinions.

Edit: Whoever downvoted me for this comment is proving my point.

Edit (for adastra22): I'm not sure that me providing a list of specific modifications to Rust syntax is meaningful to anyone anyway. I'm just a nobody. And it should be OK for people to express personal opinions that hint towards something being wrong without also being required to solve the problem. That's just life.

    Rust does not safely support all of the things folks do in C++ (complex cyclic object graphs with inscrutable logic deciding lifetimes, intentional races, etc).