That being said, Rust and possibly even Go would be strong contenders to make a new SQLite-like library/program today. At least on the Rust side, the C bindings are excellent too.
Go is a good language, one of its core pieces is to increase memory safety through the use of a garbage collector.
Much of what the SQLite post states may not be safe operations in Go.
Rust would be a safer alternative that meets many of the SQLite requirements.
The thing is, SQLite is bulletproof at this point, so do we need to replace it?
No, probably not, it is fine as it is. Still, even SQLite started out as a "for fun" project. Who knows what might happen in the future to displace it? ;)
Those aren't mutually exclusive. I for one find destroyers to be an absolute riot.
I wish I had intended that pun embedded in the comment, but from now on ;)
See https://changelog.com/podcast/201#transcript-100
>I had couple months off and I thought, "Hey, I'm just gonna go and cobble together a really quick and simple database engine that just does a few very simple SQL commands, insert the lead, update and select." No joints, wasn't trying to be efficient... All I needed to do was pull stuff off of a disk in that memory.
>And I put it out there and... I've been doing open source for years before this, putting things on my website, and people would find my thing -- or well, you know, I'd put things on my website and it'd get like five downloads per year, or something like that. I'd figured this would be just another one of those things, but for whatever reason it really resonated with people.
Nothing written in C is bulletproof.
[1] MC/DC coverage is a slightly more rigorous form than branch coverage. A condition if (a && b) requires that you only test one of (a false, b true) and (a false, b false) for 100% coverage, whereas MC/DC coverage would insist on both being tested.
Rewriting just for the sake of rewriting or because you like another language better is an almost certain recipe for disaster.
https://www.sqlite.org/testing.html
There's no question that sqlite is one of the most well understood and reliable codebases in the world.
"we are all guilty"
I'm curious though--is it true that Go brings its runtime? I'm of the impression that the runtime is only compiled in if you actually use it, but if your library is just `func Add(a, b int) int { return a + b }`, would linking against it still bring in the runtime?
And to reiterate, this is only a curiosity. Even if you could take care to avoid importing the runtime, I wouldn't think that it's worth the while.
In the present day context of vulnerabilities it's tempting to blame C. Yet it's still a good choice for many reasons. It's not wise to suggest that everything written in C must be re-written in some other language because of hand-waving reasons like buffer-overflows or off-by-one errors.
Maybe you could prove Rust is a good choice by writing your own SQLite implementation feature for feature? Until then I don't think there's going to be a compelling reason to re-write SQLite because it's written in C.
These security vulnerabilities are trivial to fix at the language level but cause extremely negative consequences if exploited successfully. Yet the default stance of C programmers is to not adress them at all. A competitor like Rust absolutely becomes necessary because of the complacency of C programmers. They will then will strongly critizise the newcomer that is built on strong fundamentals as "language of the week" that is chosen by "dumber programmers" that are trying to ride the latest hype train.
Engineering is trade offs. I think the SQLite programmers are well aware of the risk the use of C brings in the context of security vulnerabilities. Given the statistics collected here: https://www.cvedetails.com/vendor/9237/Sqlite.html it seems that their choice of C was not a security disaster, because C. It actually seems well within a risk tolerance threshold that I don't sleep uneasy at night recommending its use in systems. If their choice of language came with unavoidable vulnerabilities I'd expect those statistics to be much worse.
So what does a Rust implementation of SQLite offer anyone? That within 8 years there may be one less SQL injection attack or perhaps < 4 overflow vulnerabilities? Will it have better interoperability or performance than SQLite does now or will have once this new version is done? Will the market care?
I'm not saying that care shouldn't given when choosing to use C in a greenfield project today. I am saying that for many problem domains the risks associated with C are tolerable given a well trained, disciplined team.
And I doubt there's much of a market for an in-memory database system that is going to be feature compatible with SQLite in a few years that is also as performant as SQLite and can deployed on as many platforms. But that's just a prediction... maybe I'm wrong.
I've chosen Haskell on a greenfield project recently for many of the safety guarantees it brings... who knows?
If the problem is complacent programmers, switching languages is going to have no effect.
You're also making blanket statements about the attitudes of C programmers which have nothing to do with the C language. That probably means your generalizations are overly broad and not true, not even for a significant fraction of C programmers.
I think you need to use some logic to argue your point, not wild claims.
Please, find the vulnerability in that code, or quit making wild claims
I know, theoretically it's possible to write secure code. Even in C. Experience showed, that 99,999% of people aren't able to do that. And many reasons have its origin in the design of this bad language.
A person undertaking that task would already have a head start in that they could likely use the extensive test suite for SQLite ;-)
In terms of performance, it's really the only language out there that can claim to be as fast as C/C++. It has a minimal runtime, with the option of no runtime. It has pretty great FFI and can produce easily callable libraries for other runtime-heavy languages.
It's not, however, as broadly compatible as C. I don't think that's a problem for most cases, as most programming is now done for (MIPS|ARM|X86|X86_64), and it can handle those well enough. But microcontrollers and OS-less embedded devices still have a ways to go before Rust beats out C.
And stability is just not there. IMO, that's a good thing. Rust is the best thing to happen to systems programming in a really long time, and there are still tons of ideas with amazing potential benefits. The Rust community has been exceptional at guiding this development. I'm sure some day it will level off, but until then, I'm happy with it changing pretty rapidly.
but not necessarily posix/windows/etc
having C as a least-common-denominator toolchain which almost all platforms will provide is still useful for the embedded world where sqlite has a huge number of applications..
It's story around panic and debug handling is also improving, including cutting out formatting code for things like println! and debug! when targetting embedded platforms.
Here's exactly what I was told -- "This was achieved by manually unrolling a 10-step loop, which compiler apparently could not optimize."
The conclusion that C remains the best language is a lot harder to support. Certainly, and especially with the level of infrastructure that SQLite developed to harden its implementation, the alternatives are not so much better as to be worth the cost of migration.
Such as macOS ;)
I've done such conversions with C to Rust converter (https://gitlab.com/citrus-rs/citrus), but quickly found out that the style of writing idiomatic in C is part of the problem.
To replace code function by function you're generally forced to keep the same structs and APIs (often even internal ones) for most of the time, and these require you to erase the extra type safety, degrade smart pointers and slices to plain pointers, etc.
If you just do all the same wonky stuff that C does, but only with a slightly different syntax and compiler, you don't gain that much. The value comes from using idioms of a safer language, and that's much more work, and it's especially hard if your hands are tied by the rest of the program being C-like.
EDIT: Jokes aside, Rust sounds like the most plausible alternative, though it still lags C due to being LLVM-only, whereas C has gcc and... others.
It does however mean that there is an issue with supported platforms. Rust has support for all the major platforms already, but C is probably the most widely supported language in existence at all.
Building from source is usual, of course, given the lack of a stable ABI.
Of course, it's not usual Rust code, but no binary that small is usual code, even in C.
The point is to demonstrate that you can strip Rust down to as small as you want/need, not to suggest that every single Rust program is ultra-tiny.
Personally, I think it'd be better to just start afresh with new implementations, not necessarily under the auspices of the SQLite project. Being 'compatible' is reasonable enough and you can just ditch the lesser used or out of date features if you're not claiming to be a 100% clone.
The core is where the real compatibility issues lie and where it makes the most sense to translate, as well as where the core vulnerabilities that can be remotely exploited lie.
Candidates may be: C, C++, Rust and maybe Swift(if the deterministic ref-counting doesnt get in the way).
Of course you can do it in other langs, but it wont be able to compete with products made in languages with better fitness to this.
Eg. CouchDB vs. MongoDB. The first invented the concept, but the later used a language with better fitness for the kind of problem, and therefore was able to create a better product.
Go absolutely would not be, one of the purposes of sqlite is to be embeddable in any and all software.
That's not much of an argument considering the CLR and the JVM are written in C.
Literally every system has a C compiler, except maybe for a very small number of very old and niche systems. Assuming it cound fit on the ROM, sqlite could probably be ported to my z80 calculator, which has an 8 bit processor and 128 kiolobytes of RAM - trivially! C runs literally everywhere, which is something no other programming language can lay claim to. For a tool like sqlite there is no other choice, period.
Our platform support is limited by LLVM though, this is a good reason for sure.
Regardless, those are called "landing pads", and you can compile with an option to turn them off. Many do.
You have a hell of a lot of control over its performance and behaviour. Are there instances where you found that wasn't the case?
As far as the understanding of C goes, I can agree with that. But we are discussing rewriting sqlite in something else after all.
Seeing as Go was suggested above (which is also garbage collected), I figured I would mention Nim too.
Love to hear what those are and what language you would consider developing SQLite in.
But, I tried working in Nim without GC and out turned out to be a bigger hassle than I expected it to be. GC was supposed to be ref counted and now they're doing memory regions? There is also a -gc:stack which I'm not sure if it works the way like in C++. Then there is the pointer free paradigm. Don't get me wrong, I'm very optimistic about Nim and would be great if it really replaces C. But I feel like it's doing too many things at once.
I personally feel like Nim dev team should stop adding new features every release and work on releasing a solid 1.0.
Doubtful, rust and go do not run on nearly as many platforms as c and sqlite does. I get everyone wants to use other languages, but this incessant "rewrite in rust/go/$THING" is getting annoying.
I'm going to try to coin a new online discussion rule of programming language posts:
- At some point someone is going to suggest any problems for one language can be solved by rewriting in another language.
In this case "because c" "therefore rust". Its basically godwins law for programming language discussion.
But you'll have to overcome and provide a good reason for why anyone using sqlite should move to your new unproven shiny.
If anything sqlite is a poster child for c done (and tested/validated) right. You'll have to demonstrate a lot more than you can implement some things better in the new language.