Hacker News Reader

Comment by bb88 | Hacker News Reader

Interesting nugget about golang from their FAQ:

https://github.com/canonical/dqlite/blob/master/doc/faq.md

Why C?

The first prototype implementation of dqlite was in Go, leveraging the hashicorp/raft implementation of the Raft algorithm. The project was later rewritten entirely in C because of performance problems due to the way Go interoperates with C: Go considers a function call into C that lasts more than ~20 microseconds as a blocking system call, in that case it will put the goroutine running that C call in waiting queue and resuming it will effectively cause a context switch, degrading performance (since there were a lot of them happening). See also this issue in the Go bug tracker.

The added benefit of the rewrite in C is that it's now easy to embed dqlite into project written in effectively any language, since all major languages have provisions to create C bindings.

View on HN

They could also use D or Rust for this. If borrow-checker is too much, Rust can still do automatic, memory management with other benefits remaining. Both also support letting specific modules be unsafe where performance is critical.

> since all major languages have provisions to create C bindings.

Does WebAssembly or any of its runtimes provide a way to do this?

you can probably compile c to webassembly using clang/llvm

That's not what the parent was asking. They were asking if WASM has an FFI interface for runtimes that want to not execute the WASM code entirely sandboxed from the host, but rather want to allow you to dlload(2) code into the OS process hosting the WASM interpreter and call it through WASM ops.

Presumably, https://github.com/CraneStation/wasmtime would benefit from such an FFI API being specified.

Precisely. Do you think this is a feature the WebAssembly team should standardize or is it a bad fit for the language?

That should be doable! (Disclaimer: I work at Wasmer https://github.com/wasmerio/wasmer )

That sounds like ActiveX 2.0 at best, and an utter security nightmare at worst.

Yes, emscripten/llvm builds C code into webassembly. I'd say a significant portion of webassembly is C/C++ sourced.

You can't use emscripten/llvm with, #include <windows.h> to provide a GUI per se, can you?

Nope. There are iirc specific bindings for that in a browser context. I think there's been a lot of work for other runtimes.

Genuine question here: if you’re already going to be shipping native code, why would you need WebAssembly in the mix, especially if you’re not targeting a browser?

For example, if you want to execute "native" code in a restricted environment. WebAssembly is quite fast when (jit)compiled. Therefore, it's very tempting to use it to embed extension logic.

I don't want to flame, but I did find it curious they went with C rather than Rust. In my experience the transition is straightforward and the string handling (particularly with unicode encodings) is way better (in addition to the normal ownership benefits), and the result (an easily linkable library exposing a C ABI) is roughly the same.

Yes, and one library will be 100 MiB, the other 1.

A) disk space seems like a reasonable tradeoff for many situations, especially when binaries typically aren't the source of data consumption

B) I don't think I've seen a rust binary more than 10 megs. Cargo, rust, and ripgrep are both about 6 mb on my disk; fd is 2.5mb. These seem like a reasonable standins for a binary of significant size and complexity. sqlite3 itself is about 1.3mb.

C) Dqlite doesn't seem particularly concerned with disk-constrained systems, though I may be interpreting their site incorrectly, and the low footprint should be equally achievable with the rust runtime—surely the database itself would be a much larger concern.

This just seems like an unusually good fit for the benefits of the language—reliable client glue you can import into many runtimes where being able to prove data flow would be an strong defensive coding pattern. That said, I think that C is a good, conservative approach here, I'm certainly not knocking anyone's judgement. Overall the parent poster is absolutely correct: there's a strong correlation between use of rust's type system and size of outputted code.

EDIT: phrasing.

> Dqlite is a fast, embedded, persistent SQL database with Raft consensus that is perfect for fault-tolerant IoT and Edge devices.

Seems like, at least for embedded devices, you'd want something as small as possible so as to avoid consuming all available disk (not to say any other language will balloon it significantly or not).

I missed that it was primarily intended for embedded devices.

2.5 MiB is about what you expect the kernel size to be for an embedded device :)

Everything is an embedded device nowadays, so for reference, if you buy a WiFi AP today and open it up, you're likely to find a 8 or 16 MiB NOR flash inside, maybe a 128 MiB NAND flash (with realistically 64 MiB space since it will be doing A/B updates).

I don't think the database size is a big concern. For me the focus in dqlite is very much on the 'd' - you store atomic configuration data in there, it's not about throughput.

For an embedded database, they could certainly emphasize that more.

That's a gross exaggeration

I think it's pragmatism. There working two libraries that are already C. Also, a mentioned in another thread, type data couple bloat things a little, I think the specific statement is hyperbolic tough.

My guess is their main goal was to use this in go, where they already have experienced go and C developers and adding a third language would muddy things.

Yeah, I think that's correct too.

But it made me think that a killer app for Rust would be this entire concept done in one embedded DB with bindings for most of the popular languages.

Well they are patching SQLite and SQLite is written in C. So they'd have to maintain a C path, Rust code, and another layer of C API interface for client bindings, if I understood their architecture correctly.

Reply on news.ycombinator.com